Substack refused to let PageAssist work

[BradKML]

Maybe sites like Substack have an anti-scraping function that does JS-based rendering, can PageAssist take the article after it has been rendered in the browser and do the embedding that way, rather than recalling HTTP request (I think that is how it is currently done)?

[n4ze3m]

Yes, I am currently adding a similar feature where you can select a specific component on the site, rather than the entire site, to chat with

[BradKML]

What if I want the entire site, but they are in fact weird in their page loading? Can they just load directly from the browser rendering?

[n4ze3m]

I just tested the Substack website. The scripting is working fine, but it embedded a lot of unwanted tags, which is why the rag is not working well. I will add a separate parser for Substack websites

[BradKML]

Yeah is this an issue of article extraction? If so could you help and evaluate tools like Trafilatura? https://github.com/adbar/trafilatura
Also I am not sure if the extraction benchmark is considered "correct" now, maybe there is a way out with this? scrapinghub/article-extraction-benchmark#3

[n4ze3m]

It is Python code, and the Page Assist extension is JavaScript. But anyway, thanks for the suggestions! I will check the code and improve the Page Assist scrapping code 😃😃

[BradKML]

I think the benchmark test list includes libraries made with JS? (just that Trafilatura is the best performer in the game but "readability" comes in multiple languages)