Sign-In before Scraping

[Aswath1709]

How to scrape a website that needs to sign-in first to fetch important details?

[dosubot[bot]]

Hi @Aswath1709! I'm Dosu and I’m helping the Scrapegraph-ai team.

ScrapeGraphAI doesn't automate the login process for sites that require sign-in. The recommended approach is to use a tool like Playwright to log in to the website and save your session cookies (storage state) to a file. You then pass this file to ScrapeGraphAI using the storage_state parameter in your graph config. This allows ScrapeGraphAI to access authenticated pages as if you were logged in. You can see an example of this workflow and code snippets in this issue: GitHub Issue #953.

The relevant classes are FetchNode and ChromiumLoader, which both support the storage_state parameter for session persistence. This method works for cookie/session-based authentication, but for other methods (like OAuth or API tokens), you'll need to handle those externally and provide the necessary headers or session info yourself.

There aren't official docs or examples dedicated to authentication, but the codebase and issues provide guidance for advanced use. Be aware that session cookies can expire, so you may need to refresh them periodically.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}   

[Aswath1709]

Thanks, got it solved!