-
|
can anyone help me on how to add cookie to a url before crawling it. I have a crawler based on storm crawler which is being used to crawl news website url's. No w I want to add cookies to the url's for a news website with premium membership. |
Beta Was this translation helpful? Give feedback.
Answered by
rzo1
Nov 16, 2023
Replies: 1 comment
-
|
You could generate the cookie externally prior to the crawling and specify it in the seed metadata using the key set-cookie. You'd need to add that key to metadata.transfer in your conf so that it gets transmitted to the outlinks and persisted to the storage and enable |
Beta Was this translation helpful? Give feedback.
0 replies
Answer selected by
rzo1
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
You could generate the cookie externally prior to the crawling and specify it in the seed metadata using the key set-cookie. You'd need to add that key to metadata.transfer in your conf so that it gets transmitted to the outlinks and persisted to the storage and enable
http.use.cookies.