add_requests with a token that has an expiry time

[DoctorEvil92]

I would use add_requests to add a lot of requests to be scheduled, these requests would run for a while. If I schedule each of these requests with a token that has a 30 minute expiry time, at some point all the requests will start failing and the token has to be renewed.

My idea was this: once the request fails, get a new token value and use send_request to retry the request. But then I also need to send this token value to every other request in the queue somehow, I don't know if that is possible?

[janbuchar]

Hello @DoctorEvil92 and thanks for your interest in Crawlee! Just to be sure, what kind of token are you talking about?

[DoctorEvil92]

Auth token that has to be passed in Authorization header, like {"Authorization":"Bearer ..."} It's needed on a site for it to give you data.
I was doing some research, would this be done with pre_navigation_hook? Because the thing is that if you call a lot of requests with the same token, they will eventually start failing when the token expires.

[janbuchar]

Oh, now I understand. I would probably store the token either completely outside of the crawler or I'd use use_state context helper. If you know when the token will expire, you may set up a background task (asyncio.create_task) that will periodically re-authenticate it. If not, then I'd just detect HTTP 401 responses in the error_handler, re-authenticate and let the crawler retry the request (this happens automatically by default).

[DoctorEvil92]

Right, I would probably store the token as a global variable, and renew it when there's a 401 status detected, or an exception thrown when parsing data. I'll have to see if it returns 200 status code with an error description or just 401 status code.

One more question about making sure that the retried request uses a new token: that would be set in the pre_navigation_hook? Something like
context.request.headers = {"Authorization":"Bearer token_value"}
Because I have to make sure that the new token is fed into the request on a retry.

[janbuchar]

Ah, yes, the pre-navigation hook should work well for this purpose.

[DoctorEvil92]

Thank you. Just one more question: is the proper way to make headers like this?

from crawlee._types import HttpHeaders 
context.request.headers = HttpHeaders({"Authorization":"Bearer token_value"})

Because if I set it to a normal dict, it gives some kind of warning

UserWarning: Pydantic serializer warnings:
  PydanticSerializationUnexpectedValue(Expected `HttpHeaders` - serialized value may not be as expected

[janbuchar]

Correct, and you can do just from crawlee import HttpHeaders.

[DoctorEvil92]

One more issue that I had to go through: 401 status code didn't trigger a retry, rather it got right into failed request handler without retrying. So it seems like you have to put ignore_http_error_status_codes=[401] into crawler initialization and then call a retry manually.

[janbuchar]

Good point, thanks for noting!