Check for rate limiting while fetching posts #7
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cyle
commented
Dec 7, 2023
| case 420: // rate limited | ||
| case 429: // rate limited | ||
| // make this chill for a minute | ||
| return false; |
There was a problem hiding this comment.
looking at the calling side(s), returning false seems to make it think it just needs to wait for later? it's hard for me to understand how this is meant to work, and it's definitely not resilient for this kind of event. looks like it'll keep while looping and getting a 429 Limit Exceeded, but maybe at least now it won't fail outright and stop trying.
|
maybe @westi can help me with this |
|
I tested locally with a Tumblr blog that had over 1,000 posts, and it worked well. Any chance you can test this with a Tumblr blog that has a larger amount of posts? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
If you're trying to import posts from a blog that has thousands of posts, you will quickly hit the rate limit. I think this should make the importer chill and wait for the next minute to try again?
cc @pento