[FEATURE] Rate limit option for free API users #945
Comments
|
This is already done by the package we use to interface with the Reddit API, praw, at least according to their documentation. We might need to boost the version installed though. |
|
@Serene-Arc I thought it was, as well. I’m consistently getting 429 responses and the execution fails with an exception. Granted I am doing a lot of calls, but I suppose I expected praw to manage itself instead of failing. I’ll try updating praw. I even set my sleep to 10 minutes between bdfr executions which download 20 submissions each time. It’s better but it still fails often. It’s too hard to guess at the time until rate reset manually. EDIT: I updated praw with Additionally, I still get the exception crash with the updated praw instead of a "clean" 429 like above (not sure if this is related to me using an older approach to the progress bar). Unlike the other 429, the user is retrieved, but I guess the posts hit a 429? My thought was if a 429 is received, the x-ratelimit-reset header is parsed (or one of the other rate limit headers) and a sleep is put until the reset is reached. The 60 to continue doesn't retry the previous attempt. |
|
On second thought, it may have been the way I was using it. I authenticated with a different user than the account the app client/secret was created on because I didn't want to verify my email on one account. I created the app on the account I'm authenticated on and it seems to be going fine so far. Praw is still updated to 7.7.1. |
|
I might still force an update on the package requirements since it's been a while but I'm glad it's working. Reddit has been very difficult to work with since the API changes. |
|
Makes sense. I did a rather long run yesterday of some thousand files over some hours and I didn't get any 429 responses, so that's a plus! I also did some simple rate tests using the latest praw. From what I see it tries to keep the x-ratelimit-remaining response value on par with the x-ratelimit-reset value (converted from unix epoch time to datetime in seconds). Theoretically the requests can never drop to 0 before the reset timer resets, except it does sometimes as the matching isn't perfect. |
This is my main issue with the rate limiting. It should retry the current attempt Edit: I have made a PR for this #974 |
Description
An option that will automatically limit bdfr's API requests to a number per minute to work within Reddit's newer free API rate limits of (I think) less than 100 per minute, or averaged over 10 minutes for burst usage (most common example I've seen on reddit is 60 req/min and/or not more than 600 req for 10 min).
Currently I'm doing a sleep of a few minutes or so in-between bdfr commands. However it would be nice to be able to have functionality in bdfr that would maximize its usage while being within the free API limits by tempering its own requests. I understand PRAW handles this? Instead of an execution failing because of
prawcore.exceptions.TooManyRequests: received 429 HTTP response, could bdfr analyze the x-ratelimit-remaining, x-ratelimit-reset, x-ratelimit-used responses to wait until the program can proceed without failing?As far as I can tell the logs don't list the API requests, so it's hard to tell how many are used when executing. I'm also not sure how to check the response headers (x-ratelimit-remaining, x-ratelimit-reset, x-ratelimit-used) when running bdfr for testing or to set my sleep to the time until the ratelimit is reset.
The text was updated successfully, but these errors were encountered: