Handle Rate Limits in Pagination with Automatic Retries

[Weyaxi]

This PR enhances the paginate function in src/huggingface_hub/utils/_pagination.py/_pagination.py by implementing automatic retries using the http_backoff function. Previously, when attempting to list all models via api.list_models(), the process would fail due to rate limiting (HTTP 429) after approximately 500K models (this number depends on many factors). This led to data loss and required restarting the process from the beginning. Additionally, after restarting, the process would hit the rate limit again after another 500K models, making it impossible to scrape all models in a single run.

Changes & Improvements:

• Replaced session.get with http_backoff to enable automatic retries.
• Configured max_retries=20 and retry_on_status_codes=429 to handle API rate limits gracefully.
• Generalized solution so that all API calls using paginate benefit from improved stability (though this rate limiting is most likely to occur only when listing >1.5M models).

Why This Change?

• Prevents failures & data loss when paginating through large queries.
• Ensures uninterrupted API calls by retrying when rate limits are hit.
• Enhances pagination reliability for all functions using paginate, not just model listing.

This PR was initially planned to implement retry logic directly within _pagination.py instead of using the http_backoff function. However, thanks to @Wauplin's great guidance in pointing out that http_backoff already exists and is highly stable, this approach is now much better! 🙂

Thanks, @Wauplin! 🚀

---

Below screenshot shows the behavior before this PR. As seen in the screenshot, an exception was thrown when the rate limit was hit, and restarting the process would only result in the same issue occurring again. As stated earlier, this made it impossible to access the remaining models in a simple way. 🙂

[Wauplin]

Thanks for opening the PR @Weyaxi ! As discussed in DMs, I think it's a reasonable solution 👍

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[Wauplin]

shoud add from . import get_session, hf_raise_for_status, http_backoff, logging in import section

[Weyaxi]

Hey, I forgot, sorry. Should be done with this:

5d455cd

[Wauplin]

Let's wait for green CI and we're good

[hanouticelina]

looks good to me, thank you!

[Wauplin]

It seems this test is broken: https://github.com/huggingface/huggingface_hub/blob/main/tests/test_utils_pagination.py#L9 Would you like to check what's wrong with it? Most likely the http_backoff method that has to be mocked as well (same as currently done with get_session). Let me know if you need help.

[Wauplin]

(pushed 03f8ff9 to fix the mocked tests)