[BUG]: 'arxiv' search result is unexpectedly empty

[timsanders256]

Description of the bug

data = kb.get_data(10, "Machine Learning") sometimes returns a empty dictionary. After testing the arxiv module I found that when requesting frequently, it would return a empty search result. The reason might be found at issue #129 of arxiv:

liyucheng09 commented Oct 17, 2023
I tried about 300 attempts hourly today. More than 3000 in total. 0 out of 3000 suceeded.
By sending a user-agent to the feedparser, 28 out of 100 suceeded.
I suppose we could safely say arxiv is declining requests from programmatic clients.

So it should not be our problem.

Steps To Reproduce

No response

Additional Information

No response

[timsanders256]

Or we might inherit the Client class, and alter line #660

resp = self._session.get(url, headers={"user-agent": "arxiv.py/2.1.0"})

to pretend to be a browser.

[lwaekfjlk]

nice finding @timsanders256 ! Can we add a pytest function to test get_data and make sure it can alwasy return something?

[timsanders256]

Yes, that would be a great idea.

[lwaekfjlk]

Could you create a PR for this issue? @timsanders256

[ft2023]

I feel weird. Did you test it locally? Network issues? (vpn) That didn't happen on my side of the test. In addition, we may not call this function too often in real usage.

[lwaekfjlk]

This issue would be fixed once #59 is fixed.

[lwaekfjlk]

arxiv only calls once now. abort this issue.