Wikidata calls failing, likely due to proxy limitations / rate limits

[mekarpeles]

Problem

https://sentry.archive.org/organizations/ia-ux/issues/540536/?environment=production&p[…]kidata&referrer=issue-stream&statsPeriod=12h&stream_index=0

The first line works, attempts to connect to ipv4, the second one fails:

1739318415.282    197 207.241.234.180 TCP_TUNNEL/200 5131 CONNECT www.wikidata.org:443 - HIER_DIRECT/208.80.153.224 -
1739376095.816     13 207.241.234.180 NONE_NONE/503 0 CONNECT www.wikidata.org:443 - HIER_DIRECT/2620:0:860:ed1a::1 -

Hypothesis

The hypothesis is that, since we implemented proxy requests, we are no longer making requests from 3 servers (ol-web[0-2]) and instead just one. We need to explore a way to either cache more aggressively (for longer) or batch lookups in the same way that our bookworm affiliate-server does. Otherwise, the recommendation is we temporarily disable the calls to avoid making too many calls to wikidata.

We are hitting several thousand errors per 12h time period.

cc: @RayBB @cdrini

Reproducing the bug

1. Go to ...
2. Do ...

• Expected behavior:
• Actual behavior:

Context

• Browser (Chrome, Safari, Firefox, etc):
• OS (Windows, Mac, etc):
• Logged in (Y/N):
• Environment (prod, dev, local): prod

Breakdown

Requirements Checklist

• [ ]

Related files

Stakeholders

---

Instructions for Contributors

• Please run these commands to ensure your repository is up to date before creating a new branch to work on this issue and each time after pushing code to Github, because the pre-commit bot may add commits to your PRs upstream.

[RayBB]

@mekarpeles How many total call are we making vs how many are failing?

[jimchamp]

Are any of these calls succeeding? I think that we need to update the endpoint:

• This works: https://www.wikidata.org/w/rest.php/wikibase/v1/entities/items/Q278816
• This doesn't: https://www.wikidata.org/w/rest.php/wikibase/v0/entities/items/Q278816

[RayBB]

@jimchamp I think you're exactly right. As I understand this endpoint doesn't have a rate limit and if it did it should be giving a header to state such. So we should update the url.

[jimchamp]

We may also want to record non-200 _get_from_web() responses in Sentry somehow, in order to surface similar issues in the future.

[jimchamp]

I've opened #10452 to update the endpoint.

[jimchamp]

#10452 has been patch deployed, and I can see that entries are again being added to the wikidata table. As a follow-up, I'll make sure that all non-200 Wikidata requests are captured in Sentry.

[jimchamp]

#10456 adds Wikidata API errors to our Sentry logs.

I'm still seeing new ProxyError events in Sentry after patch-deploying yesterday, but much fewer of them:

Errors over a seven-day period.

20 errors today, so far (down from 15,900 yesterday)

[RayBB]

Add the 20 errors still 503s? Is it possible to get the IDs of the entities failing?

[jimchamp]

They're all the same: 503 ProxyErrors.

Looks like these errors from the past day have largely been triggered by bots:

The IDs are included in the event title (pictured here). Do you want a small sample, or all of the events that occurred since the patch deployment?