Reduce calls to clearlydefined

[akurtakov]

Clearlydefined being less and less stable makes me wonder whether in case it is down licensetool should fail or it should list not approved artifacts so they are reported to eclipse and hopefully the bot kicks in and adds records in it's own database thus clearlydefined being queried less often and prevent many cases like today it being down https://github.com/eclipse-platform/eclipse.platform.releng.aggregator/actions/runs/7127239536/job/19406740416 .
Isn't the bot that checks iplab issues checking clearlydefined? If that's the case, everything that's approved by clearlydefined should get a record in EF database created by the bot on request and both sides benefit by having less queries around.

[akurtakov]

@waynebeaton wdyt?

[waynebeaton]

The bot that processes IPLab issues doesn't check ClearlyDefined. I decided that when we're asked to review, we review.

I estimate that something like 20% of our approvals come from ClearlyDefined. I don't love the idea of adding that to the IP Team's work queue.

Having said that, providing more resiliency is good. I have a couple of thoughts...

We should add a "keep trying" option. Currently, when there's a failure to connect, it just fails. This was intentional to avoid prolonging calls/builds while we wait for timeouts. But we can be more resilient in the face of an HTTP 429 or 504 (the IPLab bot already does this). Whether or not we retry should be configurable to let the committer decide whether or not to potentially add the additional time required to retry (possibly multiple times).

Ultimately, I think that you're probably right and we should reduce our dependency on ClearlyDefined. My initial thought is that "we've failed to get a response from ClearlyDefined, so create a review issue" should be an option that is off by default. But... this is an option that committers don't want/need to understand and don't care enough about, so it's probably something that should be on by default.

[elahrvivaz]

I've been seeing more 429 responses from ClearlyDefined - having a retry or throttle would be great!

[marcdumais-work]

I've also noticed recently the more frequent 429 responses from ClearlyDefined.

Looking at their documentation, it seems dash-licenses could track API usage limits and self-throttle if/when the limit is close to be reached:

You can check the x-ratelimit-limit and x-ratelimit-remaining response headers to track your usage and reset window.

https://docs.clearlydefined.io/docs/get-involved/using-data#rest-api-usage-considerations