HTTP 524 Error in v1.1.1

[mhellmeier]

Situation

I tried to update the DEPENDENCIES file of eclipse-tractusx/eclipse-tractusx.github.io.

Problem

When using the latest Dash tool in version 1.1.1, the update fails with the HTTP 524 error, even after several tries:
First try:

$ java -jar ./org.eclipse.dash.licenses-1.1.1.jar package-lock.json -project automotive.tractusx -summary ./DEPENDENCIES
[main] INFO Querying Eclipse Foundation for license data for 500 items.
[main] INFO Found 148 items.
[main] INFO Querying Eclipse Foundation for license data for 500 items.
[main] INFO Found 77 items.
[main] INFO Querying Eclipse Foundation for license data for 447 items.
[main] INFO Found 73 items.
[main] INFO Querying ClearlyDefined for license data for 500 items.
[main] ERROR Error response from ClearlyDefined HTTP 524

Second try:

$ java -jar ./org.eclipse.dash.licenses-1.1.1.jar package-lock.json -project automotive.tractusx -summary ./DEPENDENCIES
[main] INFO Querying Eclipse Foundation for license data for 500 items.
[main] INFO Found 148 items.
[main] INFO Querying Eclipse Foundation for license data for 500 items.
[main] INFO Found 77 items.
[main] INFO Querying Eclipse Foundation for license data for 447 items.
[main] INFO Found 73 items.
[main] INFO Querying ClearlyDefined for license data for 500 items.
[main] ERROR Error response from ClearlyDefined HTTP 524

When using the older version 1.1.0, it works:

$ java -jar ./org.eclipse.dash.licenses-1.1.0.jar package-lock.json -project automotive.tractusx -summary ./DEPENDENCIES
[main] INFO Querying Eclipse Foundation for license data for 500 items.
[main] INFO Found 148 items.
[main] INFO Querying Eclipse Foundation for license data for 500 items.
[main] INFO Found 77 items.
[main] INFO Querying Eclipse Foundation for license data for 447 items.
[main] INFO Found 73 items.
[main] INFO Querying ClearlyDefined for license data for 500 items.
[main] INFO Found 473 items.
[main] INFO Querying ClearlyDefined for license data for 500 items.
[main] INFO Found 459 items.
[main] INFO Querying ClearlyDefined for license data for 92 items.
[main] INFO Found 92 items.
[main] INFO Vetted license information was found for all content. No further investigation is required.

Used Versions

• v1.1.0 (works)
• v1.1.1-20250207.065028-427 (doesn't work)

[waynebeaton]

There are no meaningful changes in how we access the ClearlyDefined API between those versions. We're using the HTTPClient from the JRE, so -- unless you're using a different JRE to invoke the tool -- there are no differences there. To the best of my knowledge, the calls to the API are identical in these (and every other) version.

500 level responses describe server errors. My experience has been that the ClearlyDefined server sometimes does this and you just have to wait and try again. We don't currently handle rate limits and what-not, but a 524 does not describe a rate limit related problem.

My assumption at this point is that difference must be an unfortunate consequence of timing.

Does the problem still exist?

HTTP 524 is a timeout error, so you might try playing with the timeout parameter. Although, I'm not sure that setting timeout in the client actually has an effect in this case (the error is coming from the server). Also, the default value is 120 seconds which already seems like a long time.

Another option might be to lower the batch size with the batch parameter (e.g., set it to 100). This will result in more calls, but the server may have a better time with the smaller request.

I'll add some debug code to the HTTP client so that we can at least capture the response headers to see if there's anything helpful in there.

[waynebeaton]

I've pushed an update that will appear in the next build. If you're still getting the error, please add -Dorg.slf4j.simpleLogger.defaultLogLevel=debug when you invoke the tool and share the response headers.

[mhellmeier]

Thanks! Today, it works with the v1.1.1. But if I am able to reproduce the error, I will try to add more details based on the debug log level flag.

@stephanbcbauer: Maybe this is also interesting for you!

[stephanbcbauer]

Thanks! Today, it works with the v1.1.1. But if I am able to reproduce the error, I will try to add more details based on the debug log level flag.

@stephanbcbauer: Maybe this is also interesting for you!

Thanks @mhellmeier haven't tried the 1.1.1 yet :)

[tom-rm-meyer-ISST]

I've pushed an update that will appear in the next build. If you're still getting the error, please add -Dorg.slf4j.simpleLogger.defaultLogLevel=debug when you invoke the tool and share the response headers.

Hey @waynebeaton,

for me the debug level parameter does not work:

eclipseDashTool package-lock.json -project automotive.tractusx -summary DEPENDENCIES_BRUNO_CLI -Dorg.slf4j.simpleLogger.defaultLogLevel=debug
usage: org.eclipse.dash.licenses.cli.Main [-batch <int>] [-confidence <int>]
       [-excludeSources <sources>] [-help] [-project <shortname>] [-repo <url>]
       [-review] [-summary <file>] [-timeout <seconds>] [-token <token>]

eclipseDashTool is an alias for java -jar ~/path/dashtool.jar.

Using version 1.1.0 I also run into the 524:

eclipseDashTool package-lock.json -project automotive.tractusx -summary DEPENDENCIES_BRUNO_CLI
[main] INFO Querying Eclipse Foundation for license data for 272 items.
[main] INFO Found 69 items.
[main] INFO Querying ClearlyDefined for license data for 202 items.
[main] ERROR Error response from ClearlyDefined 524

when using a smaller batch size as suggested above, it works!

eclipseDashTool package-lock.json -project automotive.tractusx -summary DEPENDENCIES_BRUNO_CLI -batch 100
[main] INFO Querying Eclipse Foundation for license data for 100 items.
[main] INFO Found 5 items.
[main] INFO Querying Eclipse Foundation for license data for 100 items.
[main] INFO Found 40 items.
[main] INFO Querying Eclipse Foundation for license data for 71 items.
[main] INFO Found 24 items.
[main] INFO Querying ClearlyDefined for license data for 100 items.
[main] INFO Found 96 items.
[main] INFO Querying ClearlyDefined for license data for 100 items.
[main] INFO Found 99 items.
[main] INFO Querying ClearlyDefined for license data for 2 items.
[main] INFO Found 2 items.
[main] INFO Vetted license information was found for all content. No further investigation is required.

[waynebeaton]

@tom-rm-meyer-ISST The debug option needs to be sent to the JVM. This is a Java idiosyncrasy that we all discover, but then forget to share. When you use the CLI, it needs to go before the -jar option.

e.g.,

java -Dorg.slf4j.simpleLogger.defaultLogLevel=debug -jar ~/path/dashtool.jar package-lock.json -project automotive.tractusx -summary DEPENDENCIES_BRUNO_CLI 

The problem is, AFAICT, with ClearlyDefined. I arrived at the default batch size of 500 through some trial and error as a happy medium between the number of calls and size of the payload; but it seems that it's time to reconsider.

It does seem like sending a smaller payload solves the problem.

There's some functionality that automatically splits the request when we get a 500 error (clearlydefined/service#957); I'll extend it to do the same on a 524. I'm thinking that this gives us the best of both worlds: we at least try to reduce the number of calls.

Side note: I just noticed that this issue is number 429; HTTP 429 is "too many requests". I find this funny.

[tom-rm-meyer-ISST]

@tom-rm-meyer-ISST The debug option needs to be sent to the JVM. This is a Java idiosyncrasy that we all discover, but then forget to share. When you use the CLI, it needs to go before the -jar option.

e.g.,

java -Dorg.slf4j.simpleLogger.defaultLogLevel=debug -jar ~/path/dashtool.jar package-lock.json -project automotive.tractusx -summary DEPENDENCIES_BRUNO_CLI 

Alright, TBH didn't think that way / wasn't aware of that. Thanks!

The problem is, AFAICT, with ClearlyDefined. I arrived at the default batch size of 500 through some trial and error as a happy medium between the number of calls and size of the payload; but it seems that it's time to reconsider.

It does seem like sending a smaller payload solves the problem.

There's some functionality that automatically splits the request when we get a 500 error (clearlydefined/service#957); I'll extend it to do the same on a 524. I'm thinking that this gives us the best of both worlds: we at least try to reduce the number of calls.

Thanks a lot! Another question beside that: Is it on purpose that there is no Github release for version 1.1.1?

Side note: I just noticed that this issue is number 429; HTTP 429 is "too many requests". I find this funny.

Haha, made me laugh out loud! 😆

[waynebeaton]

Thanks a lot! Another question beside that: Is it on purpose that there is no Github release for version 1.1.1?

1.1.1 hasn't been released yet. It's a SNAPSHOT build.