Skip to content

Multiple ES-Hadoop versions detected - false positive #2371

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
frensjan opened this issue Apr 4, 2025 · 0 comments
Open

Multiple ES-Hadoop versions detected - false positive #2371

frensjan opened this issue Apr 4, 2025 · 0 comments

Comments

@frensjan
Copy link

frensjan commented Apr 4, 2025

org.elasticsearch.hadoop.util.Version throws a RuntimeException if it finds the class itself multiple times on the class-path:

Multiple ES-Hadoop versions detected in the classpath; please use only one
[list of jars containing the Version class]

This probably is a great help in cases where issues arise from actually having different versions on the class-path. However, it actually doesn't check for whether there actually are different versions.

When deploying with Flink we run into the issue that ES Hadoop is added to the class-path twice. Once because it is packaged in a user-lib, but it is also added to the class-path for YARN deployments. This is unfortunate, not that easy to work around, especially because it also influences a lot of other class-path issues.

I'd like to provide a PR which actually checks if there are multiple different versions on the class-path.

I'm thinking about something like:

Set versions = get versions
if(versions.size() > 1) {
   throw ...
}

instead of the current implementation of

List jars = get normalized jars containing `org.elasticsearch.hadoop.util.Version`
if(jars.size() > 1) {
   throw ...
}

Any (early) feedback is more than welcome!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant