hash mismatch in fixed-output derivation

[shmish111]

I'm trying to build https://github.com/kemitix/thorp using a mvn2nix-lock.json file generated using mvn2nix pom.xml > mvn2nix-lock.json however one of the dependencies gives me a hash mismatch when I try to mvn2nix.buildMavenRepositoryFromLockFile {file = ./mvn2nix-lock.json;}. How is this possible? I can't find the incorrect hash mentioned anywhere either so I'm stuck as to where to look to investigate.

nix-build /nix/store/s3aqk7l404lygnam12s2282w2qsvd2sh-plexus-velocity-1.1.2.pom.drv
these derivations will be built:
  /nix/store/s3aqk7l404lygnam12s2282w2qsvd2sh-plexus-velocity-1.1.2.pom.drv
building '/nix/store/s3aqk7l404lygnam12s2282w2qsvd2sh-plexus-velocity-1.1.2.pom.drv'...

trying https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-velocity/1.1.2/plexus-velocity-1.1.2.pom
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  7595  100  7595    0     0  27418      0 --:--:-- --:--:-- --:--:-- 27418
hash mismatch in fixed-output derivation '/nix/store/v4k7a8b67jd54j8m8bxsxnfd0x948572-plexus-velocity-1.1.2.pom':
  wanted: sha256:1scssfbg7qnig60kq5aci7r0msgpwkcrlm5wavsc1x831h4riydv
  got:    sha256:1ki6h5kn3ffblxbq3wwnafbmzcfc5ibdiffh3y7k3cdkfqyw0pzl
error: build of '/nix/store/s3aqk7l404lygnam12s2282w2qsvd2sh-plexus-velocity-1.1.2.pom.drv' failed

[shmish111]

I can find what it is downloading from these logs

nix-prefetch-url https://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-velocity/1.1.2/plexus-velocity-1.1.2.pom
[0.0 MiB DL]
path is '/nix/store/v4k7a8b67jd54j8m8bxsxnfd0x948572-plexus-velocity-1.1.2.pom'
1ki6h5kn3ffblxbq3wwnafbmzcfc5ibdiffh3y7k3cdkfqyw0pzl

but I can't find where it is using 1scssfbg7qnig60kq5aci7r0msgpwkcrlm5wavsc1x831h4riydv instead of this hash

[shmish111]

well I worked out that the missing hash is just because the hash in the derivation is base32 and the one reported is base16. So this issue is actually about why does mvn2nix produce a lock file with the wrong hash?

[fzakaria]

@shmish111 sorry for getting back to you late. My life has been a bit hectic (new baby)

I actually diagnosed the hash mismatch for another person and unfortunately the solution I am doing here needs work. Maven itself can give you different hashes based on where you fetch the artifacts from!
(YUCK!)

https://docs.gradle.org/current/userguide/dependency_verification.html#sec:trusting-several-checksums

It’s quite common to have different checksums for the same artifact in the wild. How is that possible? Despite progress, it’s often the case that developers publish, for example, to Maven Central and JCenter separately, using different builds. In general, this is not a problem but sometimes it means that the metadata files would be different (different timestamps, additional whitespaces, …​). Add to this that your build may use several repositories or repository mirrors and it makes it quite likely that a single build can "see" different metadata files for the same component! In general, it’s not malicious (but you must verify that the artifact is actually correct), so Gradle lets you declare the additional artifact checksums.

I spent a long time writing https://github.com/NixOS/nixpkgs/blob/master/doc/languages-frameworks/maven.section.md which works using https://github.com/NixOS/mvn2nix-maven-plugin

I'd love to spend some time working on either repository and fixing the known bugs.

[shmish111]

Hmm, do you think this is an unsovable problem? I am surprised that the json that mvn2nix produced must have specified a different repository than that which it used? Or is there some repo mirroring or proxying going on or something?

[fzakaria]

@shmish111 the problem is that the tool first invokes Maven without knowing which repository it's utilizing to download the artifact. After which it walks the repository and checks all the repositories either provided or in the settings.xml/pom.xml; they might not necessarily be the same one.

[fzakaria]

I would love to tackle and solve this in the https://github.com/NixOS/mvn2nix-maven-plugin as well; I am considering forking it since I can't seem to find the active/current maintainer.

[chkl]

@fzakaria From my observation, it would help to prevent mvn2nix from using the local ~/.m2 repository and instead use only the specified repository, e.g. downloading all artifacts upon calling mvn2nix. How do you intend to tackle the problem?

Edit: I'm also interested in a solution, and happy to help.

[fzakaria]

@chkl I think at this moment, I should focus on fixing https://github.com/NixOS/mvn2nix-maven-plugin
I've codified how it works via https://nixos.org/manual/nixpkgs/unstable/#maven however it has a few open issues that need to be fixed.

Trying to orchestrate Maven itself has provided to be a big PITA; I didn't want to hook into the plugin system but /shrug.

[kuklopio]

I'm currently encountering this bug, is there any way to work around it?

[pasqui23]

I'm currently encountering this bug, is there any way to work around it?

mv ~/.m2{,.tmp}
mvn2nix
mv ~/m2{.tmp,}

[fzakaria]

This is a limitation of how it’s designed at the moment and that Maven can allow multiple SHA256 to exist for the same artifact depending on which Artifactory it is pulled from….

On Fri, May 20, 2022 at 12:53 AM pasqui23 ***@***.***> wrote: I'm currently encountering this bug, is there any way to work around it? mv ~/.m2{,.tmp} mvn2nix mv ~/m2{.tmp,} — Reply to this email directly, view it on GitHub <#29 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAETXDRHYM3VS5QME7YQO4TVK5AINANCNFSM4TMRQLZA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Farid Zakaria

[pasqui23]

@fzakaria can we specify which repository to check?

[fzakaria]

There is a command line flag to set the repository to check. Try that maybe 🤔

…

On Sat, May 21, 2022, 9:48 AM pasqui23 ***@***.***> wrote: @fzakaria <https://github.com/fzakaria> can we specify which repository to check? — Reply to this email directly, view it on GitHub <#29 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAETXDRRPKDUH6FJDHW4Q2DVLEHVVANCNFSM4TMRQLZA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[edeetee]

Is there any easy way to codify the maven gen plugin into the build process? Would be nice to see how its run and means we can make adjustments if needed. And thanks for the flake, very handy stuff.