Rust reproducibility issue - Finding the proper fix

[sundeep-kokkonda]

We are building Rust for a custom target in the Yocto framework. Here we run a test called reproducibility to ensure the rust binaries are identical between the builds.
We do that by building and comparing the rust binaries & libs in two different build directories -

1. reproducibleA &
2. reproducibleB-extended

With every rust release we've seen a several issues which makes the rust binaries are not reproducible. The issues seen are with several reasons like -

1. Regressions caused by rust updates (absolute path in proc-macro)
   a9a9798

2. Incorrect/missing Rust config settings (remap-debuginfo is not set by default)
   https://lists.openembedded.org/g/openembedded-core/message/188940

3. Compiler options and settings - rustdoc failed with codegen-units option setting
   https://lists.openembedded.org/g/openembedded-core/message/202540

4. Regressions with Crate updates - CC crate appending a unique hash to the object files
   Fix for different hashes between different build paths cc-rs#1277
   Use relative directory for obj files hash cc-rs#1270

There are a few other failure cases we've analyzed and fixed. This issue is happening for multiple reasons and these failure are resulting in lot of debug efforts and project delays.

This can be avoided by having such a reproducibility test on rust itself.

Any comments or inputs on detecting and fixing this issue during rust development itself? (via a test case or any other ways)

[kennytm]

Reproducibility bugs are tracked in #129080. You can also monitor the label A-reproducibility Area: Reproducible / deterministic builds .

The "having such a test" on the CI, as you have seen from #75362, I don't think there is much progress on it.

[workingjubilee]

This should not be in rust-lang/rfcs?

@rustbot transfer rust

[rustbot]

Error: The feature transfer is not enabled in this repository.
To enable it add its section in the triagebot.toml in the root of the repository.

Please file an issue on GitHub at triagebot if there's a problem with this bot, or reach out on #t-infra on Zulip.

[workingjubilee]

@sundeep-kokkonda Yocto's Poky build system modifies the Rust toolchain in several ways to make it more... reproducible... and unfortunately often avoids upstreaming such changes. This means that Yocto avoids receiving feedback on whether such changes are appropriate, or why. This also means that we do not hear about all of your reproducibility concerns, either, only when Yocto thinks that a workaround downstream would be harder. It is unfortunate that you frequently have complications, but unfortunately not surprising.

I would like to acknowledge, for note, that you personally do frequently pursue upstream resolutions. I am merely noting this because I cannot track every patch that Poky maintains on the Rust toolchain to upstream rustc or cargo issues, and a number do not seem to be authored or reviewed by you.

Your proposal sounds fine, but would be most meaningful if Yocto also tried to adopt an upstream-first stance and worked to remove all patches from the Poky build system. That way there would not be a significant lag or distortions in the feedback, and fixes would be both correct and actual fixes, not papering over problems. I realize you are not your brother's keeper, per se, but... you have more of an idea than I do on how to advance such a proposal with your peers, I expect.

On the rustc side, obviously, the two issues kennytm mentioned would have to be addressed.

But it would be most ideal if there was an attempt to audit for or document what the most common sources of reproducibility issues are. I mean, "build paths embedded in the binary", obviously, but in more detail (common places build paths get introduced, etc.). Somewhere in the rustc-dev-guide, perhaps? And then that could be accompanied by tests, and code that tends to make reproducibility harder could have an internal lint attached to it, combined with a (probably centralized?), blessed pattern.

[jieyouxu]

To add, a major challenge for the reproducibility problems is that it takes someone (more like, multiple interested contributors) to actually go back and revisit/re-triage the issues, and then also preferrably document the specific places where this is problematic (issues are prefectly fine) and how one might be able to achieve the irreprodubility. When you just say "rustc is not reproducible" or "bootstrap is not reproducible", it's not actionable -- there are a lot of places that might be env-sensitive or somehow non-deterministic, and without knowing specifically what, it's just going to be buried in the issue tracker.

[sundeep-kokkonda]

@workingjubilee

Yocto's Poky build system modifies the Rust toolchain in several ways to make it more... reproducible... and unfortunately often avoids upstreaming such changes.

I can see we duly communicating the pathes in rust code to upstream, I looked into current Yocto rust patches and the changes we made for rust build to work in Yocto environment are reported to upstream.
rust-lang/cargo#13922
rust-lang/cc-rs#1277
madler/zlib#764

Your proposal sounds fine, but would be most meaningful if Yocto also tried to adopt an upstream-first stance and worked to remove all patches from the Poky build system.

The remaining patches are for testsuit which disables a few tests that are getting failed in Yocto environment (we are currenltly working on fixing those tests).

Anyway, We will work more closely w/ upstream in future by bringing the issues to upstream notice (like test failures... etc)

what the most common sources of reproducibility issues are.

The repro issues are seen not only with embedded build path but for several other reasons like mentioned in the #134589 (comment) (options, confg, code changes etc). We are open to work with upstream on detecting & fixing it by writing a tests/any other better ways. Looking for your inputs on it.