Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Object file hash prefix creates determinism issues #1271

Closed
nmattia opened this issue Nov 5, 2024 · 2 comments
Closed

Object file hash prefix creates determinism issues #1271

nmattia opened this issue Nov 5, 2024 · 2 comments

Comments

@nmattia
Copy link
Contributor

nmattia commented Nov 5, 2024

Hi,

I've noticed that when generating object files, the full path of the source files is hashed and included in the object file name. This causes some issues in build systems like Bazel and Nix because the intermediary files will change depending on the directory where the build is performed. This also seems to influence the final output.

I've created a strawman PR with a fix that I've only tested on 1.0.83, though it makes some assumptions about where the source files are located.

What would be a good way of fixing this?

Thanks!
nmattia added a commit to dfinity/ic that referenced this issue Nov 5, 2024
@nmattia
fix(IDX): make replica build more deterministic
9f0d834 
This includes a couple of fixes to make the replica build more
deterministic.

This in theory allows anyone building `//rs/replica` with Bazel in the
Docker image to get a bit-for-bit reproducible replica executable, and
this should also improve Bazel cache hit rates.

This fixes for the following issues:

* Non-reproducible `jemalloc` build: the `tikv-jemalloc-sys` crate
  vendors `jemalloc` and builds it as part of `build.rs`. Unfortunately
  that build in not deterministic. To make it more deterministic we
  build `jemalloc` separately, which also speeds up rebuilds as the C
  code does not need to be rebuilt when rust versions change. We only
  enable this in Linux; this also includes a patch to support this in
  `rules_rust`: bazelbuild/rules_rust#2981

* Non-reproducible `rules_rust` build log: this includes a backport of a
  fix that disables build logs in `rules_rust` by default (backported
  because our build is not compatible with the latest `rules_rust`)
  bazelbuild/rules_rust#2974

* Non-reproducible make & pkgconfig toolchains: some toolchains packaged
  by `rules_foreign_cc` cause build determinism issues so instead we use
  the ones installed in the container and remove build logs: bazel-contrib/rules_foreign_cc#1313

* Non-reproducible obj file generation in cc-rs: the `cc-rs` crate used
  in many C builds, including the (ASM) build of `ring`'s crypto bits,
  generates object files that include the Bazel sandbox full path:
  rust-lang/cc-rs#1271

* Non-reproducible codegen: `cranelift-isle` and
  `cranelift-codegen-meta` include references to source files as
  absolute paths that include the Bazel sandbox path:
  bytecodealliance/wasmtime#9553
nmattia added a commit to dfinity/ic that referenced this issue Nov 5, 2024
@nmattia
fix(IDX): make replica build more deterministic
7021376 
This includes a couple of fixes to make the replica build more
deterministic.

This in theory allows anyone building `//rs/replica` with Bazel in the
Docker image to get a bit-for-bit reproducible replica executable, and
this should also improve Bazel cache hit rates.

This fixes for the following issues:

* Non-reproducible `jemalloc` build: the `tikv-jemalloc-sys` crate
  vendors `jemalloc` and builds it as part of `build.rs`. Unfortunately
  that build in not deterministic. To make it more deterministic we
  build `jemalloc` separately, which also speeds up rebuilds as the C
  code does not need to be rebuilt when rust versions change. We only
  enable this in Linux; this also includes a patch to support this in
  `rules_rust`: bazelbuild/rules_rust#2981

* Non-reproducible `rules_rust` build log: this includes a backport of a
  fix that disables build logs in `rules_rust` by default (backported
  because our build is not compatible with the latest `rules_rust`)
  bazelbuild/rules_rust#2974

* Non-reproducible make & pkgconfig toolchains: some toolchains packaged
  by `rules_foreign_cc` cause build determinism issues so instead we use
  the ones installed in the container and remove build logs: bazel-contrib/rules_foreign_cc#1313

* Non-reproducible obj file generation in cc-rs: the `cc-rs` crate used
  in many C builds, including the (ASM) build of `ring`'s crypto bits,
  generates object files that include the Bazel sandbox full path:
  rust-lang/cc-rs#1271

* Non-reproducible codegen: `cranelift-isle` and
  `cranelift-codegen-meta` include references to source files as
  absolute paths that include the Bazel sandbox path:
  bytecodealliance/wasmtime#9553
@nmattia nmattia mentioned this issue Nov 5, 2024
Merged
@NobodyXu
Copy link
Collaborator

NobodyXu commented Nov 5, 2024

You can use compile_intermediates, which output paths to the objects compiled and you can assmble it manually from there.

github-merge-queue bot pushed a commit to dfinity/ic that referenced this issue Nov 5, 2024
@nmattia
fix(IDX): make replica build more deterministic (#2441)
74352d1 
This includes a couple of fixes to make the replica build more
deterministic.

This in theory allows anyone building `//rs/replica` with Bazel in the
Docker image to get a bit-for-bit reproducible replica executable, and
this should also improve Bazel cache hit rates.

This fixes for the following issues:

* Non-reproducible `jemalloc` build: the `tikv-jemalloc-sys` crate
vendors `jemalloc` and builds it as part of `build.rs`. Unfortunately
that build in not deterministic. To make it more deterministic we build
`jemalloc` separately, which also speeds up rebuilds as the C code does
not need to be rebuilt when rust versions change. We only enable this in
Linux; this also includes a patch to support this in `rules_rust`:
bazelbuild/rules_rust#2981

* Non-reproducible `rules_rust` build log: this includes a backport of a
fix that disables build logs in `rules_rust` by default (backported
because our build is not compatible with the latest `rules_rust`)
bazelbuild/rules_rust#2974

* Non-reproducible make & pkgconfig toolchains: some toolchains packaged
by `rules_foreign_cc` cause build determinism issues so instead we use
the ones installed in the container and remove build logs:
bazel-contrib/rules_foreign_cc#1313

* Non-reproducible obj file generation in cc-rs: the `cc-rs` crate used
in many C builds, including the (ASM) build of `ring`'s crypto bits,
generates object files that include the Bazel sandbox full path:
rust-lang/cc-rs#1271

* Non-reproducible codegen: `cranelift-isle` and
`cranelift-codegen-meta` include references to source files as absolute
paths that include the Bazel sandbox path:
bytecodealliance/wasmtime#9553
alin-at-dfinity pushed a commit to dfinity/ic that referenced this issue Nov 7, 2024
@nmattia @alin-at-dfinity
fix(IDX): make replica build more deterministic (#2441)
fc88220 
This includes a couple of fixes to make the replica build more
deterministic.

This in theory allows anyone building `//rs/replica` with Bazel in the
Docker image to get a bit-for-bit reproducible replica executable, and
this should also improve Bazel cache hit rates.

This fixes for the following issues:

* Non-reproducible `jemalloc` build: the `tikv-jemalloc-sys` crate
vendors `jemalloc` and builds it as part of `build.rs`. Unfortunately
that build in not deterministic. To make it more deterministic we build
`jemalloc` separately, which also speeds up rebuilds as the C code does
not need to be rebuilt when rust versions change. We only enable this in
Linux; this also includes a patch to support this in `rules_rust`:
bazelbuild/rules_rust#2981

* Non-reproducible `rules_rust` build log: this includes a backport of a
fix that disables build logs in `rules_rust` by default (backported
because our build is not compatible with the latest `rules_rust`)
bazelbuild/rules_rust#2974

* Non-reproducible make & pkgconfig toolchains: some toolchains packaged
by `rules_foreign_cc` cause build determinism issues so instead we use
the ones installed in the container and remove build logs:
bazel-contrib/rules_foreign_cc#1313

* Non-reproducible obj file generation in cc-rs: the `cc-rs` crate used
in many C builds, including the (ASM) build of `ring`'s crypto bits,
generates object files that include the Bazel sandbox full path:
rust-lang/cc-rs#1271

* Non-reproducible codegen: `cranelift-isle` and
`cranelift-codegen-meta` include references to source files as absolute
paths that include the Bazel sandbox path:
bytecodealliance/wasmtime#9553
@nmattia
Copy link
Contributor Author

nmattia commented Nov 8, 2024

Fixed in #1270

@nmattia nmattia closed this as completed Nov 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@nmattia @NobodyXu