Linux cargo binary selected on Mac OS (Darwin) with bzlmod #2549

kolloch · 2024-03-14T13:19:33Z

After switching version rules_rust 0.4.0 from WORKSPACE to MODULE.bazel on Bazel 7.1.0, we get the following error on Darwin:

/private/var/tmp/_bazel_elwin.stephan/452d85a16bcec03a5e8897f11c406e89/external/rules_rust~~rust~rust_host_tools/bin/cargo: cannot execute binary file

Weirdly enough it seems to provision a Linux binary here:

file /private/var/tmp/_bazel_elwin.stephan/452d85a16bcec03a5e8897f11c406e89/external/rules_rust~~rust~rust_host_tools/bin/cargo
/private/var/tmp/_bazel_elwin.stephan/452d85a16bcec03a5e8897f11c406e89/external/rules_rust~~rust~rust_host_tools/bin/cargo: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, with debug_info, not stripped

Of course, chances are that we are just doing it wrong!

This is what we use in WORKSPACE (working):

http_archive(
    name = "rules_rust",
    integrity = "sha256-ww398ehv1QZQp26mRbOkXy8AZnsGGHpoXpVU4WfKl+4=",
    urls = ["https://github.com/bazelbuild/rules_rust/releases/download/0.40.0/rules_rust-v0.40.0.tar.gz"],
)

load("@rules_rust//rust:repositories.bzl", "rules_rust_dependencies", "rust_register_toolchains")

rules_rust_dependencies()

rust_version = "1.76.0"

rust_register_toolchains(
    edition = "2021",
    versions = [
        rust_version,
    ],
)

load("@rules_rust//crate_universe:repositories.bzl", "crate_universe_dependencies")

crate_universe_dependencies()

load("@rules_rust//crate_universe:defs.bzl", "crates_repository")

crates_repository(
    name = "crate_index",
    cargo_lockfile = "//:Cargo.lock",
    lockfile = "//:cargo-bazel-lock.json",
    manifests = [
        "//:Cargo.toml",
        "//bazel/deliverables:Cargo.toml",
        "//bazel/deliverables/gitlab:Cargo.toml",
        "//rust/kubefacts:Cargo.toml",
        "//rust/awsfacts:Cargo.toml",
    ],
)

load("@crate_index//:defs.bzl", "crate_repositories")

crate_repositories()

This is what we tried in MODULE.bazel (leading to the error above):

bazel_dep(
    name = "rules_rust",
    version = "0.40.0",
)

rust = use_extension("@rules_rust//rust:extensions.bzl", "rust")

rust.toolchain(
    edition = "2021",
)

use_repo(
    rust,
    "rust_toolchains",
)

register_toolchains("@rust_toolchains//:all")

crate = use_extension(
    "@rules_rust//crate_universe:extension.bzl",
    "crate",
)

crate.from_cargo(
    name = "crate_index",
    cargo_lockfile = "//:Cargo.lock",
    #    lockfile = "//:cargo-bazel-lock.json",
    manifests = [
        "//:Cargo.toml",
        "//bazel/deliverables:Cargo.toml",
        "//bazel/deliverables/gitlab:Cargo.toml",
        "//rust/kubefacts:Cargo.toml",
        "//rust/awsfacts:Cargo.toml",
    ],
)

use_repo(crate, "crate_index")

The text was updated successfully, but these errors were encountered:

bgeron · 2024-03-15T17:09:08Z

I'm getting the same but somehow inverted.

I'm running

on Apple Silicon (M3 arm64)
within Rosetta Docker, Ubuntu Linux amd64
rules_rust 0.40.0 + 1 commit (8b977b5), used through a local_path_override
Rust 1.75

and in the Bazel output I see it's extracting rustc-1.75.0-aarch64-apple-darwin.tar.xz which is clearly wrong.

However, all of the following seem to resolve the problem:

Using Rust 1.74 instead
Using Rust 1.76 instead
Sprinkling extra print statements throughout rules_rust.

So I suspect there is some nondeterminism going on.

bgeron · 2024-03-15T17:48:56Z

I tried again and again, even with our Docker cache mounts disabled, and the following delta in rules_rust already seems enough to mitigate the issue for me. Reliably, with this patch our build works, and without it doesn't. Bizarre. This is from a Debian image with some packages installed, users configured, and ssh configured.

(By the way I'm using Bazelisk 1.15.0 and Bazel 7.1.0rc1, not that I imagine that matters.)

--- a/rust/private/repository_utils.bzl
+++ b/rust/private/repository_utils.bzl
@@ -727,6 +727,8 @@ def load_arbitrary_tool(ctx, tool_name, tool_subdirectories, version, iso_date,
 
     sha256 = lookup_tool_sha256(ctx, tool_name, target_triple, version, iso_date, sha256)
 
+    print("Going to download", sha256, "for urls", urls)
+
     for subdirectory in tool_subdirectories:
         # As long as the sha256 value is consistent accross calls here the
         # cost of downloading an artifact is negated as by Bazel's caching.

bgeron · 2024-03-15T18:00:07Z

Upgrading Bazel to 7.1.1rc1 did not work for me, but regressing to 7.0.2 did also 'fix' the issue.

kersson · 2024-03-17T04:23:26Z

We just ran into this as well. I believe this is a duplicate of #2452. Hopefully the fix for that will get in soon.

ericmcbride · 2024-03-19T17:27:49Z

Im seeing this as well, but its working on x86_64-unknown-linux-gnu and its not working in our CI environment aarch64-unknown-linux-gnu On rules rust 0.40.0 and bazel 7.0.0

bazaglia · 2024-04-02T15:07:31Z

Any workarounds? I'm facing the same issue.

bgeron · 2024-04-02T16:03:31Z

Any workarounds? I'm facing the same issue.

Yeah, just kill your lockfile and the resolution should happen again fresh. I usually change the file to contain {} but I'm sure deleting would work just as well.

kersson · 2024-04-03T22:06:18Z

This may be addressed as of 0.41.0. See #2452 (comment).

kolloch · 2024-04-04T15:49:43Z

This may be addressed as of 0.41.0. See #2452 (comment).

I tried it with version 0.41.1 and it did NOT work.

Changed to bzlmod and committed MODULE.bazel.lock on Linux.
CI on Linux passed.
Asked colleague on Mac OS to test it: Error.
Asked colleague on Mac OS to delete lock file and retest: Worked.

The MODULE.bazel.lock includes this:

     "@@rules_rust~//rust:extensions.bzl%rust": {
       "general": {
         "bzlTransitiveDigest": "fuThwhlVrMVy1jd24+jEAspC7bT8RhfezJA1i8Pgw8c=",
@@ -4176,8 +5926,8 @@
             "bzlFile": "@@rules_rust~//rust:repositories.bzl",
             "ruleClassName": "rust_toolchain_tools_repository",
             "attributes": {
-              "exec_triple": "aarch64-apple-darwin",
-              "target_triple": "aarch64-apple-darwin",
+              "exec_triple": "x86_64-unknown-linux-gnu",
+              "target_triple": "x86_64-unknown-linux-gnu",
               "allocator_library": "@rules_rust//ffi/cc/allocator_library",
               "dev_components": false,
               "edition": "2021",

ericmcbride · 2024-04-04T16:03:56Z

This may be addressed as of 0.41.0. See #2452 (comment).

I tried it with version 0.41.1 and it did NOT work.

Changed to bzlmod and committed MODULE.bazel.lock on Linux.
CI on Linux passed.
Asked colleague on Mac OS to test it: Error.
Asked colleague on Mac OS to delete lock file and retest: Worked.

The MODULE.bazel.lock includes this:

     "@@rules_rust~//rust:extensions.bzl%rust": {
       "general": {
         "bzlTransitiveDigest": "fuThwhlVrMVy1jd24+jEAspC7bT8RhfezJA1i8Pgw8c=",
@@ -4176,8 +5926,8 @@
             "bzlFile": "@@rules_rust~//rust:repositories.bzl",
             "ruleClassName": "rust_toolchain_tools_repository",
             "attributes": {
-              "exec_triple": "aarch64-apple-darwin",
-              "target_triple": "aarch64-apple-darwin",
+              "exec_triple": "x86_64-unknown-linux-gnu",
+              "target_triple": "x86_64-unknown-linux-gnu",
               "allocator_library": "@rules_rust//ffi/cc/allocator_library",
               "dev_components": false,
               "edition": "2021",

Did you update bazel to 7.1? Im currently committing the lock to our CI and it works in CI as well on (x86_64 linux / arm64 linux (ci) / arm64 darwin).

ericmcbride · 2024-04-04T17:59:38Z

Okay follow up here, it is not resolved. It looks like it resolved the deps being tied to the triple, but not the toolchains.

kolloch · 2024-04-04T18:01:12Z

We are on bazel version 7.1.1.

I assume that you committed your MODULE.bazel.lock?

Just to be sure, on the branch I deleted the MODULE.bazel.lock and regenerated it with bazel mod deps. I got exactly the same file contents.

For the version that works on Mac OS:

❯ grep -A 17 '"rust_host_tools": {' MODULE.bazel.lock
          "rust_host_tools": {
            "bzlFile": "@@rules_rust~//rust:repositories.bzl",
            "ruleClassName": "rust_toolchain_tools_repository",
            "attributes": {
              "exec_triple": "aarch64-apple-darwin",
              "target_triple": "aarch64-apple-darwin",
              "allocator_library": "@rules_rust//ffi/cc/allocator_library",
              "dev_components": false,
              "edition": "2021",
              "rustfmt_version": "nightly/2024-03-21",
              "sha256s": {},
              "urls": [
                "https://static.rust-lang.org/dist/{}.tar.xz"
              ],
              "version": "1.77.0",
              "iso_date": ""
            }
          },

❯ grep -A 17 '"rust_host_tools": {' MODULE.bazel.lock
          "rust_host_tools": {
            "bzlFile": "@@rules_rust~//rust:repositories.bzl",
            "ruleClassName": "rust_toolchain_tools_repository",
            "attributes": {
              "exec_triple": "x86_64-unknown-linux-gnu",
              "target_triple": "x86_64-unknown-linux-gnu",
              "allocator_library": "@rules_rust//ffi/cc/allocator_library",
              "dev_components": false,
              "edition": "2021",
              "rustfmt_version": "nightly/2024-03-21",
              "sha256s": {},
              "urls": [
                "https://static.rust-lang.org/dist/{}.tar.xz"
              ],
              "version": "1.77.0",
              "iso_date": ""
            }
          },

@ericmcbride I wonder what it looks like in your config.

ericmcbride · 2024-04-04T18:10:51Z

We are on bazel version 7.1.1.

I assume that you committed your MODULE.bazel.lock?

Just to be sure, on the branch I deleted the MODULE.bazel.lock and regenerated it with bazel mod deps. I got exactly the same file contents.

For the version that works on Mac OS:

❯ grep -A 17 '"rust_host_tools": {' MODULE.bazel.lock
          "rust_host_tools": {
            "bzlFile": "@@rules_rust~//rust:repositories.bzl",
            "ruleClassName": "rust_toolchain_tools_repository",
            "attributes": {
              "exec_triple": "aarch64-apple-darwin",
              "target_triple": "aarch64-apple-darwin",
              "allocator_library": "@rules_rust//ffi/cc/allocator_library",
              "dev_components": false,
              "edition": "2021",
              "rustfmt_version": "nightly/2024-03-21",
              "sha256s": {},
              "urls": [
                "https://static.rust-lang.org/dist/{}.tar.xz"
              ],
              "version": "1.77.0",
              "iso_date": ""
            }
          },

❯ grep -A 17 '"rust_host_tools": {' MODULE.bazel.lock
          "rust_host_tools": {
            "bzlFile": "@@rules_rust~//rust:repositories.bzl",
            "ruleClassName": "rust_toolchain_tools_repository",
            "attributes": {
              "exec_triple": "x86_64-unknown-linux-gnu",
              "target_triple": "x86_64-unknown-linux-gnu",
              "allocator_library": "@rules_rust//ffi/cc/allocator_library",
              "dev_components": false,
              "edition": "2021",
              "rustfmt_version": "nightly/2024-03-21",
              "sha256s": {},
              "urls": [
                "https://static.rust-lang.org/dist/{}.tar.xz"
              ],
              "version": "1.77.0",
              "iso_date": ""
            }
          },

@ericmcbride I wonder what it looks like in your config.

Yes this is exactly what mine looks like as well

kersson · 2024-04-04T19:39:18Z

@cameron-martin looks like your work in #2453 (comment) is still needed!

cameron-martin · 2024-04-04T20:17:24Z

Right, I didn't notice that #2575 changed a totally different module extension, so this will still be an issue.

Since the host tools are os and arch specific, previously bazel would cache the resolution of these in the lock file, causing the repo for the wrong OS or arch to be used when moving between machines. Since bazel 7.1.0, module extensions can be marked as reproducible to exclude these from the lock file. However, the `rust` module extension handles fetching of all the other toolchains as well as the host tools, and we don't really want to exclude those too. Therefore the host tools are moved to their own module extension. This means we can no longer match the host toolchain's version, edition, etc with the toolchain registered via `rust.toolchain` by default, and instead default to a fixed version. This can still be overridden separately in the root module. I think this is okay, because the host tools are only used for bootstrapping and I don't think there's much need to have them match. This is tested by now checking in the MODULE.bazel.lock file of the bzlmod example, and running the bzlmod examples on multiple platforms with `--lockfile_mode` set to `error`. Resolves #2452 Resolves #2549

kolloch · 2024-04-16T09:15:33Z

Heyo, just checked with the latest release and it works :) Thank you, @cameron-martin and @illicitonion !

kolloch changed the title ~~Linux toolchain selected on Mac OS (Darwin) with bzlmod~~ Linux cargo binary selected on Mac OS (Darwin) with bzlmod Mar 14, 2024

UebelAndre added bug bzlmod labels Mar 14, 2024

cameron-martin mentioned this issue Apr 4, 2024

Make bazel lock file cross-platform #2453

Merged

illicitonion closed this as completed in #2453 Apr 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Linux cargo binary selected on Mac OS (Darwin) with bzlmod #2549

Linux cargo binary selected on Mac OS (Darwin) with bzlmod #2549

kolloch commented Mar 14, 2024

bgeron commented Mar 15, 2024

bgeron commented Mar 15, 2024

bgeron commented Mar 15, 2024

kersson commented Mar 17, 2024

ericmcbride commented Mar 19, 2024

bazaglia commented Apr 2, 2024

bgeron commented Apr 2, 2024

kersson commented Apr 3, 2024

kolloch commented Apr 4, 2024 •

edited

Loading

ericmcbride commented Apr 4, 2024

ericmcbride commented Apr 4, 2024

kolloch commented Apr 4, 2024

ericmcbride commented Apr 4, 2024

kersson commented Apr 4, 2024

cameron-martin commented Apr 4, 2024

kolloch commented Apr 16, 2024

Linux cargo binary selected on Mac OS (Darwin) with bzlmod #2549

Linux cargo binary selected on Mac OS (Darwin) with bzlmod #2549

Comments

kolloch commented Mar 14, 2024

bgeron commented Mar 15, 2024

bgeron commented Mar 15, 2024

bgeron commented Mar 15, 2024

kersson commented Mar 17, 2024

ericmcbride commented Mar 19, 2024

bazaglia commented Apr 2, 2024

bgeron commented Apr 2, 2024

kersson commented Apr 3, 2024

kolloch commented Apr 4, 2024 • edited Loading

ericmcbride commented Apr 4, 2024

ericmcbride commented Apr 4, 2024

kolloch commented Apr 4, 2024

ericmcbride commented Apr 4, 2024

kersson commented Apr 4, 2024

cameron-martin commented Apr 4, 2024

kolloch commented Apr 16, 2024

kolloch commented Apr 4, 2024 •

edited

Loading