Bindep handling during source dummification breaks embedded builds

[oddlama]

I've been trying to get crane to compile an embedded rust project, but unfortunately encountered some issues that I was unable to solve. Embedded projects often use an alternative linker called flip-link which adds stack overflow safety to embedded devices. The issue is that crane doesn't expect this to happen at some places:

1. flip-link usually searches for a memory.x linker script in the current directory, but even though I added the file to my filters, it is missing in the buildDepsOnly because mkDummySrc filter the source again and discards the memory.x file. My workaround was to add an extraDummyScript to each build* function:

   extraDummyScript = ''
     cp -a ${./memory.x} $out/memory.x
   '';

2. After adding this, the error goes away, but flip-link still fails with the following message (and here I wasn't able to find out what it was missing exactly). It looks like it was linking to the standard library even though we are using thumbv7a-none-eabihf which should cause no_std and no_main to be added to the dummy if I'm reading it correctly:

      Compiling fixed v1.24.0
      Compiling embassy-time v0.1.5 (https://github.com/embassy-rs/embassy?rev=e70c531d3d28565b6926d99d8e977c4df6c13c60#e70c531d)
      Compiling embassy-sync v0.3.0 (https://github.com/embassy-rs/embassy?rev=e70c531d3d28565b6926d99d8e977c4df6c13c60#e70c531d)
      Compiling futures v0.3.28
      Compiling embassy-executor v0.3.0 (https://github.com/embassy-rs/embassy?rev=e70c531d3d28565b6926d99d8e977c4df6c13c60#e70c531d)
      Compiling embassy-embedded-hal v0.1.0 (https://github.com/embassy-rs/embassy?rev=e70c531d3d28565b6926d99d8e977c4df6c13c60#e70c531d)
      Compiling embassy-nrf v0.1.0 (https://github.com/embassy-rs/embassy?rev=e70c531d3d28565b6926d99d8e977c4df6c13c60#e70c531d)
      Compiling nrftest v0.1.0 (/build/source)
   error: linking with `flip-link` failed: exit status: 1
     |
     = note: LC_ALL="C" PATH="/nix/store/vnfnz9f5p5xrjafx48kzjbz39q6y0dv0-rust-default-1.75.0-nightly-2023-10-21/lib/rustlib/x86_64-unknown-linux-gnu/bin:/nix/store/18bs92p6yf6w2wwxhbplgx02y6anq092-gcc-wrapper-12.3.0/bin:/nix/store/h5kvfrjmpw792v8jg7nrzfkffmn0iyy8-gcc-12.3.0/bin:/nix/store/f6in5kb2y5v06zinz1a6xy6cyg67q026-glibc-2.37-8-bin/bin:/nix/store/y9gr7abwxvzcpg5g73vhnx1fpssr5frr-coreutils-9.3/bin:/nix/store/mc6q3cdz5s0p1aj4y586bglsfsnsf2k8-binutils-wrapper-2.40/bin:/nix/store/74y3751gsixaz9797ib0hp7c658sp1y5-binutils-2.40/bin:/nix/store/vnfnz9f5p5xrjafx48kzjbz39q6y0dv0-rust-default-1.75.0-nightly-2023-10-21/bin:/nix/store/0s2bzpml283zkfnhw7qi5qy1zlisffxj-rsync-3.2.7/bin:/nix/store/33vlrbngipikrfyhgyrpjars5qrjacj2-zstd-1.5.5-bin/bin:/nix/store/5p7jmbpmq7clb3whbi6kyp50gmg54l2d-zstd-1.5.5/bin:/nix/store/ay0p9mbw1w3zkvwzx3c94xq7x8jrn9wq-patchelf-0.15.0/bin:/nix/store/wwvmrky0rnnbh795l2vc11ryy0ay20v0-flip-link-0.1.7/bin:/nix/store/y9gr7abwxvzcpg5g73vhnx1fpssr5frr-coreutils-9.3/bin:/nix/store/b6izr8wh0p7dyvh3cyg14wq2rn8d31ik-findutils-4.9.0/bin:/nix/store/q56n7lhjw724i7b33qaqra61p7m7c0cd-diffutils-3.10/bin:/nix/store/x23by79p38ll0js1alifmf3y56vqfs49-gnused-4.9/bin:/nix/store/xafzciap7acqhfx84dvqkp18bg4lrai3-gnugrep-3.11/bin:/nix/store/8kkn44iwdbgqkrj661nr4cjcpmrqqmx8-gawk-5.2.2/bin:/nix/store/89s3w7b4g78989kpzc7sy4phv0nqfira-gnutar-1.35/bin:/nix/store/2a9na7bp4r3290yqqzg503325dwglxyq-gzip-1.13/bin:/nix/store/gxknjk51s7q86llkbzpaqv43kflj9d8j-bzip2-1.0.8-bin/bin:/nix/store/2jp6cv2q4cgh91f5lp57p945rq98ldhr-gnumake-4.4.1/bin:/nix/store/xdqlrixlspkks50m9b0mpvag65m3pf2w-bash-5.2-p15/bin:/nix/store/c15ama0p8jr4mn0943yjk4rpa2hxk7ml-patch-2.7.6/bin:/nix/store/sb3sxnp4g40gfw758a0m4sjm7slvmax9-xz-5.4.4-bin/bin:/nix/store/xfjqspcc9442hi0lm0szv3sw75zswvml-file-5.45/bin" VSLANG="1033" "flip-link" "-flavor" "gnu" "/build/rustcTjhg6m/symbols.o" "/build/source/target/thumbv7em-none-eabihf/release/deps/crane_dummy_nrftest-398e8d8b6be7224a.crane_dummy_nrftest.b8cb6ebd09cc80dc-cgu.0.rcgu.o" "--as-needed" "-L" "/build/source/target/thumbv7em-none-eabihf/release/deps" "-L" "/build/source/target/release/deps" "-L" "/build/source/target/thumbv7em-none-eabihf/release/build/defmt-00212648ee29ee58/out" "-L" "/build/source/target/thumbv7em-none-eabihf/release/build/cortex-m-1d30758924454b84/out" "-L" "/build/source/target/thumbv7em-none-eabihf/release/build/cortex-m-rt-378ac32f9787dd6b/out" "-L" "/build/source/target/thumbv7em-none-eabihf/release/build/nrf52840-pac-aea3b2a5c9f25150/out" "-L" "/nix/store/vnfnz9f5p5xrjafx48kzjbz39q6y0dv0-rust-default-1.75.0-nightly-2023-10-21/lib/rustlib/thumbv7em-none-eabihf/lib" "-Bstatic" "/nix/store/qm8y5ldl85rvn0yyllmqdkpip2d3pmk7-rust-std-1.75.0-nightly-2023-10-21-thumbv7em-none-eabihf/lib/rustlib/thumbv7em-none-eabihf/lib/librustc_std_workspace_core-b5efe57d9f62c089.rlib" "/nix/store/qm8y5ldl85rvn0yyllmqdkpip2d3pmk7-rust-std-1.75.0-nightly-2023-10-21-thumbv7em-none-eabihf/lib/rustlib/thumbv7em-none-eabihf/lib/libcore-4dd3ced510320132.rlib" "/nix/store/qm8y5ldl85rvn0yyllmqdkpip2d3pmk7-rust-std-1.75.0-nightly-2023-10-21-thumbv7em-none-eabihf/lib/rustlib/thumbv7em-none-eabihf/lib/libcompiler_builtins-da2dbdac6c4ed8ee.rlib" "-Bdynamic" "--eh-frame-hdr" "-z" "noexecstack" "-L" "/nix/store/vnfnz9f5p5xrjafx48kzjbz39q6y0dv0-rust-default-1.75.0-nightly-2023-10-21/lib/rustlib/thumbv7em-none-eabihf/lib" "-o" "/build/source/target/thumbv7em-none-eabihf/release/deps/crane_dummy_nrftest-398e8d8b6be7224a" "--gc-sections" "-O1" "-Tlink.x" "-Tdefmt.x"
     = note: rust-lld: error: /build/source/target/thumbv7em-none-eabihf/release/build/cortex-m-rt-378ac32f9787dd6b/out/link.x:49: symbol not found: DefaultHandler_
             rust-lld: error: /build/source/target/thumbv7em-none-eabihf/release/build/cortex-m-rt-378ac32f9787dd6b/out/link.x:49: symbol not found: DefaultHandler_
             rust-lld: error: 
             BUG(cortex-m-rt): the reset vector is missing
             
             rust-lld: error: 
             BUG(cortex-m-rt): the exception vectors are missing
             
             rust-lld: error: 
             ERROR(cortex-m-rt): The interrupt vectors are missing.
             Possible solutions, from most likely to less likely:
             - Link to a svd2rust generated device crate
             - Check that you actually use the device/hal/bsp crate in your code
             - Disable the 'device' feature of cortex-m-rt to build a generic application (a dependency
             may be enabling it)
             - Supply the interrupt handlers yourself. Check the documentation for details.
             
             rust-lld: error: /build/source/target/thumbv7em-none-eabihf/release/build/cortex-m-rt-378ac32f9787dd6b/out/link.x:49: symbol not found: DefaultHandler_
             rust-lld: error: 
             BUG(cortex-m-rt): the reset vector is missing
             
             rust-lld: error: 
             BUG(cortex-m-rt): the exception vectors are missing
             
             rust-lld: error: 
             ERROR(cortex-m-rt): The interrupt vectors are missing.
             Possible solutions, from most likely to less likely:
             - Link to a svd2rust generated device crate
             - Check that you actually use the device/hal/bsp crate in your code
             - Disable the 'device' feature of cortex-m-rt to build a generic application (a dependency
             may be enabling it)
             - Supply the interrupt handlers yourself. Check the documentation for details.
             
             flip-link: the native linker failed to link the program normally; please check your project configuration and linker scripts
             
   error: could not compile `nrftest` (bin "crane-dummy-nrftest") due to previous error
   

3. Maybe the more general problem is that the dummy sources should not use the custom linker script at all. But I'm not really sure of anything here. cargo build does work fine in a devshell, and I can also get crane to compile the project correctly by disabling the dummy dependency building by adding cargoArtifacts = pkgs.runCommand "disable-deps" {} "mkdir -p $out/target";.

---

Have a look at this example project if you want to reproduce this. This is the relevant part of my flake.nix:

rustToolchain = pkgs.pkgsBuildHost.rust-bin.nightly.latest.default.override {
  targets = ["thumbv7em-none-eabihf"];
};

craneLib = (crane.mkLib pkgs).overrideToolchain rustToolchain;
commonArgs = {
  src = ./.;
  lib.cleanSourceWith {
    src = ./.;
    filter = path: type: (craneLib.filterCargoSources path type) || (builtins.baseNameOf path == "memory.x");
  };

  buildInputs =
    [
      pkgs.flip-link
    ]
    ++ lib.optionals pkgs.stdenv.isDarwin [
      # Additional darwin specific inputs can be set here
      pkgs.libiconv
    ];

  # Tell cargo which target we want to build (so it doesn't default to the build system).
  cargoExtraArgs = "--target thumbv7em-none-eabihf";

  # This needs to be disabled because otherwise I get errors about a missing `test` crate.
  doCheck = false;

  # Let the linker access memory.x
  CARGO_TARGET_THUMBV7EM_NONE_EABIHF_RUSTFLAGS = "-C link-arg=--library-path=.";
};

# Build *just* the cargo dependencies, so we can reuse
# all of that work (e.g. via cachix) when running in CI
cargoArtifacts = craneLib.buildDepsOnly (commonArgs // {
  extraDummyScript = ''
    cp -a ${./memory.x} $out/memory.x
  '';
});

# Build the actual package
package = craneLib.buildPackage (commonArgs
  // {
    inherit cargoArtifacts;
  });

[ipetkov]

Hi @oddlama thanks for the report!

Haven't worked with embedded rust projects before so I'm not exactly sure what is missing off the top of my head, but I wonder if we're filtering out something which needs to be present given that overriding the source dummification appears to fix things 🤔

Do you have a(n ideally minimal) flake I can check out and reproduce directly? I tried to look at the example you linked and paste (a fixed up) version of your flake excerpt, but there was just too many other stuff to try to iron out so I couldn't reproduce it in a time-boxed manner 😅

[oddlama]

Sure, I've made a small template project where the issue is reproduced here: https://github.com/oddlama/nrf-template I've included my current workaround below the cargoArtifacts definition.

If you enter the devshell, you should be able to cargo build successfully, but nix build will fail until the the cargoArtifacts are replaced with the workaround.

[ipetkov]

@oddlama thank you for the reproduction!

I see the issue, it has to do with our handling of dummy binary targets which has caused problems in the past... I need to take a closer look at making this dummification smarter (e.g. only stub targets that actually exist as some builds don't expect extra binaries to show up).

In the meantime, you can get unblocked by using

extraDummyScript = ''
  cp -a ${./memory.x} $out/memory.x
  rm -rf $out/src/bin/crane-dummy-*
'';

[mdietrich16]

Alright, sorry for the spam, the problem was nothing with crane, I just forgot that Nix builds in a basic bash shell, so you have to enable the globstar option for workspaces:

extraDummyScript = ''
  cp -a ${./memory.x} $out/memory.x
  (shopt -s globstar; rm -rf $out/**/src/bin/crane-dummy-*)
'';