How to detect which workspace crates were actually rebuilt?

[klefevre]

Hi,

We have a Rust workspace with multiple crates and want to optimize our CI pipeline to only test, release, and deploy crates that were actually updated.

Current Approach

According to Cargo documentation, I should be able to detect which artifacts were rebuilt using the fresh field. Therefore, using crane, I build the entire workspace:

mkWorkspace = craneLib.buildPackage (commonArgs
  // {
    inherit cargoArtifacts;
    doCheck = false;
    postBuild = ''
      mkdir -p $out/logs
      cp $cargoBuildLog $out/logs/cargo-build.log
    '';
  });

Then I try to detect rebuilt crates with this simple script:

jq < result/logs/cargo-build.log | \
  jq -r 'select(.reason == "compiler-artifact" and .fresh == false) | .package_id

Problem

This approach isn't working because:

1. All artifacts appear as rebuilt in every build
2. Can't differentiate between actual code changes and cache misses

Question

Is there a way to reliably detect which crates were actually rebuilt due to code changes?
Can we leverage nix's caching mechanism to help with this?
Are there alternative approaches to achieve selective testing/deployment in a workspace?

Thanks

[dpc]

IIUC, with what you described the problem is that ever nix build starts either with empty state (target/ content) (if no artifacts were inherited) or with whole existing state (artifacts were inherited). You'd would also need to somehow inherit state from the "previous build".

If you really, really want this to work, I would recommend just using nix develop -c "cargo build ..." , possibly wrapping the exact cargo commands in some script, sacrificing some benefits of the sandboxing of Nix derivations, and retain nix build ... derivations for external consumption etc. Then you should be able to implemented whatever trickery you want (use local caching, build master, then build current HEAD and pare cargo output etc.)

Though generally I think this looks all like fragile and painful to maintain trickery, and I would just throw more hardware and parallelism at it, and run all tests all the time.