docs build failure on docs.rs with FIPS enabled

[gretchenfrage]

The recent 0.11.6 release failed to build its docs on docs.rs

The build log:

Build log

# rustc version
rustc 1.84.0-nightly (f7273e004 2024-11-12)# docs.rs version
docsrs 0.6.0 (36c721fb 2024-11-06)# build log
[INFO] running `Command { std: "docker" "create" "-v" "/home/cratesfyi/workspace-builder/builds/quinn-0.11.6/target:/opt/rustwide/target:rw,Z" "-v" "/home/cratesfyi/workspace-builder/builds/quinn-0.11.6/source:/opt/rustwide/workdir:ro,Z" "-v" "/home/cratesfyi/workspace-builder/cargo-home:/opt/rustwide/cargo-home:ro,Z" "-v" "/home/cratesfyi/workspace-builder/rustup-home:/opt/rustwide/rustup-home:ro,Z" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "DOCS_RS=1" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-w" "/opt/rustwide/workdir" "-m" "6442450944" "--cpus" "6" "--user" "1001:1001" "--network" "none" "ghcr.io/rust-lang/crates-build-env/linux@sha256:4a844ea9eb2546a2d2c7022eacef16ef2e8229c7fbb2c7d4d55a9ceca922f72d" "/opt/rustwide/cargo-home/bin/cargo" "+nightly" "rustdoc" "--lib" "-Zrustdoc-map" "--all-features" "--config" "build.rustdocflags=[\"--cfg\", \"docsrs\", \"-Z\", \"unstable-options\", \"--emit=invocation-specific\", \"--resource-suffix\", \"-20241112-1.84.0-nightly-f7273e004\", \"--static-root-path\", \"/-/rustdoc.static/\", \"--cap-lints\", \"warn\", \"--extern-html-root-takes-precedence\"]" "--offline" "-Zunstable-options" "--config=doc.extern-map.registries.crates-io=\"https://docs.rs/{pkg_name}/{version}/x86_64-unknown-linux-gnu\"" "-Zrustdoc-scrape-examples" "-j6" "--target" "x86_64-unknown-linux-gnu", kill_on_drop: false }`
[INFO] [stderr] WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
[INFO] [stdout] 8543c1a566c58e5d3d366e04333e8c4a47ef75c77327cb730946f846e9bf267e
[INFO] running `Command { std: "docker" "start" "-a" "8543c1a566c58e5d3d366e04333e8c4a47ef75c77327cb730946f846e9bf267e", kill_on_drop: false }`
[INFO] [stderr] warning: Rustdoc did not scrape the following examples because they require dev-dependencies: client, connection, insecure_connection, server, single_socket
[INFO] [stderr]     If you want Rustdoc to scrape these examples, then add `doc-scrape-examples = true`
[INFO] [stderr]     to the [[example]] target configuration of at least one example.
[INFO] [stderr] warning: target filter specified, but no targets matched; this is a no-op
[INFO] [stderr]    Compiling aws-lc-fips-sys v0.12.13
[INFO] [stderr] error: failed to run custom build command for `aws-lc-fips-sys v0.12.13`
[INFO] [stderr] 
[INFO] [stderr] Caused by:
[INFO] [stderr]   process didn't exit successfully: `/opt/rustwide/target/debug/build/aws-lc-fips-sys-d91abba67a8c776d/build-script-main` (exit status: 101)
[INFO] [stderr]   --- stdout
[INFO] [stderr]   cargo:rerun-if-env-changed=AWS_LC_FIPS_SYS_NO_PREFIX
[INFO] [stderr]   cargo:rerun-if-env-changed=AWS_LC_FIPS_SYS_PREGENERATING_BINDINGS
[INFO] [stderr]   cargo:rerun-if-env-changed=AWS_LC_FIPS_SYS_EXTERNAL_BINDGEN
[INFO] [stderr]   cargo:rerun-if-env-changed=AWS_LC_FIPS_SYS_NO_ASM
[INFO] [stderr]   cargo:rustc-cfg=x86_64_unknown_linux_gnu
[INFO] [stderr]   cargo:rerun-if-env-changed=AWS_LC_FIPS_SYS_STATIC
[INFO] [stderr]   cargo:rerun-if-env-changed=AWS_LC_FIPS_SYS_STATIC
[INFO] [stderr]   cargo:rerun-if-env-changed=CMAKE_TOOLCHAIN_FILE
[INFO] [stderr]   cargo:rerun-if-env-changed=CMAKE_TOOLCHAIN_FILE_x86_64_unknown_linux_gnu
[INFO] [stderr]   CMAKE_TOOLCHAIN_FILE_x86_64-unknown-linux-gnu = None
[INFO] [stderr]   CMAKE_TOOLCHAIN_FILE_x86_64_unknown_linux_gnu = None
[INFO] [stderr]   HOST_CMAKE_TOOLCHAIN_FILE = None
[INFO] [stderr]   CMAKE_TOOLCHAIN_FILE = None
[INFO] [stderr]   CMAKE_GENERATOR_x86_64-unknown-linux-gnu = None
[INFO] [stderr]   CMAKE_GENERATOR_x86_64_unknown_linux_gnu = None
[INFO] [stderr]   HOST_CMAKE_GENERATOR = None
[INFO] [stderr]   CMAKE_GENERATOR = None
[INFO] [stderr]   CMAKE_PREFIX_PATH_x86_64-unknown-linux-gnu = None
[INFO] [stderr]   CMAKE_PREFIX_PATH_x86_64_unknown_linux_gnu = None
[INFO] [stderr]   HOST_CMAKE_PREFIX_PATH = None
[INFO] [stderr]   CMAKE_PREFIX_PATH = None
[INFO] [stderr]   CMAKE_x86_64-unknown-linux-gnu = None
[INFO] [stderr]   CMAKE_x86_64_unknown_linux_gnu = None
[INFO] [stderr]   HOST_CMAKE = None
[INFO] [stderr]   CMAKE = Some("cmake")
[INFO] [stderr]   running: cd "/opt/rustwide/target/x86_64-unknown-linux-gnu/debug/build/aws-lc-fips-sys-c835585c210695b4/out/build" && CMAKE_PREFIX_PATH="" "cmake" "/opt/rustwide/cargo-home/registry/src/index.crates.io-6f17d22bba15001f/aws-lc-fips-sys-0.12.13" "-DBUILD_SHARED_LIBS=0" "-DCMAKE_BUILD_TYPE=debug" "-DBORINGSSL_PREFIX=aws_lc_fips_0_12_13_" "-DBORINGSSL_PREFIX_HEADERS=/opt/rustwide/cargo-home/registry/src/index.crates.io-6f17d22bba15001f/aws-lc-fips-sys-0.12.13/generated-include" "-DBUILD_TESTING=OFF" "-DBUILD_LIBSSL=OFF" "-DFIPS=1" "-DCMAKE_INSTALL_PREFIX=/opt/rustwide/target/x86_64-unknown-linux-gnu/debug/build/aws-lc-fips-sys-c835585c210695b4/out" "-DCMAKE_C_FLAGS= -ffunction-sections -fdata-sections -fPIC -m64" "-DCMAKE_C_COMPILER=/usr/bin/cc" "-DCMAKE_CXX_FLAGS= -ffunction-sections -fdata-sections -fPIC -m64" "-DCMAKE_CXX_COMPILER=/usr/bin/c++" "-DCMAKE_ASM_FLAGS= -ffunction-sections -fdata-sections -fPIC -m64" "-DCMAKE_ASM_COMPILER=/usr/bin/cc" "--no-warn-unused-cli"
[INFO] [stderr]   Not searching for unused variables given on the command line.
[INFO] [stderr]   -- FIPS build mode configured
[INFO] [stderr]   -- FIPS entropy source method configured: Passive
[INFO] [stderr]   -- stdalign_check.c probe is positive, enabling AWS_LC_STDALIGN_AVAILABLE
[INFO] [stderr]   -- builtin_swap_check.c probe is positive, enabling AWS_LC_BUILTIN_SWAP_SUPPORTED
[INFO] [stderr]   -- linux_u32.c probe is positive, enabling AWS_LC_URANDOM_U32
[INFO] [stderr]   -- Configuring done
[INFO] [stderr]   -- Generating done
[INFO] [stderr]   -- Build files have been written to: /opt/rustwide/target/x86_64-unknown-linux-gnu/debug/build/aws-lc-fips-sys-c835585c210695b4/out/build
[INFO] [stderr]   running: cd "/opt/rustwide/target/x86_64-unknown-linux-gnu/debug/build/aws-lc-fips-sys-c835585c210695b4/out/build" && MAKEFLAGS="-j --jobserver-fds=8,9 --jobserver-auth=8,9" "cmake" "--build" "." "--target" "install" "--config" "Debug"
[INFO] [stderr]   [  0%] Built target global_target
[INFO] [stderr]   [  0%] Built target boringssl_prefix_symbols
[INFO] [stderr]   Consolidate compiler generated dependencies of target fipsmodule
[INFO] [stderr]   Consolidate compiler generated dependencies of target jitterentropy
[INFO] [stderr]   Consolidate compiler generated dependencies of target bcm_c_generated_asm
[INFO] [stderr]   [  0%] Generating err_data.c
[INFO] [stderr]   [  1%] Built target fipsmodule
[INFO] [stderr]   [  3%] Built target jitterentropy
[INFO] [stderr]   [  3%] Built target bcm_c_generated_asm
[INFO] [stderr]   [  4%] Generating delocate
[INFO] [stderr]   [  5%] Generating x86_64-mont.S
[INFO] [stderr]   [  6%] Generating p384/bignum_add_p384.S.S
[INFO] [stderr]   [  6%] Generating vpaes-x86_64.S
[INFO] [stderr]   [  6%] Generating p384/bignum_sub_p384.S.S
[INFO] [stderr]   [  6%] Generating p384/bignum_neg_p384.S.S
[INFO] [stderr]   [  6%] Generating p384/bignum_tomont_p384.S.S
[INFO] [stderr]   [  6%] Generating p384/bignum_deamont_p384.S.S
[INFO] [stderr] 
[INFO] [stderr]   --- stderr
[INFO] [stderr]   failed to initialize build cache at /.cache/go-build: mkdir /.cache: permission denied
[INFO] [stderr]   failed to initialize build cache at /.cache/go-build: mkdir /.cache: permission denied
[INFO] [stderr]   failed to initialize build cache at /.cache/go-build: mkdir /.cache: permission denied
[INFO] [stderr]   failed to initialize build cache at /.cache/go-build: mkdir /.cache: permission denied
[INFO] [stderr]   gmake[2]: *** [aws-lc/crypto/fipsmodule/CMakeFiles/bcm_hashunset.dir/build.make:155: aws-lc/crypto/fipsmodule/delocate] Error 1
[INFO] [stderr]   gmake[2]: *** Waiting for unfinished jobs....
[INFO] [stderr]   gmake[2]: *** [aws-lc/crypto/CMakeFiles/crypto_objects.dir/build.make:98: aws-lc/crypto/err_data.c] Error 1
[INFO] [stderr]   gmake[2]: *** Deleting file 'aws-lc/crypto/err_data.c'
[INFO] [stderr]   gmake[1]: *** [CMakeFiles/Makefile2:279: aws-lc/crypto/CMakeFiles/crypto_objects.dir/all] Error 2
[INFO] [stderr]   gmake[1]: *** Waiting for unfinished jobs....
[INFO] [stderr]   gmake[1]: *** [CMakeFiles/Makefile2:364: aws-lc/crypto/fipsmodule/CMakeFiles/bcm_hashunset.dir/all] Error 2
[INFO] [stderr]   gmake: *** [Makefile:136: all] Error 2
[INFO] [stderr]   thread 'main' panicked at /opt/rustwide/cargo-home/registry/src/index.crates.io-6f17d22bba15001f/cmake-0.1.51/src/lib.rs:1100:5:
[INFO] [stderr] 
[INFO] [stderr]   command did not execute successfully, got: exit status: 2
[INFO] [stderr] 
[INFO] [stderr]   build script failed, must exit now
[INFO] [stderr]   note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
[INFO] running `Command { std: "docker" "inspect" "8543c1a566c58e5d3d366e04333e8c4a47ef75c77327cb730946f846e9bf267e", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "rm" "-f" "8543c1a566c58e5d3d366e04333e8c4a47ef75c77327cb730946f846e9bf267e", kill_on_drop: false }`
[INFO] [stdout] 8543c1a566c58e5d3d366e04333e8c4a47ef75c77327cb730946f846e9bf267e

... Looks to me like what is likely happening is:

• The dependency graph builds quinn-proto -> aws-lc-rs -> aws-lc-fips-sys
• This starts building the aws-lc C++ project
• This perhaps starts trying to build some Go-based code?
• This starts trying to create a build cache at /.cache/go-build, and runs into a permission denied error

Indeed, we can see the same error banner on the docs.rs for aws-lc-fips-sys 0.12.13 (and other versions). Intriguingly, not for aws-lc-rs 1.10.0, or any other version.

[Ralith]

I think this will be fixed by #2040.

[gretchenfrage]

I created an upstream issue, as auto-backlinked above, so it's worth keeping an eye on if that gets fixed, and thus we can return to building it.