Build script using std can't compile

[u1f408]

I'm trying to use a build.rs file with the cc crate to compile C code as part of a library I'm using in my OS. This doesn't seem to work when used with cargo-xbuild as the std crate doesn't exist in the sysroot:

$ env RUST_TARGET_PATH="/home/user/code/rust/0x52a1/polymorphos/splatform/splatform_i686/" cargo xbuild --target=i686-unknown-polymorphos-elf

   Compiling libc v0.2.42
   Compiling dmm v0.1.0 (file:///home/user/code/rust/0x52a1/polymorphos/dmm)
error[E0463]: can't find crate for `std`=====================>         ] 32/38: libc, dmm                                                                                                                                                                                              
  |
  = note: the `i686-unknown-polymorphos-elf` target may not be installed

error: aborting due to previous error

For more information about this error, try `rustc --explain E0463`.
error: Could not compile `libc`.
warning: build failed, waiting for other jobs to finish...
error: build failed

Is there a way to make the build script compilation use the host's target and default sysroot (so that the build script can actually do it's job), and then use the target passed to cargo-xbuild and the newly built sysroot for the compilation of the crate itself?

[phil-opp]

That's strange. Do you have your code somewhere online so that I can test myself?

[u1f408]

The code for the library using the build script is here. The target specification I'm using is here if that makes any difference.

[phil-opp]

Thanks! I'm able to reproduce it.

cc @lachlansneff I think this is the same error as you had with the new version of raw_cpuid.

[phil-opp]

That's really strange. Without the build dependencies it compiles fine, but with the build script it somehow activates the use_std feature of libc:

rustc 
    --crate-name libc /…/.cargo/registry/src/github.com-[…]/libc-0.2.42/src/lib.rs 
    --crate-type lib --emit=dep-info,link -C debuginfo=2
    --cfg 'feature="use_std"'
    -C metadata=0d3fedfca66a3bb2 -C extra-filename=-0d3fedfca66a3bb2
    --out-dir /home/philipp/Documents/dmm/target/i686-unknown-polymorphos-elf/debug/deps
    --target /…/polymorphos/splatform/splatform_i686/i686-unknown-polymorphos-elf.json
    -L dependency=/…/dmm/target/i686-unknown-polymorphos-elf/debug/deps
    -L dependency=/…/dmm/target/debug/deps --cap-lints allow
    --sysroot /…/dmm/target/sysroot

Minimized cargo tree output:

dmm v0.1.0 (file:///home/philipp/Documents/dmm)
└── cstr_core v0.1.1
    ├── cty v0.1.5
    └── memchr v2.0.1
        └── libc v0.2.42
[build-dependencies]
└── regex v0.2.11
    ├── aho-corasick v0.6.6
    │   └── memchr v2.0.1 (*)
    ├── memchr v2.0.1 (*)
    ├── regex-syntax v0.5.6
    │   └── ucd-util v0.1.1
    ├── thread_local v0.3.5
    │   ├── lazy_static v1.0.2
    │   └── unreachable v1.0.0
    │       └── void v1.0.2
    └── utf8-ranges v1.0.0

I think the problem is that both cstr_core and regex depend on memchr v2.0.1, so cargo does some deduplication and just compiles libc always with the use_std feature (features are additive). I'm not sure about any of this though. I'll try to create a minimal example so that we can be sure.

[phil-opp]

Minimal example:

cargo new bar --lib
cd bar
cargo new foo --lib

In bar/foo/Cargo.toml:

[features]
default = ["use_std"]
use_std = []

In bar/foo/src/lib.rs:

#[cfg_attr(not(feature = "use_std"), no_std)]

pub fn return_42() -> u32 { 42 }

#[cfg(feature = "use_std")]
pub fn print_42() {
    println!("{}", return_42());
}

In bar/Cargo.toml:

[dependencies.foo]
path = "foo"
default-features = false

[build-dependencies.foo]
path = "foo"

In bar/src/lib.rs:

#![no_std]

extern crate foo;

pub fn return_42() -> u32 {
    foo::return_42()
}

Compiling with cargo build --verbose:

   Compiling foo v0.1.0 (file:///…/bar/foo)
     Running `rustc --crate-name foo foo/src/lib.rs --crate-type lib --emit=dep-info,link 
              -C debuginfo=2
              --cfg 'feature="default"' --cfg 'feature="use_std"'
              -C metadata=5cd2c90dafa7c435 -C extra-filename=-5cd2c90dafa7c435
              --out-dir /…/bar/target/debug/deps
              -C incremental=/…/bar/target/debug/incremental
              -L dependency=/…/bar/target/debug/deps`
   Compiling bar v0.1.0 (file:///…/bar)                 
     Running `rustc --crate-name bar src/lib.rs --crate-type lib --emit=dep-info,link
              -C debuginfo=2 -C metadata=0a475a34b1347c78
              -C extra-filename=-0a475a34b1347c78 --out-dir /…/bar/target/debug/deps
              -C incremental=/…/bar/target/debug/incremental
              -L dependency=/…/bar/target/debug/deps
              --extern foo=/…/bar/target/debug/deps/libfoo-5cd2c90dafa7c435.rlib`

We see that --cfg 'feature="default"' --cfg 'feature="use_std"' is passed, i.e. the use_std feature is activated even though default-features = false is used. When we comment out the [build-dependencies.foo] section, foo is built without the use_std feature.

The problem is that compilation for custom targets using xargo/cargo-xbuild fails when the std feature is activated for the custom target.

[phil-opp]

I opened rust-lang/cargo#5730 for this.

[cbeck88]

@phil-opp I run into this today -- I agree that #5730 is an issue in cargo, but I think that is not the "real cause" of this issue.

The real cause is that when using cargo xbuild, the native sysroot, and host triple, should be used for build dependencies and dev dependencies, while for other dependencies they use the custom sysroot.

My experience is that if I try to use cargo xbuild with any crate using cc in a build.rs, the build fails, because eventually it tries to get symbols for libc

     Running `rustc --crate-name libc /root/mobilenode/cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.60/src/lib.rs --color always --crate-type lib --emit=dep-info,metadata,link -C debuginfo=2 --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=142df66e5ee1bd27 -C extra-filename=-142df66e5ee1bd27 --out-dir /root/mobilenode/src/enclave/target/x86_64-unknown-none-sgx/debug/deps --target /root/mobilenode/src/enclave/x86_64-unknown-none-sgx.json -L dependency=/root/mobilenode/src/enclave/target/x86_64-unknown-none-sgx/debug/deps -L dependency=/root/mobilenode/src/enclave/target/debug/deps --cap-lints allow -D warnings -C link-dead-code --sysroot /root/mobilenode/src/enclave/target/sysroot --cfg libc_priv_mod_use --cfg libc_union --cfg libc_const_size_of --cfg libc_align --cfg libc_core_cvoid --cfg libc_packedN`
error[E0412]: cannot find type `c_char` in the crate root
  --> /root/mobilenode/cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.60/src/unix/mod.rs:43:29
   |
43 |         pub gr_name: *mut ::c_char,
   |                             ^^^^^^ help: a type alias with a similar name exists: `c_schar`

error[E0412]: cannot find type `c_char` in the crate root
  --> /root/mobilenode/cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.60/src/unix/mod.rs:44:31
   |
44 |         pub gr_passwd: *mut ::c_char,
   |                               ^^^^^^ help: a type alias with a similar name exists: `c_schar`

error[E0412]: cannot find type `c_char` in the crate root
  --> /root/mobilenode/cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.60/src/unix/mod.rs:46:33
   |
46 |         pub gr_mem: *mut *mut ::c_char,
   |                                 ^^^^^^ help: a type alias with a similar name exists: `c_schar`


...
error: Could not compile `libc`.

Caused by:
  process didn't exit successfully: `rustc --crate-name libc /root/mobilenode/cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.60/src/lib.rs --color always --crate-type lib --emit=dep-info,metadata,link -C debuginfo=2 --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=142df66e5ee1bd27 -C extra-filename=-142df66e5ee1bd27 --out-dir /root/mobilenode/src/enclave/target/x86_64-unknown-none-sgx/debug/deps --target /root/mobilenode/src/enclave/x86_64-unknown-none-sgx.json -L dependency=/root/mobilenode/src/enclave/target/x86_64-unknown-none-sgx/debug/deps -L dependency=/root/mobilenode/src/enclave/target/debug/deps --cap-lints allow -D warnings -C link-dead-code --sysroot /root/mobilenode/src/enclave/target/sysroot --cfg libc_priv_mod_use --cfg libc_union --cfg libc_const_size_of --cfg libc_align --cfg libc_core_cvoid --cfg libc_packedN` (exit code: 1)

Here cargo is applying --sysroot ... flag to the build of libc, which is wrong -- even if features are being unified unexpectedly, this flag should not be used for libc because it is only a build dependency.

[cbeck88]

@phil-opp In your comment above beginning "that's really strange" we have this line:

rustc 
    --crate-name libc /…/.cargo/registry/src/github.com-[…]/libc-0.2.42/src/lib.rs 
    --crate-type lib --emit=dep-info,link -C debuginfo=2
    --cfg 'feature="use_std"'
    -C metadata=0d3fedfca66a3bb2 -C extra-filename=-0d3fedfca66a3bb2
    --out-dir /home/philipp/Documents/dmm/target/i686-unknown-polymorphos-elf/debug/deps
    --target /…/polymorphos/splatform/splatform_i686/i686-unknown-polymorphos-elf.json
    -L dependency=/…/dmm/target/i686-unknown-polymorphos-elf/debug/deps
    -L dependency=/…/dmm/target/debug/deps --cap-lints allow
    --sysroot /…/dmm/target/sysroot

Even if the issue of use_std feature is resolved, it doesn't fix the problem that --sysroot /../dmm/target/sysroot is present. If we can make that flag not be there for libs that are only build dependencies, then it doesn't matter that use_std flag is present.

[cbeck88]

@phil-opp I think you are right -- I guess it means that OP is using libc in a no_std mode in their project but in std mode for the build-script, and cargo is munging the features

[phil-opp]

@cbeck88 I'm not sure how OP is using libc, but builds scripts that use std work fine for me, as long as no feature unification takes place. For example, the bootloader crate, which is compiled for a custom target using cargo-xbuild, uses std in its build script without problems.

[cbeck88]

@phil-opp So you think there's no way that cargo-xbuild can work around this, it can only be fixed upstream in cargo?

[phil-opp]

Yes, I don't think there's a way how cargo-xbuild can work around rust-lang/cargo#5730. Cargo-xbuild is just a wrapper that builds the sysroot before invoking cargo build. Changing the fundamental dependency resolution of cargo is not possible for this project.

[hi-T0day]

Hi, @cbeck88. It seems that the error you mentioned happened again when I build tvm in sgx. How did you solve it?

Running rustc --crate-name libc /root/mobilenode/cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.60/src/lib.rs --color always --crate-type lib --emit=dep-info,metadata,link -C debuginfo=2 --cfg 'feature="default"' --cfg 'feature="std"' -C metadata=142df66e5ee1bd27 -C extra-filename=-142df66e5ee1bd27 --out-dir /root/mobilenode/src/enclave/target/x86_64-unknown-none-sgx/debug/deps --target /root/mobilenode/src/enclave/x86_64-unknown-none-sgx.json -L dependency=/root/mobilenode/src/enclave/target/x86_64-unknown-none-sgx/debug/deps -L dependency=/root/mobilenode/src/enclave/target/debug/deps --cap-lints allow -D warnings -C link-dead-code --sysroot /root/mobilenode/src/enclave/target/sysroot --cfg libc_priv_mod_use --cfg libc_union --cfg libc_const_size_of --cfg libc_align --cfg libc_core_cvoid --cfg libc_packedN
error[E0412]: cannot find type c_char in the crate root

[u1f408]

As of nightly 2020-02-23, Cargo should resolve dependencies correctly in this scenario when executed with -Zfeatures=build_dep. See rust-lang/cargo#5730 (comment) and rust-lang/cargo#7915 for details.

[mogenson]

This thread helped me figure out that adding cargo-make as a dev-dependency adds use_std to the features for memchr, breaking the build.

I'm using nightly 2020-05-08, but using -Zfeatures=build_dep did not seem to help.

[safarir]

This thread helped me figure out that adding cargo-make as a dev-dependency adds use_std to the features for memchr, breaking the build.

I'm using nightly 2020-05-08, but using -Zfeatures=build_dep did not seem to help.

Did you ever find a solution to this problem ? I am stock in the same situation.

I have a dev-dependency (serde_json) that enable the"std" feature of serde. I don't expect this feature to be enabled on non test build but it is ...

[phil-opp]

Try enabling the unstable host_dep feature of cargo, either through a .cargo/config file or by passing -Z host_dep on the command line.