Run e2e test for release build and fix it by switching back to stable

[NobodyXu]

Plus printing RUSTFLAGS for justfile target build & check

Signed-off-by: Jiahao XU [email protected]

[NobodyXu]

Hmmm, I suspect the error is not caused by #817 since macos and windows CI also failed.
Might need to do a bisect.

[NobodyXu]

It seems like a dns issue of https://crates.io since we have been using trust-dns-resolver v0.22.0 for a long time and the failure is all related to dns failure.

But then I don't understand why the debug build succeeded.

On MacOS it failed with:

 WARN Failed to load root certificate specified by env SSL_CERT_FILE: Expected extension .pem or .der, but found Some(
    "crt",
)

invalid peer certificate contents: invalid peer certificate: UnknownIssuer

On Linux it failed with:

error trying to connect: dns error: request timed out

On Windows, it has been stuck for 1h.

[NobodyXu]

Could it be that #774 is responsible for this?

On MacOS there's a "failed to load certificate error".
Perhaps we should only read from CARGO_BINSTALL_ROOT_CERT and then also document this behavior better in help page.

But then I'm still not sure why Linux and Windows release build failed.

[passcod]

hmm we shouldn't really be checking extension, I believe der has a magic number, we can see that. or just try der then fallback to pem

[NobodyXu]

hmm we shouldn't really be checking extension, I believe der has a magic number, we can see that. or just try der then fallback to pem

I've found one crate file-format that can find out detect file format pem and der.

[passcod]

Sure, or we could check the first two bytes, and if they're 0x30 0x82, it's DER.

[NobodyXu]

Sure, or we could check the first two bytes, and if they're 0x30 0x82, it's DER.

But then we still need to determine whether it's pem or reject it.
Using file-format seems to be most robust solution.

[NobodyXu]

The mysterious error is still there:

macos:

error sending request for url (https://
  │   crates.io/api/v1/crates/b3sum): error trying to connect: invalid peer
  │   certificate contents: invalid peer certificate: UnknownIssuer

linux:

error trying to connect: dns error: request timed out

and then windows is stuck forever.

Since they all happened when downloading from https://crates.io, I suspect this to be an upstream bug in crates_io_api.
I will have to spend more time investigating this.

[NobodyXu]

I am unable to reproduce this on my M1 Macbook air.

@passcod If you are free, can you try reproducing the error in the CI using please:

export CI=true
export JUST_FOR_RELEASE=true
just build && just e2e-tests

[NobodyXu]

My rustc +nightly -vV used for testing yesterday:

rustc 1.69.0-nightly (065852def 2023-02-13)
binary: rustc
commit-hash: 065852def0903296da33a9eaf557f230bcf3a61a
commit-date: 2023-02-13
host: aarch64-apple-darwin
release: 1.69.0-nightly
LLVM version: 15.0.7

I will try to update it and see if it causes the release build to fail.

[NobodyXu]

Upgrading to:

rustc 1.69.0-nightly (5243ea5c2 2023-02-20)
binary: rustc
commit-hash: 5243ea5c29b136137c36bd773e5baa663790e097
commit-date: 2023-02-20
host: aarch64-apple-darwin
release: 1.69.0-nightly
LLVM version: 15.0.7

cause the release build to fail, can confirm this is a regression in rustc nightly now, though I don't know what caused the regression.

I would probably open a channel in zulip.

[passcod]

I know it's a bit slower and doesn't have the buildstd etc but using stable to build would probably save us a chunk of time spent debugging this stuff.

[passcod]

I can't even build the current main locally because of missing deps at the moment, which isn't helping. Will see what I can do later today

[NobodyXu]

I know it's a bit slower and doesn't have the buildstd etc but using stable to build would probably save us a chunk of time spent debugging this stuff.

It's actually miropt level 4 that causes the mis-compilation here and I've disabled it.
buildstd has actually been working quite reliably for us.

[NobodyXu]

Link to zulip discussion: https://rust-lang.zulipchat.com/#narrow/stream/122651-general/topic/rustc.20nightly.20regression.20for.20cargo-binstall

[NobodyXu]

Now blocked on upstream rust-cross/cargo-zigbuild#94

[NobodyXu]

Now blocked on upstream rust-cross/cargo-zigbuild#94

Looks like this is another regression on rustc nightly.
We are really being the beta tester and should get a medal for this.

@passcod I really want to remove all -Z options and use stable, but it seems like split-debuginfo="packed" does not work without build-std (comment wrote by you) so looks like we are stuck on nightly for now.

[NobodyXu]

I opened rust-lang/rust#108392 at upstream for this.

[passcod]

Yeah I think I was mistaken on the build-std x debuginfo stuff, I'll investigate that.

[passcod]

We are really being the beta tester and should get a medal for this.

🏅 !

[messense]

ld.lld: warning: Linking two modules of different target triples: '/tmp/rustcBaYWUF/libzstd_sys-f1eed7ecf7f44df3.rlib(zstd_common.o at 118376)' is 'armv6kz-unknown-linux-musleabihf' whereas 'ld-temp.o' is 'armv7-unknown-linux-gnueabihf'

This warning is also interesting.

[NobodyXu]

To reproduce error in CI:

export CI=true
export JUST_FOR_RELEASE=true
export CARGO_BUILD_TARGET=armv7-unknown-linux-gnueabihf
export GLIBC_VERSION=2.17
export JUST_USE_CARGO_ZIGBUILD=true
just build

[NobodyXu]

Yeah I think I was mistaken on the build-std x debuginfo stuff, I'll investigate that.

Turns out that build-std is not required and I've now trying to switch back to stable.
If it goes well, then we won't have this failure again.