Integration tests failing silently

[mfisher87]

😱

https://github.com/nsidc/earthaccess/actions/runs/9666562376/job/26666365020

I believe part of the problem is poetry run, as I cannot reproduce this without Poetry.

[mfisher87]

I can't reproduce it with Poetry on my local machine either. But I can reproduce with Act:

| =========================== short test summary info ============================
| ERROR tests/integration/test_cloud_download.py - AssertionError: False is not true
| ERROR tests/integration/test_cloud_open.py - AssertionError: False is not true
| ERROR tests/integration/test_onprem_download.py - AssertionError: False is not true
| ERROR tests/integration/test_onprem_open.py - AssertionError: False is not true
| !!!!!!!!!!!!!!!!!!! Interrupted: 4 errors during collection !!!!!!!!!!!!!!!!!!!!
| ========================= 1 skipped, 4 errors in 2.08s =========================
[Integration Tests/integration-tests-3]   ✅  Success - Main Test

[mfisher87]

I can make the step fail by replacing poetry run ... with exit 1. Certainly would have been weird if not.

I can't get the step to fail by manually activating the venv with poetry out of the picture: . .venv/bin/activate && ./scripts/integration-test.sh. I'm very confused.

[mfisher87]

Just grasping at straws at this point. Entirely removing poetry from that workflow and replacing with pip, while it makes the workflow much simpler, does not cause the job to fail when expected.

Running pytest directly in the workflow without the script does not cause the job to fail when expected.

[mfisher87]

I think this is the oldest run on main which fails silently: https://github.com/nsidc/earthaccess/actions/runs/9035463040/job/24830297618

[mfisher87]

Finally! Got it to fail loudly by passing a path argument to pytest, e.g. pytest tests/integration instead of just pytest.

[mfisher87]

@betolink reminded me that we have code that allows up to 10% failure rate:

https://github.com/nsidc/earthaccess/blob/e7cb5bc948a88217b521ef76f7c93c9c384e9e3f/tests/integration/conftest.py

Right now, can print/log a message stating why the failure was tolerated so nobody goes off on a wild goose chase trying to figure this out 😆

Long term, the root problem is that we're testing against a random list of datasets. We should test against a fixed list of datasets that we expect to work (#215).