Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

remove uses of 'rapidsai' conda channel #210

Draft
wants to merge 10 commits into
base: main
Choose a base branch
from

Conversation

jameslamb
Copy link
Member

@jameslamb jameslamb commented Feb 10, 2025

Recently, as this project crossed the boundary between is 25.01.x and 25.03.x release series, we observed some conda install legate-boost in CI falling back to published packages instead of using those built in CI.

Here I tried a couple approaches to prevent that:

  • explicitly passing ${CPP_CHANNEL}::legate-boost in CI scripts
    • (where CPP_CHANNEL is an absolute filepath to downloaded conda packages)
    • this failed because conda install does not appear to support that
  • using strict channel priority
    • this failed in hard-to-understand ways, I think because candidate legate / cupynumeric packages are spread over 3 different channels (labels are equivalent to channels in some ways)

In the time I was fiddling with this, these builds started working again... I'm not sure why.

Anyway, the changes here should at least partially help. Removing 2 unnecessary channels, which should reduce the risk of the types of conflicts that led to falling back to the wrong package in CI.

  • legate/label/branch-25.01 (no longer necessary as of Update legate to 25.03 #199)
  • rapidsai (was never necessary here, just copied in from other RAPIDS repos)

ci/build_docs.sh Outdated
@@ -38,7 +38,7 @@ rapids-mamba-retry install \
--channel legate/label/branch-25.01 \
--channel legate/label/experimental \
--channel conda-forge \
"legate-boost=${LEGATEBOOST_VERSION}"
"${RAPIDS_LOCAL_CONDA_CHANNEL}::legate-boost<25"
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like this pattern (passing an absolute filepath in the {channel}::package syntax) is not supported.

Multi-download failed. Reason: Transfer finalized, status: 404 [https://conda.anaconda.org/local-conda-packages/noarch/repodata.json] 3085 bytes

(build link)

Tried a couple versions of that locally too, including appending file://.

docker run \
    --rm \
    --env RAPIDS_BUILD_TYPE=nightly \
    --env RAPIDS_NIGHTLY_DATE=2025-02-09 \
    --env RAPIDS_REF_NAME=branch-25.04 \
    --env RAPIDS_REPOSITORY=rapidsai/cudf \
    --env RAPIDS_SHA=428dc188cab5a51c1e15fb90c93a231ad95b7be2 \
    -it rapidsai/ci-conda \
    bash

CPP_CHANNEL=$(rapids-download-conda-from-s3 cpp)

conda install \
    --channel ${CPP_CHANNEL} \
    "file://${CPP_CHANNEL}::libcudf"

Similar result.

UnavailableInvalidChannel: HTTP 404 NOT FOUND for channel cpp_channel <https://conda.anaconda.org/cpp_channel>

The channel is not accessible or is invalid.

I have some other ideas we can try here.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried using strict channel priority and it does look like the builds are using the locally-download packages... but conda chose the CPU-only build on a test machine with a GPU 😭

...
  + legate-boost    25.03.00dev6  cuda12_py310_0_cpu                /tmp/local-conda-packages      345kB

(build link)

Going to try something else here.

@jameslamb jameslamb changed the title WIP: force conda-based CI jobs to use packages built in the same CI run remove uses of 'rapidsai' conda channel Feb 19, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant