[ci] [python-package] temporarily stop testing against scikit-learn nightlies, load lib_lightgbm earlier #6654
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jameslamb
requested review from
guolinke,
shiyu1994,
jmoralez,
borchero and
StrikerRUS
as code owners
jameslamb
changed the title
StrikerRUS
approved these changes
Sep 15, 2024
|
😫 QEMU aarch64 job is failing with this error again:
For lots of prior context on this: #6509 |
|
Back in #6509 (comment), I found that much of the static TLS was being used by libraries for cloud providers (AWS / Azure / GCP). I just pushed 0229097 switching the CI environments here from conda-forge's That |
|
Switching to I'm able to reproduce it locally (in Docker on my M2 mac, which importantly is also arm64): environment setup in docker similar to CI (click me)docker run \
--rm \
-v $(pwd):/opt/LightGBM \
-w /opt/LightGBM \
-it lightgbm/vsts-agent:manylinux2014_aarch64 \
bash
curl \
-sL \
-o miniforge.sh \
https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh
sh miniforge.sh -b -p "${HOME}/miniconda3"
export PATH="${HOME}/miniconda3/bin:${PATH}"
conda create \
-y \
-c conda-forge \
-n test-env \
--file ./.ci/conda-envs/ci-core.txt \
"python=3.12"
source activate test-envThis fails with the same error seen in CI: source activate test-env
sh build-python.sh bdist_wheel
pip install --no-deps \
./dist/lightgbm-4.5.0.99-py3-none-linux_aarch64.whl
python -c "import lightgbm; print(lightgbm.__version__)"
# OSError: /root/miniconda3/envs/test-env/bin/../lib/libgomp.so.1: cannot allocate memory in static TLS blockUninstalling conda uninstall --yes scikit-learn
python -c "import lightgbm; print(lightgbm.__version__)"
# 4.5.0.99output of 'conda info' (click me)output of 'conda env export' (click me)That doesn't mean anything I suspected that maybe the issue is from mixing I'll try to investigate more tomorrow 😫 |
jameslamb
commented
Sep 21, 2024
python-package/lightgbm/__init__.py
Outdated
|
|
||
| try: | ||
| # this issue seems specific to libgomp, so no need to attempt e.g. libomp or libiomp | ||
| _ = ctypes.CDLL("libgomp.so.1", ctypes.RTLD_GLOBAL) |
There was a problem hiding this comment.
This did seem to fix the issues observed in CI!
But it would probably be simpler to just load lib_lightgbm.{dylib,dll,so} earlier instead. I will test that and push some changes in a few hours. Let's not merge this yet, please.
jameslamb
changed the title
jameslamb
changed the title
jameslamb
changed the title
|
@StrikerRUS I've changed this significantly since your first review, in response to this issue: #6654 (comment) Could you please review again whenever you have time? |
StrikerRUS
approved these changes
Sep 24, 2024
|
Haha yep, thanks very much! I appreciate you reviewing this, I know the evidence supporting this fix is very dense. At least this one feels like it will make the experience permanently better and isn't just a workaround... and I think it's getting better with every release as we learn more. |
|
@jameslamb |
|
ugh, sorry I keep forgetting to do that! Thank you for reminding me. I just removed that version from RTD. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As described in #6653,
lightgbmis currently failing scikit-learn compatibility tests against the latestscikit-learnnightlies (1.6.0dev0).That's being worked on in #6651.
This PR proposes temporarily dropping
scikit-learnfrom the list of projects whose nightlieslightgbmis tested against, to unblock CI here.Update (Sep 22)
While working on this, CI for the Python package started failing in another way:
This proposes a permanent fix for that as well... trying to
dlopen()load_lightgbm as early as possible when runningimport lightgbm.