Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ci] [python-package] replace 'python setup.py' with a shell script #5837

Merged
merged 20 commits into from
May 4, 2023
Merged

[ci] [python-package] replace 'python setup.py' with a shell script #5837

Show file tree
Hide file tree
Changes from 18 commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
8c0d209
[ci] replace setup.py with a shell script
jameslamb Apr 18, 2023
9acbc3d
update CI scripts
jameslamb Apr 18, 2023
258ea5a
update docs
jameslamb Apr 18, 2023
ba5db2e
ensure all expected files are included
jameslamb Apr 18, 2023
25984ec
fix things
jameslamb Apr 18, 2023
8330c81
fix --precompile
jameslamb Apr 18, 2023
4737691
fix windows builds
jameslamb Apr 19, 2023
553310b
more Windows fixes
jameslamb Apr 19, 2023
22d0e59
try to fix Windows builds
jameslamb Apr 19, 2023
4ffa63d
Windows is fun
jameslamb Apr 19, 2023
ddbfe9b
build a wheel when 'install' is called without '--precompile'
jameslamb Apr 19, 2023
874d556
get better logs on MINGW Appveyor build
jameslamb Apr 19, 2023
9ceef1a
work around sh.exe issue on mingw
jameslamb Apr 20, 2023
0984609
quoting is hard
jameslamb Apr 20, 2023
41fb149
re-enable appveyor MSVC
jameslamb Apr 20, 2023
7a647cf
add comment
jameslamb Apr 20, 2023
91227aa
add doccs to build script
jameslamb Apr 20, 2023
c7456da
grammar
jameslamb Apr 20, 2023
245f1eb
Merge branch 'master' into python/use-a-script
jameslamb May 4, 2023
8cbc27c
Merge branch 'master' into python/use-a-script
jameslamb May 4, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .appveyor.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,6 @@ clone_depth: 5

install:
- git submodule update --init --recursive # get `external_libs` folder
- set PATH=%PATH:C:\Program Files\Git\usr\bin;=% # delete sh.exe from PATH (mingw32-make fix)
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was originally put in to work around the following error using mingw32-make.exe + CMake:

CMake Error at C:/Program Files (x86)/CMake/share/cmake-3.16/Modules/CMakeMinGWFindMake.cmake:12 (message):
  sh.exe was found in your PATH, here:
  C:/Program Files/Git/usr/bin/sh.exe
  For MinGW make to work correctly sh.exe must NOT be in your path.
  Run cmake from a shell that does not have sh.exe in your PATH.
  If you want to use a UNIX shell, then use MSYS Makefiles.
Call Stack (most recent call first):
  CMakeLists.txt:39 (project)

If we're going to encourage the use of a shell script to build a development version of the package, that's a problem.

Thankfully, I found that passing a flag through to CMake that says "don't worry, I'm confident having sh.exe on PATH won't be a problem" (as suggested in this Stack Overflow post) works for LightGBM's Python build!

- set PATH=C:\mingw-w64\x86_64-8.1.0-posix-seh-rt_v6-rev0\mingw64\bin;%PATH%
- set PYTHON_VERSION=%CONFIGURATION%
- set CONDA_ENV="test-env"
Expand Down
73 changes: 42 additions & 31 deletions .ci/test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -146,19 +146,21 @@ if [[ $OS_NAME == "macos" ]] && [[ $COMPILER == "clang" ]]; then
fi

if [[ $TASK == "sdist" ]]; then
cd $BUILD_DIRECTORY/python-package && python setup.py sdist || exit -1
sh $BUILD_DIRECTORY/.ci/check_python_dists.sh $BUILD_DIRECTORY/python-package/dist || exit -1
pip install --user $BUILD_DIRECTORY/python-package/dist/lightgbm-$LGB_VER.tar.gz -v || exit -1
cd $BUILD_DIRECTORY && sh ./build-python.sh sdist || exit -1
sh $BUILD_DIRECTORY/.ci/check_python_dists.sh $BUILD_DIRECTORY/dist || exit -1
pip install --user $BUILD_DIRECTORY/dist/lightgbm-$LGB_VER.tar.gz -v || exit -1
if [[ $PRODUCES_ARTIFACTS == "true" ]]; then
cp $BUILD_DIRECTORY/python-package/dist/lightgbm-$LGB_VER.tar.gz $BUILD_ARTIFACTSTAGINGDIRECTORY
cp $BUILD_DIRECTORY/dist/lightgbm-$LGB_VER.tar.gz $BUILD_ARTIFACTSTAGINGDIRECTORY
fi
pytest $BUILD_DIRECTORY/tests/python_package_test || exit -1
exit 0
elif [[ $TASK == "bdist" ]]; then
if [[ $OS_NAME == "macos" ]]; then
cd $BUILD_DIRECTORY/python-package && python setup.py bdist_wheel --plat-name=macosx --python-tag py3 || exit -1
sh $BUILD_DIRECTORY/.ci/check_python_dists.sh $BUILD_DIRECTORY/python-package/dist || exit -1
mv dist/lightgbm-$LGB_VER-py3-none-macosx.whl dist/lightgbm-$LGB_VER-py3-none-macosx_10_15_x86_64.macosx_11_6_x86_64.macosx_12_5_x86_64.whl
cd $BUILD_DIRECTORY && sh ./build-python.sh bdist_wheel || exit -1
sh $BUILD_DIRECTORY/.ci/check_python_dists.sh $BUILD_DIRECTORY/dist || exit -1
mv \
dist/lightgbm-$LGB_VER-py3-none-macosx*.whl \
dist/lightgbm-$LGB_VER-py3-none-macosx_10_15_x86_64.macosx_11_6_x86_64.macosx_12_5_x86_64.whl
if [[ $PRODUCES_ARTIFACTS == "true" ]]; then
cp dist/lightgbm-$LGB_VER-py3-none-macosx*.whl $BUILD_ARTIFACTSTAGINGDIRECTORY
fi
Expand All @@ -169,21 +171,22 @@ elif [[ $TASK == "bdist" ]]; then
else
PLATFORM="manylinux2014_$ARCH"
fi
cd $BUILD_DIRECTORY/python-package && python setup.py bdist_wheel --integrated-opencl --plat-name=$PLATFORM --python-tag py3 || exit -1
sh $BUILD_DIRECTORY/.ci/check_python_dists.sh $BUILD_DIRECTORY/python-package/dist || exit -1
cd $BUILD_DIRECTORY && sh ./build-python.sh bdist_wheel --integrated-opencl || exit -1
mv \
./dist/*.whl \
./dist/lightgbm-$LGB_VER-py3-none-$PLATFORM.whl
sh $BUILD_DIRECTORY/.ci/check_python_dists.sh $BUILD_DIRECTORY/dist || exit -1
if [[ $PRODUCES_ARTIFACTS == "true" ]]; then
cp dist/lightgbm-$LGB_VER-py3-none-$PLATFORM.whl $BUILD_ARTIFACTSTAGINGDIRECTORY
fi
# Make sure we can do both CPU and GPU; see tests/python_package_test/test_dual.py
export LIGHTGBM_TEST_DUAL_CPU_GPU=1
fi
pip install --user $BUILD_DIRECTORY/python-package/dist/*.whl || exit -1
pip install --user $BUILD_DIRECTORY/dist/*.whl || exit -1
pytest $BUILD_DIRECTORY/tests || exit -1
exit 0
fi

mkdir $BUILD_DIRECTORY/build && cd $BUILD_DIRECTORY/build

# temporarily pin pip to versions that support 'pip install --install-option'
# ref: https://github.com/microsoft/LightGBM/issues/5061#issuecomment-1510642287
if [[ $METHOD == "pip" ]]; then
Expand All @@ -194,18 +197,20 @@ if [[ $TASK == "gpu" ]]; then
sed -i'.bak' 's/std::string device_type = "cpu";/std::string device_type = "gpu";/' $BUILD_DIRECTORY/include/LightGBM/config.h
grep -q 'std::string device_type = "gpu"' $BUILD_DIRECTORY/include/LightGBM/config.h || exit -1 # make sure that changes were really done
if [[ $METHOD == "pip" ]]; then
cd $BUILD_DIRECTORY/python-package && python setup.py sdist || exit -1
sh $BUILD_DIRECTORY/.ci/check_python_dists.sh $BUILD_DIRECTORY/python-package/dist || exit -1
pip install --user $BUILD_DIRECTORY/python-package/dist/lightgbm-$LGB_VER.tar.gz -v --install-option=--gpu || exit -1
cd $BUILD_DIRECTORY && sh ./build-python.sh sdist || exit -1
sh $BUILD_DIRECTORY/.ci/check_python_dists.sh $BUILD_DIRECTORY/dist || exit -1
pip install --user $BUILD_DIRECTORY/dist/lightgbm-$LGB_VER.tar.gz -v --install-option=--gpu || exit -1
pytest $BUILD_DIRECTORY/tests/python_package_test || exit -1
exit 0
elif [[ $METHOD == "wheel" ]]; then
cd $BUILD_DIRECTORY/python-package && python setup.py bdist_wheel --gpu || exit -1
sh $BUILD_DIRECTORY/.ci/check_python_dists.sh $BUILD_DIRECTORY/python-package/dist || exit -1
pip install --user $BUILD_DIRECTORY/python-package/dist/lightgbm-$LGB_VER*.whl -v || exit -1
cd $BUILD_DIRECTORY && sh ./build-python.sh bdist_wheel --gpu || exit -1
sh $BUILD_DIRECTORY/.ci/check_python_dists.sh $BUILD_DIRECTORY/dist || exit -1
pip install --user $BUILD_DIRECTORY/dist/lightgbm-$LGB_VER*.whl -v || exit -1
pytest $BUILD_DIRECTORY/tests || exit -1
exit 0
elif [[ $METHOD == "source" ]]; then
mkdir $BUILD_DIRECTORY/build
cd $BUILD_DIRECTORY/build
cmake -DUSE_GPU=ON ..
fi
elif [[ $TASK == "cuda" ]]; then
Expand All @@ -215,43 +220,49 @@ elif [[ $TASK == "cuda" ]]; then
sed -i'.bak' 's/gpu_use_dp = false;/gpu_use_dp = true;/' $BUILD_DIRECTORY/include/LightGBM/config.h
grep -q 'gpu_use_dp = true' $BUILD_DIRECTORY/include/LightGBM/config.h || exit -1 # make sure that changes were really done
if [[ $METHOD == "pip" ]]; then
cd $BUILD_DIRECTORY/python-package && python setup.py sdist || exit -1
sh $BUILD_DIRECTORY/.ci/check_python_dists.sh $BUILD_DIRECTORY/python-package/dist || exit -1
pip install --user $BUILD_DIRECTORY/python-package/dist/lightgbm-$LGB_VER.tar.gz -v --install-option=--cuda || exit -1
cd $BUILD_DIRECTORY && sh ./build-python.sh sdist || exit -1
sh $BUILD_DIRECTORY/.ci/check_python_dists.sh $BUILD_DIRECTORY/dist || exit -1
pip install --user $BUILD_DIRECTORY/dist/lightgbm-$LGB_VER.tar.gz -v --install-option=--cuda || exit -1
pytest $BUILD_DIRECTORY/tests/python_package_test || exit -1
exit 0
elif [[ $METHOD == "wheel" ]]; then
cd $BUILD_DIRECTORY/python-package && python setup.py bdist_wheel --cuda || exit -1
sh $BUILD_DIRECTORY/.ci/check_python_dists.sh $BUILD_DIRECTORY/python-package/dist || exit -1
pip install --user $BUILD_DIRECTORY/python-package/dist/lightgbm-$LGB_VER*.whl -v || exit -1
cd $BUILD_DIRECTORY && sh ./build-python.sh bdist_wheel --cuda || exit -1
sh $BUILD_DIRECTORY/.ci/check_python_dists.sh $BUILD_DIRECTORY/dist || exit -1
pip install --user $BUILD_DIRECTORY/dist/lightgbm-$LGB_VER*.whl -v || exit -1
pytest $BUILD_DIRECTORY/tests || exit -1
exit 0
elif [[ $METHOD == "source" ]]; then
mkdir $BUILD_DIRECTORY/build
cd $BUILD_DIRECTORY/build
cmake -DUSE_CUDA=ON ..
fi
elif [[ $TASK == "mpi" ]]; then
if [[ $METHOD == "pip" ]]; then
cd $BUILD_DIRECTORY/python-package && python setup.py sdist || exit -1
sh $BUILD_DIRECTORY/.ci/check_python_dists.sh $BUILD_DIRECTORY/python-package/dist || exit -1
pip install --user $BUILD_DIRECTORY/python-package/dist/lightgbm-$LGB_VER.tar.gz -v --install-option=--mpi || exit -1
cd $BUILD_DIRECTORY && sh ./build-python.sh sdist || exit -1
sh $BUILD_DIRECTORY/.ci/check_python_dists.sh $BUILD_DIRECTORY/dist || exit -1
pip install --user $BUILD_DIRECTORY/dist/lightgbm-$LGB_VER.tar.gz -v --install-option=--mpi || exit -1
pytest $BUILD_DIRECTORY/tests/python_package_test || exit -1
exit 0
elif [[ $METHOD == "wheel" ]]; then
cd $BUILD_DIRECTORY/python-package && python setup.py bdist_wheel --mpi || exit -1
sh $BUILD_DIRECTORY/.ci/check_python_dists.sh $BUILD_DIRECTORY/python-package/dist || exit -1
pip install --user $BUILD_DIRECTORY/python-package/dist/lightgbm-$LGB_VER*.whl -v || exit -1
cd $BUILD_DIRECTORY && sh ./build-python.sh bdist_wheel --mpi || exit -1
sh $BUILD_DIRECTORY/.ci/check_python_dists.sh $BUILD_DIRECTORY/dist || exit -1
pip install --user $BUILD_DIRECTORY/dist/lightgbm-$LGB_VER*.whl -v || exit -1
pytest $BUILD_DIRECTORY/tests || exit -1
exit 0
elif [[ $METHOD == "source" ]]; then
mkdir $BUILD_DIRECTORY/build
cd $BUILD_DIRECTORY/build
cmake -DUSE_MPI=ON -DUSE_DEBUG=ON ..
fi
else
mkdir $BUILD_DIRECTORY/build
cd $BUILD_DIRECTORY/build
cmake ..
fi

make _lightgbm -j4 || exit -1

cd $BUILD_DIRECTORY/python-package && python setup.py install --precompile --user || exit -1
cd $BUILD_DIRECTORY && sh ./build-python.sh install --precompile --user || exit -1
pytest $BUILD_DIRECTORY/tests || exit -1

if [[ $TASK == "regular" ]]; then
Expand Down
22 changes: 11 additions & 11 deletions .ci/test_windows.ps1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,15 +65,15 @@ if ($env:TASK -ne "bdist") {
if ($env:TASK -eq "regular") {
mkdir $env:BUILD_SOURCESDIRECTORY/build; cd $env:BUILD_SOURCESDIRECTORY/build
cmake -A x64 .. ; cmake --build . --target ALL_BUILD --config Release ; Check-Output $?
cd $env:BUILD_SOURCESDIRECTORY/python-package
python setup.py install --precompile ; Check-Output $?
cd $env:BUILD_SOURCESDIRECTORY
sh $env:BUILD_SOURCESDIRECTORY/build-python.sh install --precompile ; Check-Output $?
cp $env:BUILD_SOURCESDIRECTORY/Release/lib_lightgbm.dll $env:BUILD_ARTIFACTSTAGINGDIRECTORY
cp $env:BUILD_SOURCESDIRECTORY/Release/lightgbm.exe $env:BUILD_ARTIFACTSTAGINGDIRECTORY
}
elseif ($env:TASK -eq "sdist") {
cd $env:BUILD_SOURCESDIRECTORY/python-package
python setup.py sdist --formats gztar ; Check-Output $?
sh $env:BUILD_SOURCESDIRECTORY/.ci/check_python_dists.sh $env:BUILD_SOURCESDIRECTORY/python-package/dist ; Check-Output $?
cd $env:BUILD_SOURCESDIRECTORY
sh $env:BUILD_SOURCESDIRECTORY/build-python.sh sdist ; Check-Output $?
sh $env:BUILD_SOURCESDIRECTORY/.ci/check_python_dists.sh $env:BUILD_SOURCESDIRECTORY/dist ; Check-Output $?
cd dist; pip install @(Get-ChildItem *.gz) -v ; Check-Output $?
}
elseif ($env:TASK -eq "bdist") {
Expand All @@ -87,17 +87,17 @@ elseif ($env:TASK -eq "bdist") {
Get-ItemProperty -Path Registry::HKEY_LOCAL_MACHINE\SOFTWARE\Khronos\OpenCL\Vendors

conda activate $env:CONDA_ENV
cd $env:BUILD_SOURCESDIRECTORY/python-package
python setup.py bdist_wheel --integrated-opencl --plat-name=win-amd64 --python-tag py3 ; Check-Output $?
sh $env:BUILD_SOURCESDIRECTORY/.ci/check_python_dists.sh $env:BUILD_SOURCESDIRECTORY/python-package/dist ; Check-Output $?
cd $env:BUILD_SOURCESDIRECTORY
sh "build-python.sh" bdist_wheel --integrated-opencl ; Check-Output $?
sh $env:BUILD_SOURCESDIRECTORY/.ci/check_python_dists.sh $env:BUILD_SOURCESDIRECTORY/dist ; Check-Output $?
cd dist; pip install --user @(Get-ChildItem *.whl) ; Check-Output $?
cp @(Get-ChildItem *.whl) $env:BUILD_ARTIFACTSTAGINGDIRECTORY
} elseif (($env:APPVEYOR -eq "true") -and ($env:TASK -eq "python")) {
cd $env:BUILD_SOURCESDIRECTORY\python-package
cd $env:BUILD_SOURCESDIRECTORY
if ($env:COMPILER -eq "MINGW") {
python setup.py install --mingw ; Check-Output $?
sh $env:BUILD_SOURCESDIRECTORY/build-python.sh install --mingw ; Check-Output $?
} else {
python setup.py install ; Check-Output $?
sh $env:BUILD_SOURCESDIRECTORY/build-python.sh install ; Check-Output $?
}
}

Expand Down
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -399,6 +399,7 @@ lightgbm.model
/cmake-build-debug/

# Files from local Python install
lightgbm-python/
python-package/LICENSE
python-package/build_cpp/
python-package/compile/
Expand Down
Loading