Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[python] Windows wheels that support GPUs via OpenCL #3403

Merged
merged 70 commits into from
Dec 19, 2020
Merged
Changes from 1 commit
Commits
Show all changes
70 commits
Select commit Hold shift + click to select a range
b1841bd
Add specialized OpenCL/Python package build path
tpboudreau May 27, 2020
27dffcb
Refer to upstream OpenCL repository
tpboudreau Jun 2, 2020
2958084
Reset build job count in setup.py
tpboudreau Jun 3, 2020
eeb1104
Run test for dual CPU/GPU code in CI
itamarst Jun 3, 2020
1494730
Merge remote-tracking branch 'tpboudreau/build_python_package_library…
itamarst Jun 3, 2020
519a5b9
TEMPORARY: refer to OpenCL fork to ensure Linux CI builds succeed
tpboudreau Jun 3, 2020
239dc49
Try switching to OpenCL 2.0
itamarst Jun 3, 2020
d8241b7
Add --opencl-python-pacakge option to wheel
itamarst Jun 3, 2020
e21fbc9
Fix some typos.
itamarst Jun 3, 2020
84b6804
Remove intermediate cmake target
tpboudreau Jun 3, 2020
ec4b3f4
Restrict OpenCL headers to documented API version
tpboudreau Jun 4, 2020
c5263c3
Merge base branch in again
itamarst Jun 5, 2020
ed65913
Refresh the environment after installation
itamarst Jun 5, 2020
3b0b910
Try testing with two processes
itamarst Jun 5, 2020
1fc7878
Use command line definition to activate integrated build
tpboudreau Jun 8, 2020
bf226e3
Try newer OpenCL runtime
itamarst Jun 9, 2020
3e57182
If we hit OpenCL code path that's good enough
itamarst Jun 9, 2020
8e66b4e
Merge branch 'master' into build_python_package_library
tpboudreau Jun 9, 2020
7747d63
Make tests run correctly
itamarst Jun 9, 2020
ed344fb
Correct directory
itamarst Jun 9, 2020
af176a6
Flag reference to unofficial repo with FIXME
tpboudreau Jun 9, 2020
e0c71cb
Merge branch 'master' into build_python_package_library
tpboudreau Jun 30, 2020
81e6773
Merge branch 'master' into build_python_package_library
tpboudreau Aug 10, 2020
4dac9f5
Merge branch 'master' into build_python_package_library (with conflic…
tpboudreau Aug 27, 2020
180c929
TEMPORARY: update private repo tag for dependency
tpboudreau Aug 27, 2020
3616acc
Merge remote-tracking branch 'tpboudreau/build_python_package_library…
itamarst Sep 2, 2020
2bf13b9
Merge remote-tracking branch 'microsoft/master' into windows-gpu-buil…
itamarst Sep 22, 2020
7ce6028
Further post-merge fixes.
itamarst Sep 22, 2020
d9ad5a7
Fix attribute name.
itamarst Sep 23, 2020
69381fd
Reformat with black.
itamarst Sep 23, 2020
67f4cb8
Rewrite config.h before library compilation for Windows wheel
tpboudreau Nov 2, 2020
c9b94a6
Merge branch 'master' into windows-gpu-builds-take-3
tpboudreau Nov 2, 2020
25b1d88
Run dual cpu/gpu test once only for Windows bdist CI
tpboudreau Nov 3, 2020
6df3eb6
Avoid C API tests when building gpu library for wheel
tpboudreau Nov 4, 2020
67f2540
TEMPORARY -- add clinfo tool to vm for debugging
tpboudreau Nov 4, 2020
4f04a94
Increase compilation logging
tpboudreau Nov 4, 2020
1c28c46
Rename OpenCL registry parameter after driver installation
tpboudreau Nov 4, 2020
477fd42
Install legacy OpenCL runtime
tpboudreau Nov 6, 2020
f6d0dee
...
tpboudreau Nov 6, 2020
5fd426b
Remove config.h change, clinfo installation, latest opencl runtime re…
tpboudreau Nov 10, 2020
4d3a724
Add explicit script exit
tpboudreau Nov 10, 2020
7403314
Merge branch 'master' into windows-gpu-builds-take-3
tpboudreau Nov 17, 2020
97489e3
revert to 18.1 intel driver
tpboudreau Nov 17, 2020
cbfec02
Run test_dual.py in two different modes
tpboudreau Nov 19, 2020
78bd0a2
Revert "Run test_dual.py in two different modes"
tpboudreau Nov 24, 2020
9f4c453
Remove unused expected failure test
tpboudreau Nov 24, 2020
011fc9a
Merge from master with conflict resolution
tpboudreau Dec 10, 2020
48f25fa
Install OpenCL in separate powershell script
tpboudreau Dec 11, 2020
a38977f
fix setup.py imports
tpboudreau Dec 11, 2020
b00d907
activate conda env after session update
tpboudreau Dec 11, 2020
7e8c5bc
separate update functions; exit opencl installation on error
tpboudreau Dec 11, 2020
e721d95
nothing; start pipeline
tpboudreau Dec 11, 2020
ae74d0e
download from original source
tpboudreau Dec 11, 2020
cd50a78
specify tls version for installer download
tpboudreau Dec 11, 2020
41915de
eliminate variable from command
tpboudreau Dec 11, 2020
c3031b6
download opencl installer in parts
tpboudreau Dec 11, 2020
e7d7d2f
Merge branch 'master' into windows-gpu-builds-take-3
tpboudreau Dec 11, 2020
bcaceb8
remove duplicate setup.py command class entry
tpboudreau Dec 12, 2020
9396486
download smaller parts
tpboudreau Dec 12, 2020
88b9c49
remove redundant bdist wheel class
tpboudreau Dec 12, 2020
54a2eb4
Import chocolatey modules and use RefreshEnv
tpboudreau Dec 12, 2020
cf52ec0
insert pipeline cache for opencl installer
tpboudreau Dec 13, 2020
817065c
rerun to ensure pipeline cache hit
tpboudreau Dec 13, 2020
7abc286
add comments, test another pipeline cache hit
tpboudreau Dec 13, 2020
a119912
remove cache task; download full installer file from new location
tpboudreau Dec 15, 2020
2f13375
try another installer download timing
tpboudreau Dec 15, 2020
761a48c
reinsert pipeline cache
tpboudreau Dec 15, 2020
822bb34
retain cache task; download full file on cache miss
tpboudreau Dec 15, 2020
f456487
test cache hit again
tpboudreau Dec 15, 2020
29932a7
remove todo
tpboudreau Dec 16, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Remove config.h change, clinfo installation, latest opencl runtime re…
…ference
  • Loading branch information
tpboudreau committed Nov 10, 2020
commit 5fd426bfb2a382bf212b22ebb5b05bda46a5dafd
15 changes: 0 additions & 15 deletions .ci/test_windows.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -50,8 +50,6 @@ elseif ($env:TASK -eq "sdist") {
elseif ($env:TASK -eq "bdist") {
# Install the Intel CPU runtime, so we can run tests against OpenCL
Write-Output "Downloading OpenCL runtime"
#curl -o opencl_runtime_18.1_x64_setup.msi http://registrationcenter-download.intel.com/akdlm/irc_nas/vcp/13794/opencl_runtime_18.1_x64_setup.msi
#$msiarglist = "/i opencl_runtime_18.1_x64_setup.msi /quiet /norestart /log msi.log"
curl -o opencl_runtime_16.1.2_x64_setup.msi http://registrationcenter-download.intel.com/akdlm/irc_nas/12512/opencl_runtime_16.1.2_x64_setup.msi
$msiarglist = "/i opencl_runtime_16.1.2_x64_setup.msi /quiet /norestart /log msi.log"
Write-Output "Installing OpenCL runtime"
Expand All @@ -67,19 +65,6 @@ elseif ($env:TASK -eq "bdist") {
Write-Output "Current OpenCL drivers:"
Get-ItemProperty -Path Registry::HKEY_LOCAL_MACHINE\SOFTWARE\Khronos\OpenCL\Vendors

# TEMPORARY for debugging
Write-Output "Interrogating OpenCL runtime"
curl https://ci.appveyor.com/api/projects/oblomov/clinfo/artifacts/clinfo.exe?job=platform%3a+x64 -o clinfo.exe
.\clinfo.exe
# /TEMPORARY

Set-Variable -Name CONFIG_HEADER -Value "$env:BUILD_SOURCESDIRECTORY/include/LightGBM/config.h"
(Get-Content (Get-Variable CONFIG_HEADER -valueOnly)).replace('std::string device_type = "cpu";', 'std::string device_type = "gpu";') | Set-Content (Get-Variable CONFIG_HEADER -valueOnly)
If (!(Select-String -Path (Get-Variable CONFIG_HEADER -valueOnly) -Pattern 'std::string device_type = "gpu";' -Quiet)) {
Write-Output "Rewriting config.h for GPU device type failed"
Exit -1
}

cd $env:BUILD_SOURCESDIRECTORY/python-package
python setup.py bdist_wheel --integrated-opencl --plat-name=win-amd64 --universal ; Check-Output $?
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Possibly want to delete the --universal?

cd dist; pip install --user @(Get-ChildItem *.whl) ; Check-Output $?
Expand Down