Initial attempt at building OpenCL wheel for Windows #3051

itamarst · 2020-05-05T18:31:27Z

Still untested on actual CI, but this should create a wheel that uses OpenCL, for #2263 based on local testing. I don't quite understand how the wheels get published from this build, but in any case that requires PyPI credentials so you'll have to finish that up.

Tasks:

Build wheel that uses OpenCL.
Make tests pass
Switch to using OpenCL headers from Khronos, instead of the AMD SDK
Maybe make bdist_wheel support GPU install options, rather than using env variables
Add new PyPI project (lightgbm-opencl maybe?)
Make sure release process uploads the new wheel to the new PyPI project

Someone with more access than I have will likely need to do last two items.

msftclas · 2020-05-05T18:31:39Z

All CLA requirements met.

itamarst · 2020-05-05T18:43:30Z

Looks like this iteration won't work at the very least because the --gpu etc. arguments won't work with bdist_wheel.

itamarst · 2020-05-12T14:38:53Z

I have improved setup.py so build options can be set via environment variables, and then the GPU build uses that.

itamarst · 2020-05-12T15:38:02Z

I am not sure why the DLL isn't being found in tests, when I build wheel locally it has the DLL, and logs for bdist_opencl show the DLL being added. Any ideas?

StrikerRUS · 2020-05-12T15:59:35Z

I have improved setup.py so build options can be set via environment variables, and then the GPU build uses that.

I believe that environment variables is not the best way to control installation options. Maybe there are some other methods?.. I remember we already had some proposals for bdist_wheel flags in #1863.

Linking #2263 here as the original issue.

For the general idea of having separate OpenCL wheels for Windows and possible compatibility issues ping @huanzhang12 .

Also I wonder how demanded this wheel will be, given total NVIDIA (read CUDA) domination.

itamarst · 2020-05-12T16:05:04Z

As I understand it there is no specifically CUDA lightgbm at the moment, though? On Nvidia you just use OpenCL installed by CUDA?

itamarst · 2020-05-12T16:07:09Z

I can certainly change it from env variables to bdist_wheel options if that is preferred, although the benefit of env variables is that you can also use this with pip wheel which I don't think you can do with extra args to bdist_wheel.

itamarst · 2020-05-12T16:51:15Z

I wonder if tests are failing because there's no OpenCL runtime installed...

itamarst · 2020-05-13T15:42:19Z

Looks like tests are likely passing now for the OpenCL build, (the docs test failure are unrelated since I haven't touched docs). So remaining questions are:

Will you accept this at all?
If so, are environment variables OK or do you want a larger refactor to how setup.py works?
Will you take on the work of publishing separate PyPI package?

StrikerRUS · 2020-05-13T17:08:28Z

@itamarst

the docs test failure are unrelated since I haven't touched docs

Sorry for the inconvenience! I just re-run it.

If so, are environment variables OK or do you want a larger refactor to how setup.py works?

Please give me some time to read about other possible ways.

I believe we should set device parameter for this wheel to gpu. At least it should be done for the CI test purposes. But for the final wheel it also makes sense, I think.
Refer to GPU *nix tests:

LightGBM/.ci/test.sh

Lines 126 to 127 in 42ebb4e

    
           sed -i'.bak' 's/std::string device_type = "cpu";/std::string device_type = "gpu";/' $BUILD_DIRECTORY/include/LightGBM/config.h 
        
           grep -q 'std::string device_type = "gpu"' $BUILD_DIRECTORY/include/LightGBM/config.h || exit -1  # make sure that changes were really done

itamarst · 2020-05-13T18:19:42Z

Thanks, good point re device. I will be working on this again next week sometime.

itamarst · 2020-05-19T15:27:44Z

@StrikerRUS I've made the OpenCL wheel have GPU device by default. Have you decided how you feel about env variables? My feeling is that a larger setup.py refactor would be useful, but is perhaps orthogonal to this issue.

StrikerRUS · 2020-05-19T20:32:13Z

@itamarst

I've made the OpenCL wheel have GPU device by default.

Thanks! But it seems that now all tests are failing.

Have you decided how you feel about env variables

Sorry, haven't had a chance to look into this yet.

StrikerRUS · 2020-05-24T19:52:58Z

Have you decided how you feel about env variables

Sorry, haven't had a chance to look into this yet.

Seems that we can make bdist_wheel command work with our arguments by inheriting wheel command class.

diff --git a/python-package/setup.py b/python-package/setup.py
index 73f123ba..68834dde 100644
--- a/python-package/setup.py
+++ b/python-package/setup.py
@@ -16,6 +16,60 @@ from setuptools import find_packages, setup
 from setuptools.command.install import install
 from setuptools.command.install_lib import install_lib
 from setuptools.command.sdist import sdist
+from wheel.bdist_wheel import bdist_wheel
+
+
+class CustomBdistWheel(bdist_wheel):
+
+    user_options = bdist_wheel.user_options + [
+        ('mingw', 'm', 'Compile with MinGW'),
+        ('gpu', 'g', 'Compile GPU version'),
+        ('mpi', None, 'Compile MPI version'),
+        ('nomp', None, 'Compile version without OpenMP support'),
+        ('hdfs', 'h', 'Compile HDFS version'),
+        ('bit32', None, 'Compile 32-bit version'),
+        ('precompile', 'p', 'Use precompiled library'),
+        ('boost-root=', None, 'Boost preferred installation prefix'),
+        ('boost-dir=', None, 'Directory with Boost package configuration file'),
+        ('boost-include-dir=', None, 'Directory containing Boost headers'),
+        ('boost-librarydir=', None, 'Preferred Boost library directory'),
+        ('opencl-include-dir=', None, 'OpenCL include directory'),
+        ('opencl-library=', None, 'Path to OpenCL library')
+    ]
+
+    def initialize_options(self):
+        bdist_wheel.initialize_options(self)
+        self.mingw = 0
+        self.gpu = 0
+        self.boost_root = None
+        self.boost_dir = None
+        self.boost_include_dir = None
+        self.boost_librarydir = None
+        self.opencl_include_dir = None
+        self.opencl_library = None
+        self.mpi = 0
+        self.hdfs = 0
+        self.precompile = 0
+        self.nomp = 0
+        self.bit32 = 0
+
+    def run(self):
+        install = self.distribution.get_command_obj('install')
+        install.user_options = self.user_options
+        install.mingw = self.mingw
+        install.gpu = self.gpu
+        install.boost_root = self.boost_root
+        install.boost_dir = self.boost_dir
+        install.boost_include_dir = self.boost_include_dir
+        install.boost_librarydir = self.boost_librarydir
+        install.opencl_include_dir = self.opencl_include_dir
+        install.opencl_library = self.opencl_library
+        install.mpi = self.mpi
+        install.hdfs = self.hdfs
+        install.precompile = self.precompile
+        install.nomp = self.nomp
+        install.bit32 = self.bit32
+        bdist_wheel.run(self)
 
 
 def find_lib():
@@ -98,6 +152,7 @@ def compile_cpp(use_mingw=False, use_gpu=False, use_mpi=False,
     os.chdir(os.path.join(CURRENT_DIR, "build_cpp"))
 
     logger.info("Starting to compile the library.")
+    print(boost_dir)
 
     cmake_cmd = ["cmake", "../compile/"]
     if use_gpu:
@@ -285,6 +340,7 @@ if __name__ == "__main__":
               'install': CustomInstall,
               'install_lib': CustomInstallLib,
               'sdist': CustomSdist,
+              'bdist_wheel': CustomBdistWheel,
           },
           packages=find_packages(),
           include_package_data=True,

Refer to https://github.com/pypa/wheel/blob/master/src/wheel/bdist_wheel.py.

Note that code above is only POC and there is a lot of room for better code, e.g. removing duplicated list of options and setting of properties.

Also, we can make these changes optional to not force users install wheel in case they are installing LightGBM via ordinary setup.py install, like here:
https://github.com/unbit/uwsgi/blob/085015b281ff3d0e81fa52cc9c2fc6c4e3917393/setup.py#L95-L99

Below are some checks:

python setup.py install

running install
INFO:LightGBM:Starting to compile the library.
None
INFO:LightGBM:Starting to compile with MSBuild from existing solution file.
...

python setup.py install --boost-dir=ALERT

running install
INFO:LightGBM:Starting to compile the library.
ALERT
INFO:LightGBM:Starting to compile with MSBuild from existing solution file.
...

python setup.py bdist_wheel --plat-name=win-amd64 --universal

running bdist_wheel
running build
running build_py
running egg_info
writing lightgbm.egg-info\PKG-INFO
writing dependency_links to lightgbm.egg-info\dependency_links.txt
writing requirements to lightgbm.egg-info\requires.txt
writing top-level names to lightgbm.egg-info\top_level.txt
reading manifest file 'lightgbm.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
no previously-included directories found matching 'build'
warning: no files found matching '*.txt'
warning: no files found matching '*.so' under directory 'lightgbm'
warning: no files found matching '*.so' under directory 'compile'
warning: no files found matching '*.dll' under directory 'compile\Release'
warning: no files found matching '*' under directory 'compile\compute'
warning: no files found matching '*.dll' under directory 'compile\windows\x64\DLL'
warning: no previously-included files matching '*.py[co]' found anywhere in distribution
writing manifest file 'lightgbm.egg-info\SOURCES.txt'
installing to build\bdist.win-amd64\wheel
running install
INFO:LightGBM:Starting to compile the library.
None
INFO:LightGBM:Starting to compile with MSBuild from existing solution file.
...

python setup.py bdist_wheel --plat-name=win-amd64 --universal --boost-dir=ALERT

running bdist_wheel
running build
running build_py
running egg_info
writing lightgbm.egg-info\PKG-INFO
writing dependency_links to lightgbm.egg-info\dependency_links.txt
writing requirements to lightgbm.egg-info\requires.txt
writing top-level names to lightgbm.egg-info\top_level.txt
reading manifest file 'lightgbm.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
no previously-included directories found matching 'build'
warning: no files found matching '*.txt'
warning: no files found matching '*.so' under directory 'lightgbm'
warning: no files found matching '*.so' under directory 'compile'
warning: no files found matching '*.dll' under directory 'compile\Release'
warning: no files found matching '*' under directory 'compile\compute'
warning: no files found matching '*.dll' under directory 'compile\windows\x64\DLL'
warning: no previously-included files matching '*.py[co]' found anywhere in distribution
writing manifest file 'lightgbm.egg-info\SOURCES.txt'
installing to build\bdist.win-amd64\wheel
running install
INFO:LightGBM:Starting to compile the library.
ALERT
INFO:LightGBM:Starting to compile with MSBuild from existing solution file.
...

itamarst · 2020-06-09T13:55:48Z

Replaced by #3143.

github-actions · 2023-08-24T13:06:23Z

This pull request has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.

itamarst closed this May 5, 2020

itamarst reopened this May 5, 2020

itamarst changed the title ~~Initial attempt at building OpenCL wheel.~~ Initial attempt at building OpenCL wheel for Windows May 5, 2020

Initial attempt at building OpenCL wheel.

6e3d9c4

jameslamb added the feature label May 11, 2020

itamarst marked this pull request as ready for review May 12, 2020 15:37

itamarst requested review from chivee, henry0312, jameslamb, Laurae2, StrikerRUS and wxchan as code owners May 12, 2020 15:37

StrikerRUS requested a review from huanzhang12 May 12, 2020 15:59

itamarst added 2 commits May 12, 2020 14:31

Support setting build options with environment variables.

beecd21

Try to use correct env variable.

cd09472

itamarst added 2 commits May 13, 2020 15:21

Install an OpenCL runtime.

2a918ea

Set default device to GPU in OpenCL wheel

38fc9aa

Show all logs

49a7353

Check exit code of MSI install

8feb65d

itamarst added 6 commits May 26, 2020 11:49

Maybe refreshing environment will detect newly installed OpenCL

7745b57

Temporarily disable non-gpu tasks, for faster feedback loop.

9045599

Switch to Khronos headers for OpenCL

0ff411b

Merge branch 'master' into windows-gpu-builds

c963350

Use Khronos ICD as source of opencl.lib, and fix header location

0674bda

Try to fix msbuild path

87ad11c

itamarst closed this Jun 9, 2020

github-actions bot locked as resolved and limited conversation to collaborators Aug 24, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Initial attempt at building OpenCL wheel for Windows #3051

Initial attempt at building OpenCL wheel for Windows #3051

itamarst commented May 5, 2020 •

edited

Loading

msftclas commented May 5, 2020 •

edited

Loading

itamarst commented May 5, 2020

itamarst commented May 12, 2020

itamarst commented May 12, 2020

StrikerRUS commented May 12, 2020

itamarst commented May 12, 2020

itamarst commented May 12, 2020

itamarst commented May 12, 2020

itamarst commented May 13, 2020

StrikerRUS commented May 13, 2020

itamarst commented May 13, 2020

itamarst commented May 19, 2020

StrikerRUS commented May 19, 2020

StrikerRUS commented May 24, 2020

itamarst commented Jun 9, 2020

github-actions bot commented Aug 24, 2023

Initial attempt at building OpenCL wheel for Windows #3051

Initial attempt at building OpenCL wheel for Windows #3051

Conversation

itamarst commented May 5, 2020 • edited Loading

msftclas commented May 5, 2020 • edited Loading

itamarst commented May 5, 2020

itamarst commented May 12, 2020

itamarst commented May 12, 2020

StrikerRUS commented May 12, 2020

itamarst commented May 12, 2020

itamarst commented May 12, 2020

itamarst commented May 12, 2020

itamarst commented May 13, 2020

StrikerRUS commented May 13, 2020

itamarst commented May 13, 2020

itamarst commented May 19, 2020

StrikerRUS commented May 19, 2020

StrikerRUS commented May 24, 2020

itamarst commented Jun 9, 2020

github-actions bot commented Aug 24, 2023

itamarst commented May 5, 2020 •

edited

Loading

msftclas commented May 5, 2020 •

edited

Loading