PYD libraries can't be loaded in 32bit Python on Windows

[mstarecek]

Hello,
In version 0.3.0 I'm getting the following error when starting 'pack-manager' (or more generally 'pyocd')

OSError: cannot load library '...venv_39_32\lib\site-packages\cmsis_pack_manager_native__lib.cp39-win32.pyd': error 0xc1

This error occurs on Python 3.6 up to 3.9, but only in the 32bit versions.
It would seem that the 32bit versions of wheels contain 64bit libraries because on 64bit Python everything works just fine (I've tried Python 3.6 up to 3.9)

Best Regards,
Michal

[flit]

Hi @mstarecek! Sorry we missed this issue until now. Thanks very much for the report. I believe I only tested with 64-bit Windows.

[flit]

Fwiw, you can probably work around this by forcing a full reinstall and building from source.

pip install --force-reinstall --no-binary ':all:' cmsis-pack-manager

You'll need Rust installed, but that's easy to set up with rustup.

[flit]

@mstarecek Could you test to see if CPM v0.40 that was just released fixes the problem with Win32? We upgraded to the latest cibuildwheel version, so I'm hoping they fixed whatever the issue is. (Sorry, I haven't had time to investigate the root cause.)

[flit]

CPM v0.4.0 build with cibuildwheel 2.x did not fix this issue.

What's weird is that the cmsis_pack_manager-0.4.0-cp310-cp310-win32.whl wheel contains all of the files shown below. All of the .pyd shared libraries appear to be x86-64 instead of i386, regardless the name. (Checked with dumpbin.)

total 28M                                                                   
-rw-r--r-- 1 creed 197609  14K Nov 14 12:30 __init__.py                     
drwxr-xr-x 1 creed 197609    0 Nov 14 12:30 __pycache__/                    
-rw-r--r-- 1 creed 197609  207 Nov 14 12:30 _native.py                      
-rw-r--r-- 1 creed 197609 2.3K Nov 14 12:30 _native__ffi.py                 
-rwxr-xr-x 1 creed 197609 3.1M Nov 14 12:30 _native__lib.cp310-win32.pyd*   
-rwxr-xr-x 1 creed 197609 3.1M Nov 14 12:30 _native__lib.cp36-win32.pyd*    
-rwxr-xr-x 1 creed 197609 3.1M Nov 14 12:30 _native__lib.cp36-win_amd64.pyd*
-rwxr-xr-x 1 creed 197609 3.1M Nov 14 12:30 _native__lib.cp37-win32.pyd*    
-rwxr-xr-x 1 creed 197609 3.1M Nov 14 12:30 _native__lib.cp37-win_amd64.pyd*
-rwxr-xr-x 1 creed 197609 3.1M Nov 14 12:30 _native__lib.cp38-win32.pyd*    
-rwxr-xr-x 1 creed 197609 3.1M Nov 14 12:30 _native__lib.cp38-win_amd64.pyd*
-rwxr-xr-x 1 creed 197609 3.1M Nov 14 12:30 _native__lib.cp39-win32.pyd*    
-rwxr-xr-x 1 creed 197609 3.1M Nov 14 12:30 _native__lib.cp39-win_amd64.pyd*
-rw-r--r-- 1 creed 197609  147 Nov 14 12:30 _version.py                     
-rw-r--r-- 1 creed 197609 8.8K Nov 14 12:30 pack_manager.py                 

For a temporary workaround so pyocd will at least work, I'll add an exception handler to catch OSError when pyocd tries to load CPM. Pyocd will run, but the pack subcommand won't work.

[Gargy007]

Hello Chris,
I'm just want to ask you to check this issue, because it's being more and more blocking to us to move to new version of PyOCD. I checked the latest PyOCD on Python 32 bits (Widows) and the issue is still here (added exception handler probably doesn't work).

I tried to check the package compiler if I'm able to fix it on my own, but without investigating a lot of time, I'm not.

So we you find some time to check it, you make so guys in NXP happy :-)

Petr

[saper]

Do we even tell CI to build 32-bit wheels? I see them being built - despite -win32 flag being set.

https://stackoverflow.com/a/71754959 seems to suggest the need to additionally specify

      matrix:
        toolchain: [stable]
        os: [windows-latest]
        target: [x86_64-pc-windows-msvc, i686-pc-windows-msvc]

https://rust-lang.github.io/rustup/installation/windows.html?highlight=target#windows notes

By default rustup on Windows configures Rust to target the MSVC ABI, that is a target triple of either i686-pc-windows-msvc or x86_64-pc-windows-msvc depending on the CPU architecture of the host Windows OS. The toolchains that rustup chooses to install, unless told otherwise through the toolchain specification, will be compiled to run on that target triple host and will target that triple by default.

[flit]

@saper Thanks for the suggestion! 🙏🏽 I'll give it a try as soon as I have a chance.

[mstarecek]

Hello @flit, it has been a while :)
I was trying to create the 32b windows build on my end, however, I'm struggling to get the cibuildwheel working locally. I was hoping, that by forking the repo I'll be able to run the wheel build via the workflow_dispatch event on my end. So I can test things before submitting a pull request.
However, on my end, the Actions page is empty :( Do you have any ideas on how to proceed? I'm afraid building everything locally via bdist_wheel may yield different results than the CI build

[sishuiruhuo]

Hello @flit,
I have the same problem.I use Python(3.10.4 32-bit) on Windows and the problem persists(CPM v0.4.0).
OSError: cannot load library '..\site-packages\cmsis_pack_manager_native__lib.cp310-win32.pyd': error 0xc1.

[mathias-arm]

Could someone check if the wheels build with maturin (see #195) work on Windows 32bit https://github.com/pyocd/cmsis-pack-manager/actions/runs/3509644772

[mstarecek]

Hello @mathias-arm,
I've tried Python 3.7 up to 3.11, both 32b and 64b versions, and... EVERYTHING WORKS!!! 🥳
My test consists of the following steps: update pip, install cmsis-pack-manager from the wheel, install pyocd, and run pyocd list (since I don't know how to explicitly test cmsis-pack-manager 😅 I tried the case that started this whole thing)
Many thanks for your effort!

[mathias-arm]

Hello @mathias-arm, I've tried Python 3.7 up to 3.11, both 32b and 64b versions, and... EVERYTHING WORKS!!!

Thanks a lot for your tests!