investigate building Python ./configure --with-system-ffi
#27
Comments
|
Now hitting this issue on amd64:
$ otool -L /usr/lib/libffi.dylib
/usr/lib/libffi.dylib:
/usr/lib/libffi.dylib (compatibility version 1.0.0, current version 1.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.250.1)(not helpful!) |
|
Reading the And |
|
Instead of fighting too many losing battles with opaque toolchains, I raised the minimum requirements to MacOS 12: 7dd2448 That seems to work. |
|
With 5.0.4 on a M1 I'm still unable to run Xpra: Is there something I'm doing wrong / something else I could try? |
|
@cpatulea someone needs to figure out what to do with |
|
My guess is that we're hitting a variation of https://bugs.python.org/issue44556#msg406626 See also: |
|
Just some notes as I'm looking into this issue. If it's too much noise let me know. Trying to reproduce by just loading
so I think Here's where Indeed, packaged Here's the Apple libffi which has the symbol: https://github.com/apple-oss-distributions/libffi/blob/fbc0bca21c2649677d64bfaa19acdf91d1756dec/darwin/include/ffi.h#L396 [edit] The "system" libffi is not a real file on the filesystem (/usr/lib/libffi.dylib) but loaded from the "built-in dynamic linker cache" (https://ladydebug.com/blog/2021/04/16/big-sur-and-built-in-dynamic-linker-cache/). CommandLineTools has ffi.h, which does have the symbol: I think Homebrew libffi does not have the needed symbol: Here's a libffi issue with very similar issue (just different symbol) suggesting a workaround by setting some I might try to build MacOS binaries (https://github.com/Xpra-org/xpra/blob/master/docs/Build/MacOS.md) to see if I can reproduce the issue with my own build.
Tried to just skip that, now hitting another problem with Working on setting up a GitHub workflow for the Mac OS build, to make it more reproducible and easier to quickly test changes. It's getting stuck on the same issue as before (www.cairographics.org down so can't fetch pixman): https://github.com/cpatulea/gtk-osx-build/actions/runs/7391404577/job/20108152043 |
|
Trying to install pyobjc as recommended by documentation (https://pyobjc.readthedocs.io/en/latest/install.html#installation-using-pip) (using pip):
|
|
Tried a simple hack to disable the packaged but I guess due to the versioned dependency, that won't work. |
|
FYI Homebrew Python also appears to use system libffi: though I don't find Note from CPython upstream, |
|
Managed to set up a GitHub workflow (Mac OS x86_64) that successfully runs Todo list for next time:
|
|
See also https://github.com/orgs/Xpra-org/discussions/4091#discussioncomment-8003248 @cpatulea do you want to make PRs for some of these useful changes?
|
|
Yes eventually I intend to submit PR. Thanks for looking through the changes and suggesting ones to submit. I plan to keep working until I get a full build, then show you everything and see what you want to pick. |
|
OK, so after all the adventures with the Mac OS build in Xpra-org/xpra#4017 I'm back to this, which was to test Before: After: so I think probably Python configure |
|
I nuked the libffi in gtk install root: and now we are good! That "No module named" is a good result, it means the module was imported successfully. |
|
OK, when I looked at this again I thought that maybe we really wanted to ship our own libffi since we can't control which version is going to be present on the system we run on (and the compatibility issues that this seems to create). So instead of using Can we really build something stable without shipping a know good version of |
|
I see your point. I don't know enough about libffi stability to have an opinion on that. Note that Python upstream is removing so perhaps packaging one might stop being an option (or might become difficult to support a different configuration than upstream). For reference, Homebrew pyobjc also links against system libffi: |
|
I'm not sure exactly if this the right issue to describe this, but looking into why it was crashing for me on M2 MBA with MacOS14.2, I decided that since glib had made changes with how it no longer uses fallback for libintl in the post 2.76 version, I would disable the glib-2.78.1 patch. This seems to work for me and I'm not getting crash on launch as I was before. |
|
Cool, now we "just" need to make it work reliably! |
|
I'm going to try this: |
This should solve the problem where the build uses include files from
vanilla libffi, but links against system (Apple) libffi, which causes
missing symbols at runtime:
$ jhbuild run 'python3 -m objc'
Traceback (most recent call last):
File "<frozen runpy>", line 189, in _run_module_as_main
File "<frozen runpy>", line 148, in _get_module_details
File "<frozen runpy>", line 112, in _get_module_details
File "/Users/catalinp/gtk/inst/lib/python3.11/site-packages/objc/__init__.py", line 6, in <module>
from . import _objc
ImportError: dlopen(/Users/catalinp/gtk/inst/lib/python3.11/site-packages/objc/_objc.cpython-311-darwin.so, 0x0002): symbol not found in flat namespace '_ffi_find_closure_for_code_np'
Xpra-org#27 (comment)
|
I had done some builds with https://github.com/cpatulea/gtk-osx-build/actions/runs/8273499988/job/22637378808 which later causes multiple problems (pipenv install fails because typogrify won't install properly; and because of that, jhbuild of librsvg fails). So there's still some issues with |
|
Discussion at pyenv: pyenv/pyenv#2137 |
|
Hmm... at commit cpatulea@55f7d98 I'm observing the following: First build (pyenv) build log: second Python build (jhbuild): and check out this snippet of Python setup.py: https://github.com/python/cpython/blob/fa7a6f23036537567592647d15f043722c7144ad/setup.py#L1382 System libffi (Apple libffi) is already the default on Mac OS. Compare this to the build where I specifically set |
This should solve the problem where the build uses include files from
vanilla libffi, but links against system (Apple) libffi, which causes
missing symbols at runtime:
$ jhbuild run 'python3 -m objc'
Traceback (most recent call last):
File "<frozen runpy>", line 189, in _run_module_as_main
File "<frozen runpy>", line 148, in _get_module_details
File "<frozen runpy>", line 112, in _get_module_details
File "/Users/catalinp/gtk/inst/lib/python3.11/site-packages/objc/__init__.py", line 6, in <module>
from . import _objc
ImportError: dlopen(/Users/catalinp/gtk/inst/lib/python3.11/site-packages/objc/_objc.cpython-311-darwin.so, 0x0002): symbol not found in flat namespace '_ffi_find_closure_for_code_np'
Xpra-org#27 (comment)
Otherwise, we can have a broken build: 2024-03-14T00:28:23.7126080Z /private/var/folders/1k/qq3pcbf12vb6vyblh81736p40000gn/T/python-build.20240314002650.3136/Python-3.11.7/Modules/_ctypes/callbacks.c:438:18: error: call to undeclared function 'ffi_prep_closure'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration] 2024-03-14T00:28:23.7172330Z result = ffi_prep_closure(p->pcl_write, &p->cif, closure_fcn, p); 2024-03-14T00:28:23.7173100Z ^ 2024-03-14T00:28:23.7173690Z 1 error generated. 2024-03-14T00:28:24.1751690Z 2024-03-14T00:28:24.1753070Z Failed to build these modules: 2024-03-14T00:28:24.1780620Z _ctypes which causes problems later (Xpra-org#27 (comment)).
|
It looks like Python is built with system libffi: but objc is built with libffi from source (jhbuild): That doesn't seem right.. I guess when making the app (Xpra) the rpaths get modified, so it 'looks' like _ctypes depends on the bundled libffi: but that's not how it was originally built. |
|
@cpatulea Great bug hunting! I'm not sure I would be able to figure out the Can we just build everything against the system (that still leaves the compatibility problems between versions... but with MacOS 12+, not as bad?) |
|
I tried that, glib then fails to find libffi: https://github.com/cpatulea/gtk-osx-build/actions/runs/8412947646/job/23034441279 I'm planning to spend some trying to to figure out how to get glib to pick up system libffi. |
|
On my local machine, needed to manually remove traces of libffi from ~/gtk/inst/, and then easy to reproduce: Homebrew glib does use system libffi: I think Apple doesn't provide a libffi.pc but Homebrew packages their own (?): But on my machine at least, pkg-config does seem to find it out of the box: ... so why did Meson fail to find it? Ah, because it uses the jhbuild hermetic pkg-config: Looks like we can tell the jhbuild hermetic pkg-config to use Homebrew pc stub files: Seems like there is some precedent of using PKG_CONFIG_PATH to point pkg-config to additional places: gtk-osx-build/jhbuildrc-gtk-osx Line 641 in 80f8b45 |
|
cffi just got updated to 1.17.0: 3fb8628 @cpatulea any idea how I would go about hiding the system's |
|
Sorry for the delay. I'm not sure how to hide system libffi. Based on Homebrew builds with system libffi (#27 (comment)) and Python moved to always build with system libffi (#27 (comment)) I was pursuing the opposite direction: to point all jhbuild packages to build with system libffi. I ran into an obstacle (#27 (comment)) but perhaps with more time, it could be made to work. |
I don't understand how that can be a good idea seeing that this ticket is about binary incompatibilities between libffi versions and some of the packages that link to it. (cffi, pyobjc, etc) It seems that rebuilding everything from scratch (deleting the whole |
|
My understanding is the problem is not "package is linked with system libffi". It's "package is compiled with vendored headers and linked with system libffi". Header vs binary mismatch. I think either of these options should work in theory:
To choose between 1 or 2, I have been looking at what other major projects do. Since both Python and Homebrew seem to have chosen 2, I figured it would be a reasonable choice for Xpra too. |
|
Makes sense! |
Details in jralls#81 (comment)
The text was updated successfully, but these errors were encountered: