Can't build Pillow when using Xcode 12 (and its command line tools) #556
Comments
|
I think this line is the problem: So it seems like one of the dependencies is not building. Perhaps try again with a later version of pillow? You can change that in the recipes folder and try rebuilding. |
|
Thanks @Zen-CODE, I'll give this a try and keep you in touch! |
|
Unfortunately, it fails with exactly the same issue :-( including the "couldn't find library libwebpmux" part... I'll try to see if there's something I can install with brew to fix this issue? |
|
AIUI the "biglink" thing is supposed to create the static library, and it's failing because it doesn't give arguments to ar, and it doesn't do that because it's looking for At least Pillow does seem to build now, but I'm hitting a (totally unrelated I think - probably has to do with Buildozer instead) other issue: Thanks for your help, hope my little edit of |
|
@Zen-CODE On the other hand, could you please try a fully-fresh build on your end? I'm expecting yours to fail just as badly as mine did, it seems very unlikely the filenames would be so different on either end. Possibly having to do with Cython version...? I'm using the very latest release, simply installed with Also FWIW, regarding the "3.8" thing - I don't know about hostpython3 (which seems to be 3.8 regardless) but using a freshly brew-installed 3.7 on OSX instead of 3.8 makes no difference (as one would expect). So as you noted in the other issue, it's not a 3.8 thing - at least not on the OSX side of things (but perhaps changing hostpython3 to 3.7 could help - I honestly very much doubt it). @Zen-CODE Can you also please provide the filesystem layout of your successful build in both cases? It's probably a good idea that I make |
|
Okay, perhaps that's the misunderstanding. I'm building using py3.7 in the venv, but py3.8 as hostpython using the latest checkout. Can't test right now, but try using Py3.7 in you venv? |
|
Yes, as I mentioned that's what I did (the "freshly brew-installed 3.7 on OSX" bit), and it doesn't seem to make a difference - the only thing that seems to help is editing the "biglink" script to refer to the correct paths. Or maybe I'm misunderstanding what you expect me to do? I can certainly give it another go :-) |
|
Puzzling. Fresh checkout. Below uses 3.7.7 in the venv. And all builds fine. Then finaly. |
|
You |
|
Now that's puzzling indeed! That is correct, I'm not actually doing a venv, so I'll literally try to copypaste your approach now and I'll keep you in touch. I'm sure we'll get to the bottom of this, however weird it currently seems! Thanks again for your patience @Zen-CODE, greatly appreciated! |
|
Ah! By reproducing your approach, I might have found a candidate issue! You seem to (rightfully!) pin Cython to 0.29.17 in requirements.txt, while I just pip installed the latest one (resulting in 0.29.21) - that would likely explain a change in build paths, right? Before closing this, I'll finish the venv build, then try a bare-metal build with 0.29.17 (not that it's a good idea to do bare-metal but at least that'll prove Cython version was the issue). Thanks again! EDIT: @Zen-CODE - would it be possible that you attempt a build using latest Cython on your end, to confirm that it's also causing issues there? I suppose eventually we'll need to get it to work, so my biglink edit above might still be relevant? As-is it's not useful though, we need to support both naming conventions apparently (Kivy build failed if using only the biglink above). |
|
Some news! Unfortunately, it doesn't seem to be "only" a Cython version thing - I tried buildozer again in bare metal after downgrading it, but I still get the exact same issue (libpillow.a isn't archived because of biglink failing). Nonetheless, I clearly do not get it when using kivy-ios directly (at least from the venv, maybe I'll try in bare-metal as well), so I think most of this issue can be attributed to buildozer. Sorry for the waste of time! |
|
... Never mind. I somehow didn't try pillow itself and I get the exact same issue in the venv. Logs here: https://gist.github.com/Cheaterman/95625d6d17ab12e579061a148b391ab9 Pip freeze: Filesystem state: |
|
Can't help but notice the only actual In any case, getting the webp thing to work on my machine should indeed help, but that might be highlighting a bigger issue in general with PIL and kivy-ios AIUI? |
|
It seems you are trying too much at once here. Go back to basics. Forget buildozer for now. Do exactly the steps I posted above (and the ones on the kivy-ios site). Does that not build pillow successfully? |
|
Yeah, I just tried from scratch, and I can confirm it doesn't build pillow successfully - same output :-( The full build log is half a million lines, Github gists is having a bit of an issue processing it, hehe. And pastebin freezes when I paste it. >__< |
|
Then please paste:
Thanks |
|
So, here is all of this info: Full build log from scratch in venv: https://gist.github.com/Cheaterman/da46650d6239be223ade0ddc9f325832 As you can see there, I merely did: |
|
For completion: Thanks again for your help @Zen-CODE, really appreciated! :-) |
|
Have you tried building python, kivy and openssl first? |
|
Alrighty, let me give it a try - cleaning pillow first of course I suppose! EDIT: I might also need to install Cython in the venv for Kivy, so I'm doing that as well. I'll paste the full build log when it's complete. :-) EDIT2: It's using Cython from the system instead, I suppose that's fine too. I'm letting it build. :-) |
|
https://gist.github.com/Cheaterman/2f05e256a07219e0b4a89087787b113e Here is the build log (still unsuccessful :-( ) |
|
Check the full log here: https://gist.githubusercontent.com/Cheaterman/a493c1e12a127e0d59000316b064ad76/raw/ed8d479922f875a4052abff2fa0d8ce1b6b01c78/kivy-ios%2520Pillow%2520build%2520log%2520when%2520trying%2520to%2520build%2520Kivy%2520first It's truncated by default on gists (or click the "see full file" link at the top of the page). |
|
Try |
|
I think it won't change the result, given the output: |
|
I still rebuilt pillow and can confirm it doesn't change the result :-( |
|
I see this in your logs: And I notice cython does not appear in your pip freeze. Try installing that and try again? Las If still not, please wipe, clone the repo and enter these instructions exactly. |
|
Thanks again for your help @Zen-CODE, really appreciated - I'll do exactly as you said and post back the results! |
|
So first things first, installing Cython inside the venv didn't help. Lemme clone the repo and apply your instructions to the letter, I'll keep you in touch! |
|
Well, the argument that it's inevitable we will have to upgrade is a strong one :-). So it makes most sense if I upgrade. Just not sure when I will be able to do this. Probably not until next week to be honest.... |
|
I think I did a magic in the meantime, PR might be coming soon if it happens to work on all recipes I'm trying ( |
|
It does build!!! I'll be doing a PR, but in the meantime, if you guys could review this: #!/usr/bin/env python3
import fnmatch
import pathlib
import subprocess
import sys
def list_library_names(libs_dirs):
lib_names = set()
for libs_dir in libs_dirs:
libs_path = pathlib.Path(libs_dir)
for lib_file in libs_path.glob('*.so.libs'):
# Example: pyobjus.cpython-38-darwin.so.libs
lib_name, target, so, libs = lib_file.name.rsplit('.', 4)
lib_names.add(lib_name)
return lib_names
def get_current_build_root(libs_dir):
current_path = pathlib.Path(libs_dir)
while(
not fnmatch.fnmatch(current_path.name, 'lib.*')
and current_path.name != '' # Emergency exit
):
current_path = current_path.parent
build_dir = current_path.parent
assert build_dir.name == 'build' # Sanity check
return build_dir
def main():
target_library, lib_dirs = sys.argv[1], sys.argv[2:]
print(
f'Biglink will create library "{target_library}" '
'from dirs:\n{}'.format("\n".join(lib_dirs))
)
lib_names = list_library_names(lib_dirs)
build_root = get_current_build_root(lib_dirs[0])
temp_dir = next(build_root.glob('temp.*'))
target_object_files = set()
for object_file in temp_dir.rglob('*.o'):
if object_file.stem in lib_names:
target_object_files.add(str(object_file))
lib_names.remove(object_file.stem)
if not lib_names:
# We found everything we need
break
# ... or maybe we didn't
assert not lib_names
command = ['ar', '-q', target_library, *target_object_files]
print(f'Biglink: running "{" ".join(command)}"')
subprocess.run(command)
if __name__ == '__main__':
main()In particular, if you could check that everything still builds (and works!) properly for you afterwards @Zen-CODE ? I myself want to actually build my app against this environment and test it on a live device before claiming victory, hehe. Thanks a lot in advance, and thanks again for all your help @Zen-CODE ! |
|
@Cheaterman Nice work and well done! +1 Look forward to testing it :-) |
|
Okay, so some more news: I created the xcode project and tried to run it, got an error about First, that would seem to generally confirm Xcode 12 is indeed breaking builds for people (not just us!), so that's probably indeed why I am having issues in the first place. Secondly, I need to fix something in the recipe to make it properly build for iOS ; I'm not quite sure what yet, but I'll investigate the various env vars that are mentioned on the SO post and see if something clicks. :-) EDIT: Also FWIW, |
|
as a temporary mesure, seems this should still be possible? https://stackoverflow.com/a/14756127/1209937 (you could check if the older version you want is still available before deleting Xcode from the applications). |
|
Thanks a bunch for that Gab'! That's actually a good workaround to keep under the elbow in case crap starts hitting the fan for some other users :-) fortunately (and you know how rare it is in real scenarios...) I can afford to "do things right" for once instead of having to "do things quick", so I'll keep investigating the linking issues - I'm fairly sure we're very close to a solution, possibly just a few env vars (or Xcode GUI tweaks) away! I'll spend more time investigating the subject this afternoon and evening, I'll keep you guys informed :-) as I said I expect it to be no more than just a little recipe tweak away from completion! |
Cheaterman
changed the title
|
So. After a disastrous waste of time trying to fix the recipe, I finally downgraded as per Gab's suggestion, and interestingly got the same "libpillow.a was build for macOS" error message. This led me to think Xcode tools version might not be the only issue, so I stashed everything and reverted back to kivy-ios master... Which sorta works (I'm now hitting this thing which seems easy enough to fix). There's not that many differences between master and my branch either, mostly the biglink rewrite, also I removed debugging symbols from the pillow recipe ( FWIW, I'm doing EDIT: From all the above, and if I manage to make my biglink rewrite work, I suspect it'll be fine for Xcode 12 as well? Given I'm probably going to spend the day on this again, might as well make it right :-) EDIT2: Apparently not - the most likely issue with my version of biglink is that it doesn't use the |
|
So. For now, my new Some people seem to have switched to native means for QR code decoding, which might indeed be ideal ; the thing is, I want to avoid platform-specific code (especially written/maintained by myself) given this is a cross-platform app I'm distributing (Android/iOS). I'm going to reinstall Xcode 12 and its command line tools and retry, I'll keep you guys in touch. EDIT: The current state of biglink: #!/usr/bin/env python3
import fnmatch
import pathlib
import subprocess
import sys
def lib_name(path):
return path.name.split('.', 1)[0]
def list_library_names(libs_dirs):
lib_names = set()
for libs_dir in libs_dirs:
libs_path = pathlib.Path(libs_dir)
for lib_file in libs_path.glob('*.so.libs'):
# Example: pyobjus.cpython-38-darwin.so.libs
lib_names.add(lib_name(lib_file))
return lib_names
def get_current_build_root(libs_dir):
current_path = pathlib.Path(libs_dir)
while(
not fnmatch.fnmatch(current_path.name, 'lib.*')
and current_path.name != '' # Emergency exit
):
current_path = current_path.parent
build_dir = current_path.parent
assert build_dir.name == 'build' # Sanity check
return build_dir
def main():
target_library, lib_dirs = sys.argv[1], sys.argv[2:]
print(
f'Biglink will create library "{target_library}" '
'from dirs:\n{}'.format("\n".join(lib_dirs))
)
lib_names = list_library_names(lib_dirs)
build_root = get_current_build_root(lib_dirs[0])
lib_dir = next(build_root.glob('lib.*'))
target_object_files = set()
for object_file in lib_dir.rglob('*.so.o'):
if not object_file.with_suffix('.libs').exists():
continue
object_lib = lib_name(object_file)
if object_lib in lib_names:
target_object_files.add(str(object_file))
lib_names.remove(object_lib)
if not lib_names:
# We found everything we need
break
# ... or maybe we didn't
assert not lib_names
command = ['ar', '-q', target_library, *target_object_files]
print(f'Biglink: running "{" ".join(command)}"')
subprocess.run(command)
if __name__ == '__main__':
main() |
|
Okay, so I installed Xcode 12 and tried to run my project in the simulator as-is, and I'm getting this: EDIT: Setting "Build active architecture only" to "yes" in "Build Settings" of my target seems to fix this. |
|
So, there is a lot to process here :-) Thanks for all the effort and feedback. So, you seemed to have got it working? Are there any changes we need to make? |
|
So, I'm not quite clear just yet about everything - I at least believe we should change the template to have the "Build active architecture only" setting set to "yes" in the xcode project by default, and it's possible (I'll be doing more tests) that my whole biglink rewrite is mostly useless (might still PR it if you think it looks good). It has become a bit more urgent for me to get things working quickly, so my current situation is that my dist was built with Xcode 11 tools but I'm making the final package with Xcode 12 - I'll need to rebuild my dist from scratch with Xcode 12 to make sure it actually works :-) |
|
More investigation, more knowledge... The liblink tool fails silently, that's why the .so.o's don't get generated. As I understand it, the "ld: building for iOS Simulator, but linking in object file built for macOS" thing used to be a warning and is now an error. I believe we need to use a "-platform" argument to EDIT: A small recommendation regarding liblink: diff --git a/kivy_ios/tools/liblink b/kivy_ios/tools/liblink
index 05d59d0..a21a427 100755
--- a/kivy_ios/tools/liblink
+++ b/kivy_ios/tools/liblink
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
import sys
import subprocess
@@ -85,4 +85,5 @@ else:
min_version_flag = '-ios_simulator_version_min'
call = [ld, '-r', '-o', output + '.o', min_version_flag, '8.0', '-arch', arch] + objects
print("Linking: {}".format(" ".join(call)))
-subprocess.call(call)
+if subprocess.call(call): # Non-zero exit code
+ raise RuntimeError(f'An error occured while biglinking {output} into {output}.o') |
|
Pity. Unfortunately, I really cannot help here are this time. The solution for the moment is just to use a compatible XCode. Good luck :-) |
|
I agree with you - I have mostly given up at this point. At the very least I might PR these few changes to make the failure more obvious and ease future issue triage, but I think we'll have to wait until we get a clearer idea of how to fix our command invocations for newer toolchains, and ask people to use Xcode 11 for now. Let's leave this issue open, people might come up with ideas :-) |
|
After some research it could be linked on how Xcode 12.x handles the available platforms. Let's pin that, I'm keeping it on my list after some cleanup of the warnings and errors Xcode 12 is throwing. |
|
Thanks for all your work Mirko!
|
|
I just did a PR (#579) that should fix the issue. After a long research I found out that a simple I don't have an use case, so I can't test it on runtime. Can someone give it a try? P.S: We should be able to update Pillow to |
|
@misl6 I'm afraid I'm leaving and won't be able to help until a weeks time. Will retry as soon as I can. |
|
I'm not sure this is entirely fixed, as I'm hitting #540 (comment) - which seems to be Pillow-related? 🙂 thanks in advance! |
|
Probably not a recommended course of action but wanted to share what worked for me. I was trying to After a lot of research for over a week and trying to make the Pillow build work for a few days I simply copied and pasted the Pillow folders ("PIL" and "Pillow-8.1.0.dist-info") from: to: This will probably come to bite me back in the *** at some point in the future but the build worked and doesn't seem to be creating any issues... While not directly relevant, I'll comment #585, as it seems to be the only related open comment on Pillow not building. macOS Big Sur 11.1 |
|
@n-march Thanks for that. But I think anyone experiencing issues should create a new tickets with fresh logs. This conversation is getting too long to make sense of. And I'm suffering from the classic problem of "it works for me"...:-) |
EDIT: Update 07-10-2020: As Xcode 12 seems to be the root cause, the title of the issue has been changed to better reflect this :-)
Hi, as the title suggests, Pillow seems to fail to build on a fresh OS X install.
The error is essentially "lipo can't open input file: libpillow.a (No such file or directory)", which is why Google pointed me to #527, but given #530 and #532 are closed, and @Zen-CODE asked, here I am on a fresh ticket - hopefully it'll help people 😄
Here are the build logs: https://gist.github.com/Cheaterman/6281d5d23923e88072220bc5f1786c1f
Thanks in advance!
The text was updated successfully, but these errors were encountered: