Python import from REPL results in a runtime error when linking a library with -l

[zachgrayio]

Again, this is brought over from the mailing list - attempting to edit the original message down to something relevant to this specific issue.

import Python fails when I include the -lTFExample flag to the REPL invocation to link to the built TFExample library so I can access third-party packages.

  1> import Python
error: Couldn't lookup symbols:
  _swift_FORCE_LOAD_$_swiftPython

If I don't include that flag, I'll get runtime errors when trying to interact with imported third-party libraries which is expected because the symbols will be missing.

[dan-zheng]

Could you try adding -lswiftPython to your command?

If that solves the problem, I'll look into making the REPL link libswiftPython.so and libswiftTensorFlow.so by default. Currently, I believe it only links libswiftCore.so.

[zachgrayio]

awesome, I figured this was likely the issue here and probably related to #5 but didn't have time to find the lib names.

This works: swift -I/usr/lib/swift/clang/include -I/usr/src/TFExample/.build/debug -L/usr/src/TFExample/.build/debug -lswiftPython -lTFExample

important to note however that the order in which the flags are supplied matters (the order above is correct, reverse the -l and it fails) which I guess I didn't expect, but maybe that's just due to how rusty I am with the toolchain right now.

sounds like getting this linking out of the way up front is the solution. also, I haven't built from source yet so I'm not sure on what's captured in current docs, but probably want to note all this for people who roll their own.

[dan-zheng]

Thanks for confirming -lswiftPython works! The solution should just be a matter of adding the flag by default, then.

Can you share what test program you're running? I'd like to replicate the "link flag ordering" issue.

[zachgrayio]

Sure, it's the example here but I'll paste it below for ease.

Starting the container with:

docker run --rm --security-opt seccomp:unconfined -itv ${PWD}:/usr/src \
    zachgray/swift-tensorflow:4.2 \
    swift \
    -I/usr/lib/swift/clang/include \
    -I/usr/src/TFExample/.build/debug \
    -L/usr/src/TFExample/.build/debug \
    -lswiftPython \
    -lswiftTensorFlow \
    -lTFExample

and I can run the following:

  1> import RxSwift
  2> import Python
  3> import TensorFlow
  4> var x = Tensor([[1, 2], [3, 4]])
x: TensorFlow.Tensor<Double> = [[1.0, 2.0], [3.0, 4.0]]
  5> _ = Observable.from([1,2]).subscribe(onNext: { print($0) })
1
2
  6> var x: PyValue = [1, "hello", 3.14]
x: Python.PyValue = [1, 'hello', 3.14]
  7> :exit

However, if we change the above run command to:

docker run --rm --security-opt seccomp:unconfined -itv ${PWD}:/usr/src \
    zachgray/swift-tensorflow:4.2 \
    swift \
    -I/usr/lib/swift/clang/include \
    -I/usr/src/TFExample/.build/debug \
    -L/usr/src/TFExample/.build/debug \
    -lTFExample \
    -lswiftPython \
    -lswiftTensorFlow

the result is as follows:

  1> import RxSwift
  2> import Python
error: Couldn't lookup symbols:
  _swift_FORCE_LOAD_$_swiftPython

and also, the test case and results for #4 are also applicable again with the flags in the wrong order.

[dan-zheng]

Thanks for sharing!

That's weird, I don't get the _swift_FORCE_LOAD_$_swiftPython error:

$ docker run --rm --security-opt seccomp:unconfined -itv ${PWD}:/usr/src \
    zachgray/swift-tensorflow:4.2 \
    swift \
    -I/usr/lib/swift/clang/include \
    -I/usr/src/TFExample/.build/debug \
    -L/usr/src/TFExample/.build/debug \
    -lTFExample \
    -lswiftPython \
    -lswiftTensorFlow
Welcome to Swift version 4.2-dev (LLVM 04bdb56f3d, Clang b44dbbdf44). Type :help for assistance.
  1> import RxSwift
  2> import Python
  3> PyValue([1, "hello", 3.14])
$R0: Python.PyValue = [1, 'hello', 3.14]

From my observations, there are some other factors affecting the REPL and linking. Sometimes, commands that don't initially work magically work after I run commands that do link libraries correctly. The interpreter/compiler seem to have more predictable behavior.

In any case, I'm looking into a fix for the REPL!

[zachgrayio]

Actually, this confirms what I saw once or twice myself but hadn't brought up yet. The REPL does behave incosistently. The really strange part is that the linker flags passed in a specific order seems to work more predictably.

[dan-zheng]

Sorry for the delay! I believe this is fixed in 1969380.
You can try the pre-built packages from 05-10 to verify.

Verifying the fix was a bit difficult because the REPL doesn't seem to fail consistently.
I made sure that the Couldn't lookup symbols error didn't occur to me anymore for the commands listed above involving the TFExample library.

It's fully possible that linking more libraries causes other issues. If you find any, please file a new issue and I'll take a look. 😄