Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ccall((function, library), ...) does not work in __init__ when the library is a position independent executable #29972

Closed
tkf opened this issue Nov 9, 2018 · 1 comment
Closed

ccall((function, library), ...) does not work in __init__ when the library is a position independent executable #29972

tkf opened this issue Nov 9, 2018 · 1 comment

Comments

@tkf
Copy link
Member

tkf commented Nov 9, 2018

Original discussion at: JuliaPy/PyCall.jl#612

Executing ccall((function_name, library), ...) when library is a position independent executable (PIE) Python yields a segmentation fault when done in __init__. However, it works if (1) it's done in REPL or script (i.e., top-level statements) or (2) Libdl.dlsym is used. This is maybe Linux-specific. I can reproduce this with Julia 1.0.1 and master. Here is a demo: clone https://gist.github.com/tkf/871c893d8bebf6741e10ee14f46c30fe and run make or

$ conda create --prefix py defaults::python
...

$ file -L py/bin/python
py/bin/python: ELF 64-bit LSB pie executable x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, with debug_info, not stripped

$ cat PIEPyCall-0.jl
module PIEPyCall

using Libdl

const pyhome = abspath("py")
const libpython = "$pyhome/bin/python3"
const wPYTHONHOME = Base.cconvert(Cwstring, string(pyhome, ':', pyhome))
const wpyprogramname = Base.cconvert(Cwstring, libpython)

function __init__()
    h = Libdl.dlopen(libpython, Libdl.RTLD_LAZY|Libdl.RTLD_GLOBAL)
    @show unsafe_string(ccall((:Py_GetVersion, libpython), Ptr{UInt8}, ()))

    @show ccall((:Py_IsInitialized, libpython), Cint, ())

    ccall((:Py_SetProgramName, libpython), Cvoid, (Ptr{Cwchar_t},), wpyprogramname)
    ccall((:Py_SetPythonHome, libpython), Cvoid, (Ptr{Cwchar_t},), wPYTHONHOME)

    ccall((:Py_InitializeEx, libpython), Cvoid, (Cint,), 0)

    @show ccall((:Py_IsInitialized, libpython), Cint, ())
end

end

$ julia --startup-file=no PIEPyCall-0.jl
unsafe_string(ccall((:Py_GetVersion, libpython), Ptr{UInt8}, ())) = "3.7.1 (default, Oct 23 2018, 19:19:42) \n[GCC 7.3.0]"
ccall((:Py_IsInitialized, libpython), Cint, ()) = 0

signal (11): Segmentation fault
in expression starting at /home/takafumi/repos/scratch/gist/871c893d8bebf6741e10ee14f46c30fe/PIEPyCall-0.jl:10
fileno_unlocked at /usr/lib/libc.so.6 (unknown line)
_PySys_BeginInit at /tmp/build/80754af9/python_1540319607830/work/Python/sysmodule.c:2292
_Py_InitializeCore_impl at /tmp/build/80754af9/python_1540319607830/work/Python/pylifecycle.c:753
_Py_InitializeCore at /tmp/build/80754af9/python_1540319607830/work/Python/pylifecycle.c:859
_Py_InitializeFromConfig at /tmp/build/80754af9/python_1540319607830/work/Python/pylifecycle.c:1002
Py_InitializeEx at /tmp/build/80754af9/python_1540319607830/work/Python/pylifecycle.c:1034
__init__ at /home/takafumi/repos/scratch/gist/871c893d8bebf6741e10ee14f46c30fe/PIEPyCall-0.jl:16
unknown function (ip: 0x7fc31c786c3c)
jl_fptr_trampoline at /buildworker/worker/package_linux64/build/src/gf.c:1831
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2184
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1537 [inlined]
jl_module_run_initializer at /buildworker/worker/package_linux64/build/src/toplevel.c:90
jl_module_load_time_initialize at /buildworker/worker/package_linux64/build/src/toplevel.c:122 [inlined]
jl_eval_module_expr at /buildworker/worker/package_linux64/build/src/toplevel.c:276
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:651
jl_parse_eval_all at /buildworker/worker/package_linux64/build/src/ast.c:838
jl_load at /buildworker/worker/package_linux64/build/src/toplevel.c:847
include at ./boot.jl:317 [inlined]
include_relative at ./loading.jl:1041
include at ./sysimg.jl:29
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2184
exec_options at ./client.jl:229
_start at ./client.jl:421
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2184
unknown function (ip: 0x401ae8)
unknown function (ip: 0x401513)
__libc_start_main at /usr/lib/libc.so.6 (unknown line)
unknown function (ip: 0x4015b4)
Allocations: 104122 (Pool: 104099; Big: 23); GC: 0
zsh: segmentation fault (core dumped)  julia --startup-file=no PIEPyCall-0.jl
julia --startup-file=no PIEPyCall-0.jl  0.58s user 0.99s system 131% cpu 1.193 total

$ cat PIEPyCall-1.jl
module PIEPyCall

using Libdl

const pyhome = abspath("py")
const libpython = "$pyhome/bin/python3"
const wPYTHONHOME = Base.cconvert(Cwstring, string(pyhome, ':', pyhome))
const wpyprogramname = Base.cconvert(Cwstring, libpython)

# function __init__()
    h = Libdl.dlopen(libpython, Libdl.RTLD_LAZY|Libdl.RTLD_GLOBAL)
    @show unsafe_string(ccall((:Py_GetVersion, libpython), Ptr{UInt8}, ()))

    @show ccall((:Py_IsInitialized, libpython), Cint, ())

    ccall((:Py_SetProgramName, libpython), Cvoid, (Ptr{Cwchar_t},), wpyprogramname)
    ccall((:Py_SetPythonHome, libpython), Cvoid, (Ptr{Cwchar_t},), wPYTHONHOME)

    ccall((:Py_InitializeEx, libpython), Cvoid, (Cint,), 0)

    @show ccall((:Py_IsInitialized, libpython), Cint, ())
# end

end

$ julia --startup-file=no PIEPyCall-1.jl
unsafe_string(ccall((:Py_GetVersion, libpython), Ptr{UInt8}, ())) = "3.7.1 (default, Oct 23 2018, 19:19:42) \n[GCC 7.3.0]"
ccall((:Py_IsInitialized, libpython), Cint, ()) = 0
ccall((:Py_IsInitialized, libpython), Cint, ()) = 1

$ cat PIEPyCall-2.jl
module PIEPyCall

using Libdl

const pyhome = abspath("py")
const libpython = "$pyhome/bin/python3"
const wPYTHONHOME = Base.cconvert(Cwstring, string(pyhome, ':', pyhome))
const wpyprogramname = Base.cconvert(Cwstring, libpython)

function __init__()
    h = Libdl.dlopen(libpython, Libdl.RTLD_LAZY|Libdl.RTLD_GLOBAL)
    @show unsafe_string(ccall(Libdl.dlsym(h, :Py_GetVersion), Ptr{UInt8}, ()))

    @show ccall(Libdl.dlsym(h,:Py_IsInitialized), Cint, ())

    ccall(Libdl.dlsym(h,:Py_SetProgramName), Cvoid, (Ptr{Cwchar_t},), wpyprogramname)
    ccall(Libdl.dlsym(h,:Py_SetPythonHome), Cvoid, (Ptr{Cwchar_t},), wPYTHONHOME)

    ccall(Libdl.dlsym(h,:Py_InitializeEx), Cvoid, (Cint,), 0)

    @show ccall(Libdl.dlsym(h,:Py_IsInitialized), Cint, ())
end

end

$ julia --startup-file=no PIEPyCall-2.jl
unsafe_string(ccall(Libdl.dlsym(h, :Py_GetVersion), Ptr{UInt8}, ())) = "3.7.1 (default, Oct 23 2018, 19:19:42) \n[GCC 7.3.0]"
ccall(Libdl.dlsym(h, :Py_IsInitialized), Cint, ()) = 0
ccall(Libdl.dlsym(h, :Py_IsInitialized), Cint, ()) = 1

PIEPyCall-0.jl runs ccall inside __init__ (and segfaults) but PIEPyCall-1.jl does it at top-level (and works):

--- PIEPyCall-0.jl      2018-11-08 22:04:45.813491118 -0800
+++ PIEPyCall-1.jl      2018-11-08 22:02:55.983491566 -0800
@@ -7,7 +7,7 @@
 const wPYTHONHOME = Base.cconvert(Cwstring, string(pyhome, ':', pyhome))
 const wpyprogramname = Base.cconvert(Cwstring, libpython)

-function __init__()
+# function __init__()
     h = Libdl.dlopen(libpython, Libdl.RTLD_LAZY|Libdl.RTLD_GLOBAL)
     @show unsafe_string(ccall((:Py_GetVersion, libpython), Ptr{UInt8}, ()))

@@ -19,6 +19,6 @@
     ccall((:Py_InitializeEx, libpython), Cvoid, (Cint,), 0)

     @show ccall((:Py_IsInitialized, libpython), Cint, ())
-end
+# end

 end

PIEPyCall-2.jl uses Libdl.dlsym:

--- PIEPyCall-0.jl      2018-11-08 22:04:45.813491118 -0800
+++ PIEPyCall-2.jl      2018-11-08 22:11:10.503489548 -0800
@@ -9,16 +9,16 @@

 function __init__()
     h = Libdl.dlopen(libpython, Libdl.RTLD_LAZY|Libdl.RTLD_GLOBAL)
-    @show unsafe_string(ccall((:Py_GetVersion, libpython), Ptr{UInt8}, ()))
+    @show unsafe_string(ccall(Libdl.dlsym(h, :Py_GetVersion), Ptr{UInt8}, ()))

-    @show ccall((:Py_IsInitialized, libpython), Cint, ())
+    @show ccall(Libdl.dlsym(h,:Py_IsInitialized), Cint, ())

-    ccall((:Py_SetProgramName, libpython), Cvoid, (Ptr{Cwchar_t},), wpyprogramname)
-    ccall((:Py_SetPythonHome, libpython), Cvoid, (Ptr{Cwchar_t},), wPYTHONHOME)
+    ccall(Libdl.dlsym(h,:Py_SetProgramName), Cvoid, (Ptr{Cwchar_t},), wpyprogramname)
+    ccall(Libdl.dlsym(h,:Py_SetPythonHome), Cvoid, (Ptr{Cwchar_t},), wPYTHONHOME)

-    ccall((:Py_InitializeEx, libpython), Cvoid, (Cint,), 0)
+    ccall(Libdl.dlsym(h,:Py_InitializeEx), Cvoid, (Cint,), 0)

-    @show ccall((:Py_IsInitialized, libpython), Cint, ())
+    @show ccall(Libdl.dlsym(h,:Py_IsInitialized), Cint, ())
 end

 end

cc @isuruf @stevengj
@tkf tkf changed the title ccall((function, library), ...) does not work in __init__ when the library is PIE ccall((function, library), ...) does not work in __init__ when the library is a position independent executable Nov 9, 2018
@tkf
Copy link
Member Author

tkf commented Nov 10, 2018

@isuruf founds the answer JuliaPy/PyCall.jl#612 (comment)

looks like ccall((function, library)) calls dlopen with RTLD_DEEPBIND even if the library has been loaded before. using a library handle directly works.

@tkf tkf closed this as completed Nov 10, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@tkf