Skip to content
This repository has been archived by the owner on Jan 20, 2022. It is now read-only.

libQt5Core.so.5 shared Library not found #2420

Closed
dzygann opened this issue Jun 3, 2021 · 17 comments
Closed

libQt5Core.so.5 shared Library not found #2420

dzygann opened this issue Jun 3, 2021 · 17 comments

Comments

@dzygann
Copy link

dzygann commented Jun 3, 2021

By using the memcached example I want to run murmurd in Graphene. By running make, the manifest file for my binary is created. The loader.env looks like this:

loader.env.LD_LIBRARY_PATH = "/lib:{{ arch_libdir }}:/usr/lib:/usr/{{ arch_libdir }}

It's also mounted via:


fs.mount.lib2.type = "chroot"
fs.mount.lib2.path = "{{ arch_libdir }}"     
fs.mount.lib2.uri = "file:{{ arch_libdir }}"

The ldd command returns the following for murmurd:

	linux-vdso.so.1 (0x00007fffd4fd9000)
	libQt5Sql.so.5 => /lib/x86_64-linux-gnu/libQt5Sql.so.5 (0x00007fcbc110b000)
	libcap.so.2 => /lib/x86_64-linux-gnu/libcap.so.2 (0x00007fcbc1102000)
	libdns_sd.so.1 => /lib/x86_64-linux-gnu/libdns_sd.so.1 (0x00007fcbc10f6000)
	libQt5DBus.so.5 => /lib/x86_64-linux-gnu/libQt5DBus.so.5 (0x00007fcbc105a000)
	libIce.so.37 => /lib/x86_64-linux-gnu/libIce.so.37 (0x00007fcbc0c7b000)
	libprotobuf.so.17 => /lib/x86_64-linux-gnu/libprotobuf.so.17 (0x00007fcbc0983000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fcbc095e000)
	libssl.so.1.1 => /lib/x86_64-linux-gnu/libssl.so.1.1 (0x00007fcbc08cb000)
	libcrypto.so.1.1 => /lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007fcbc05f5000)
	libQt5Network.so.5 => /lib/x86_64-linux-gnu/libQt5Network.so.5 (0x00007fcbc0434000)
	libQt5Xml.so.5 => /lib/x86_64-linux-gnu/libQt5Xml.so.5 (0x00007fcbc03f2000)
	libQt5Core.so.5 => /lib/x86_64-linux-gnu/libQt5Core.so.5 (0x00007fcbbfea9000) <- This one causes the issue
	libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fcbbfcc6000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fcbbfcab000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fcbbfab9000)
        ... some more libs

I get the following output if I try to run murmur in Graphene with debug enabled:

...  some other output
// This one looks a little bit suspicous, but I don't know what is it about
[P7088:T1:murmurd] warning: Not supported flag (0x3001) passed to arch_prctl
... some other output
[P7088:T1:murmurd] debug: adding a library for gdb: file:/lib/x86_64-linux-gnu/libcrypto.so.1.1
[P7088:T1:murmurd] debug: glibc register library /lib/x86_64-linux-gnu/libQt5Network.so.5 loaded at 0x486ee6631000
[P7088:T1:murmurd] debug: adding a library for gdb: file:/lib/x86_64-linux-gnu/libQt5Network.so.5
[P7088:T1:murmurd] debug: glibc register library /lib/x86_64-linux-gnu/libQt5Xml.so.5 loaded at 0x486ee65ef000
[P7088:T1:murmurd] debug: adding a library for gdb: file:/lib/x86_64-linux-gnu/libQt5Xml.so.5
murmurd: error while loading shared libraries: libQt5Core.so.5: cannot open shared object file: No such file or directory
[P7088:T1:murmurd] debug: ---- shim_exit_group (returning 127)
[P7088:i1:murmurd] debug: IPC worker: exiting worker thread
[P7088:T1:murmurd] debug: process 7088 exited with status 127

You can see in the log that the other libraries are loaded correctly. Unfortunately, the libQt5Core don't and I don't have a idea what is wrong with this one. If I run murmur without Graphene it works fine. Do you have an idea what I'm doing wrong?

Thanks in advance.
@dimakuv
Copy link

dimakuv commented Jun 4, 2021

// This one looks a little bit suspicous, but I don't know what is it about
[P7088:T1:murmurd] warning: Not supported flag (0x3001) passed to arch_prctl

You can ignore this, not important.

murmurd: error while loading shared libraries: libQt5Core.so.5: cannot open shared object file: No such file or directory

This is indeed very strange. Couple questions:

  1. Could you provide details on how you install Murmurd, how you port it to Graphene, and how you run it with Graphene? It should be easy to reproduce on our local systems, and we (Graphene developers) could quickly debug the root cause.

  2. Could you try running the original Murmurd application under strace? Something like strace -f <murmurd with all your command-line arguments>. This will give you a log of what the original application does. In this log, please search for libQt5Core.so.5. It may turn out that this library is loaded not from /lib/x86_64-linux-gnu/ after all...

  3. It looks like you currently try the non-SGX ("direct") version of Graphene. Is this correct? (This is good, it's always better to first run in non-SGX mode, and only then move to Graphene-SGX.)

@dzygann
Copy link
Author

dzygann commented Jun 4, 2021

Hi,

Could you provide details on how you install Murmurd, how you port it to Graphene, and how you run it with Graphene? It should be easy to reproduce on our local systems, and we (Graphene developers) could quickly debug the root cause.

I pulled mumble from https://github.com/mumble-voip/mumble and followed the build process. Then I copied all files from the build folder (here are the executables located for mumble(client) and murmur(server))and put it into the Graphene Example folder. Then I copied the Makefile and the manifest.template from memcached and made the customization for murmur.

Could you try running the original Murmurd application under strace? Something like strace -f . This will give you a log of what the original application does. In this log, please search for libQt5Core.so.5. It may turn out that this library is loaded not from /lib/x86_64-linux-gnu/ after all...

strace gives the following output for libQt5Core:

openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libQt5Core.so.5", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\200C\v\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=5519624, ...}) = 0
mmap(NULL, 5537872, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fe40031f000
mprotect(0x7fe4003a2000, 4931584, PROT_NONE) = 0
mmap(0x7fe4003a2000, 3014656, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x83000) = 0x7fe4003a2000
mmap(0x7fe400682000, 1912832, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x363000) = 0x7fe400682000
mmap(0x7fe400856000, 53248, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x536000) = 0x7fe400856000
mmap(0x7fe400863000, 16464, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fe400863000
close(3)                                = 0

It looks like you currently try the non-SGX ("direct") version of Graphene. Is this correct? (This is good, it's always better to first run in non-SGX mode, and only then move to Graphene-SGX.)

Yes, I try the direct mode, because my current machine is not SGX enabled...

@dimakuv
Copy link

dimakuv commented Jun 4, 2021

Everything looks very normal in the strace. The libQt5Core.so.5 file is where it is expected to be, this file is only ~5MB in size, it is a normal ELF file... Everything is good, and Graphene shouldn't complain.

Can you set the log level in the Graphene debug output to all? You need to modify the manifest file with something like loader.log_level = "all". This will give you the most verbose Graphene output. Please re-run your Murmur app under Graphene with this setting and paste the appropriate part of the Graphene log here.

@dzygann
Copy link
Author

dzygann commented Jun 4, 2021 •
edited
Loading

I could find the libQt5Core.so.5 in both places /lib/x86_64-linux-gnu and in /usr/lib/x86_64-linux-gnu on my machine.
The trace confuse me a little bit. Does the output mean that the shim_openat function can find the lib?

[P1665:T1:murmurd] trace: ---- shim_openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libQt5Core.so.5", O_RDONLY|0x80000, 0000) = 0x3

But it is not possible to register it to glibc?

[P1665:T1:murmurd] debug: glibc register library /lib/x86_64-linux-gnu/libQt5Xml.so.5 loaded at 0x1a2d659d3000
[P1665:T1:murmurd] debug: adding a library for gdb: file:/lib/x86_64-linux-gnu/libQt5Xml.so.5
[P1665:T1:murmurd] trace: ---- shim_openat(AT_FDCWD, "/lib/libQt5Core.so.5", O_RDONLY|0x80000, 0000) = -2
[P1665:T1:murmurd] trace: ---- shim_openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libQt5Core.so.5", O_RDONLY|0x80000, 0000) = 0x3
[P1665:T1:murmurd] trace: ---- shim_read(3, 0x1a2d66a6f1b8, 0x340) ...
[P1665:T1:murmurd] trace: ---- return from shim_read(...) = 0x340
[P1665:T1:murmurd] trace: ---- shim_close(3) = 0x0
[P1665:T1:murmurd] trace: ---- shim_openat(AT_FDCWD, "/usr/lib/tls/x86_64/libQt5Core.so.5", O_RDONLY|0x80000, 0000) = -2
[P1665:T1:murmurd] trace: ---- shim_newfstatat(AT_FDCWD, "/usr/lib/tls/x86_64", 0x1a2d66a6f060, 0) = -2
[P1665:T1:murmurd] trace: ---- shim_openat(AT_FDCWD, "/usr/lib/tls/libQt5Core.so.5", O_RDONLY|0x80000, 0000) = -2
[P1665:T1:murmurd] trace: ---- shim_newfstatat(AT_FDCWD, "/usr/lib/tls", 0x1a2d66a6f060, 0) = -2
[P1665:T1:murmurd] trace: ---- shim_openat(AT_FDCWD, "/usr/lib/x86_64/libQt5Core.so.5", O_RDONLY|0x80000, 0000) = -2
[P1665:T1:murmurd] trace: ---- shim_newfstatat(AT_FDCWD, "/usr/lib/x86_64", 0x1a2d66a6f060, 0) = -2
[P1665:T1:murmurd] trace: ---- shim_openat(AT_FDCWD, "/usr/lib/libQt5Core.so.5", O_RDONLY|0x80000, 0000) = -2
[P1665:T1:murmurd] trace: ---- shim_newfstatat(AT_FDCWD, "/usr/lib", 0x1a2d66a6f060, 0) = -2
[P1665:T1:murmurd] trace: ---- shim_openat(AT_FDCWD, "/usr//lib/x86_64-linux-gnu/tls/x86_64/libQt5Core.so.5", O_RDONLY|0x80000, 0000) = -2
[P1665:T1:murmurd] trace: ---- shim_newfstatat(AT_FDCWD, "/usr//lib/x86_64-linux-gnu/tls/x86_64", 0x1a2d66a6f060, 0) = -2
[P1665:T1:murmurd] trace: ---- shim_openat(AT_FDCWD, "/usr//lib/x86_64-linux-gnu/tls/libQt5Core.so.5", O_RDONLY|0x80000, 0000) = -2
[P1665:T1:murmurd] trace: ---- shim_newfstatat(AT_FDCWD, "/usr//lib/x86_64-linux-gnu/tls", 0x1a2d66a6f060, 0) = -2
[P1665:T1:murmurd] trace: ---- shim_openat(AT_FDCWD, "/usr//lib/x86_64-linux-gnu/x86_64/libQt5Core.so.5", O_RDONLY|0x80000, 0000) = -2
[P1665:T1:murmurd] trace: ---- shim_newfstatat(AT_FDCWD, "/usr//lib/x86_64-linux-gnu/x86_64", 0x1a2d66a6f060, 0) = -2
[P1665:T1:murmurd] trace: ---- shim_openat(AT_FDCWD, "/usr//lib/x86_64-linux-gnu/libQt5Core.so.5", O_RDONLY|0x80000, 0000) = 0x3
[P1665:T1:murmurd] trace: ---- shim_read(3, 0x1a2d66a6f1b8, 0x340) ...
[P1665:T1:murmurd] trace: ---- return from shim_read(...) = 0x340
[P1665:T1:murmurd] trace: ---- shim_close(3) = 0x0
[P1665:T1:murmurd] trace: ---- shim_newfstatat(AT_FDCWD, "/usr//lib/x86_64-linux-gnu", 0x1a2d66a6f060, 0) = 0x0
[P1665:T1:murmurd] trace: ---- shim_openat(AT_FDCWD, "/home/developer/workspaces/graphene/LibOS/../Runtime/etc/ld.so.cache", O_RDONLY|0x80000, 0000) = -2
[P1665:T1:murmurd] trace: ---- shim_openat(AT_FDCWD, "/home/developer/workspaces/graphene/LibOS/../Runtime/lib/tls/x86_64/libQt5Core.so.5", O_RDONLY|0x80000, 0000) = -2
[P1665:T1:murmurd] trace: ---- shim_newfstatat(AT_FDCWD, "/home/developer/workspaces/graphene/LibOS/../Runtime/lib/tls/x86_64", 0x1a2d66a6f060, 0) = -2
[P1665:T1:murmurd] trace: ---- shim_openat(AT_FDCWD, "/home/developer/workspaces/graphene/LibOS/../Runtime/lib/tls/libQt5Core.so.5", O_RDONLY|0x80000, 0000) = -2
[P1665:T1:murmurd] trace: ---- shim_newfstatat(AT_FDCWD, "/home/developer/workspaces/graphene/LibOS/../Runtime/lib/tls", 0x1a2d66a6f060, 0) = -2
[P1665:T1:murmurd] trace: ---- shim_openat(AT_FDCWD, "/home/developer/workspaces/graphene/LibOS/../Runtime/lib/x86_64/libQt5Core.so.5", O_RDONLY|0x80000, 0000) = -2
[P1665:T1:murmurd] trace: ---- shim_newfstatat(AT_FDCWD, "/home/developer/workspaces/graphene/LibOS/../Runtime/lib/x86_64", 0x1a2d66a6f060, 0) = -2
[P1665:T1:murmurd] trace: ---- shim_openat(AT_FDCWD, "/home/developer/workspaces/graphene/LibOS/../Runtime/lib/libQt5Core.so.5", O_RDONLY|0x80000, 0000) = -2
[P1665:T1:murmurd] trace: ---- shim_newfstatat(AT_FDCWD, "/home/developer/workspaces/graphene/LibOS/../Runtime/lib", 0x1a2d66a6f060, 0) = -2
murmurd: error while loading shared libraries: libQt5Core.so.5: cannot open shared object file: No such file or directory
[P1665:T1:murmurd] trace: ---- shim_writev(2, 0x1a2d66a6ed00, 10) = 0x7a
[P1665:T1:murmurd] debug: ---- shim_exit_group (returning 127)
[P1665:i1:murmurd] debug: IPC worker: exiting worker thread
[P1665:T1:murmurd] debug: process 1665 exited with status 127

--- Edit ---
I just see there is a double slash in /usr//lib/x86_64-linux-gnu/libQt5Core.so.5 . Could this be the reason?

@dimakuv
Copy link

dimakuv commented Jun 4, 2021

Heh, that's very interesting.

The "good" strace has these lines:

openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libQt5Core.so.5", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\200C\v\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=5519624, ...}) = 0
mmap(NULL, 5537872, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fe40031f000

The "bad" Graphene trace has these lines:

[P1665:T1:murmurd] trace: ---- shim_openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libQt5Core.so.5", O_RDONLY|0x80000, 0000) = 0x3
[P1665:T1:murmurd] trace: ---- shim_read(3, 0x1a2d66a6f1b8, 0x340) ...
[P1665:T1:murmurd] trace: ---- return from shim_read(...) = 0x340
[P1665:T1:murmurd] trace: ---- shim_close(3) = 0x0

Note that 832 == 0x340, so both reads get the same number of characters from the file.

But the good strace continues with fstat and mmap -- so apparently the application is happy with what it read in the ELF header section of the libQt5Core.so.5 file.

However, the bad Graphene trace simply performs a close -- which hints that the application is unhappy with the ELF header section of the libQt5Core.so.5 file.

So this brings me to the following suspicion: Graphene supplies by default Glibc version 2.33. I bet that your OS distro has another Glibc version. Typically it doesn't matter because Glibc implementations are almost 100% compatible with each other. But maybe libQt5Core expects a very particular version of Glibc?

What is your OS distro?

@dzygann
Copy link
Author

dzygann commented Jun 4, 2021

My OS is Ubuntu 20.04. To check the the GLIBC I have used ldd --version, which returns
ldd (Ubuntu GLIBC 2.31-0ubuntu9.2) 2.31

Should I try to upgrade glibc or to downgrade the Glibc version of Graphene?

@dimakuv
Copy link

dimakuv commented Jun 4, 2021

Try to downgrade the Glibc version used in Graphene to 2.31 (fortunately, we have it).

Please re-build Graphene from scratch using environment variable GLIBC_VERSION = '2.31'. And then try again.

It would be also good to google a bit about libQt5Core and its requirements. Maybe someone else hit this problem (independently of Graphene), and there is a solution to it?

@dzygann
Copy link
Author

dzygann commented Jun 4, 2021

I changed the value in the LibOS/Makefile and did a make clean and make again.
It is still the same issue:

P194431:T1:murmurd] trace: ---- shim_openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/libQt5Core.so.5", O_RDONLY|0x80000, 0000) = 0x3
[P194431:T1:murmurd] trace: ---- shim_read(3, 0x386dabfba228, 0x340) ...
[P194431:T1:murmurd] trace: ---- return from shim_read(...) = 0x340
[P194431:T1:murmurd] trace: ---- shim_close(3) = 0x0
[P194431:T1:murmurd] trace: ---- shim_stat("/usr/lib/x86_64-linux-gnu", 0x386dabfba0d0) = 0x0
[P194431:T1:murmurd] trace: ---- shim_openat(AT_FDCWD, "/home/developer/workspaces/graphene/LibOS/../Runtime/etc/ld.so.cache", O_RDONLY|0x80000, 0000) = -2
[P194431:T1:murmurd] trace: ---- shim_openat(AT_FDCWD, "/home/developer/workspaces/graphene/LibOS/../Runtime/lib/tls/x86_64/libQt5Core.so.5", O_RDONLY|0x80000, 0000) = -2
[P194431:T1:murmurd] trace: ---- shim_stat("/home/developer/workspaces/graphene/LibOS/../Runtime/lib/tls/x86_64", 0x386dabfba0d0) = -2
[P194431:T1:murmurd] trace: ---- shim_openat(AT_FDCWD, "/home/developer/workspaces/graphene/LibOS/../Runtime/lib/tls/libQt5Core.so.5", O_RDONLY|0x80000, 0000) = -2
[P194431:T1:murmurd] trace: ---- shim_stat("/home/developer/workspaces/graphene/LibOS/../Runtime/lib/tls", 0x386dabfba0d0) = -2
[P194431:T1:murmurd] trace: ---- shim_openat(AT_FDCWD, "/home/developer/workspaces/graphene/LibOS/../Runtime/lib/x86_64/libQt5Core.so.5", O_RDONLY|0x80000, 0000) = -2
[P194431:T1:murmurd] trace: ---- shim_stat("/home/developer/workspaces/graphene/LibOS/../Runtime/lib/x86_64", 0x386dabfba0d0) = -2
[P194431:T1:murmurd] trace: ---- shim_openat(AT_FDCWD, "/home/developer/workspaces/graphene/LibOS/../Runtime/lib/libQt5Core.so.5", O_RDONLY|0x80000, 0000) = -2
[P194431:T1:murmurd] trace: ---- shim_stat("/home/developer/workspaces/graphene/LibOS/../Runtime/lib", 0x386dabfba0d0) = -2
murmurd: error while loading shared libraries: libQt5Core.so.5: cannot open shared object file: No such file or directory

Is it possible to double check the Graphene GLIBC version somehow?

@dimakuv
Copy link

dimakuv commented Jun 4, 2021

I guess the easiest would be something like:
~/graphene/Runtime$ strings libc.so.6 | grep "GLIBC_2.31"

@dimakuv
Copy link

dimakuv commented Jun 4, 2021

I didn't read through the rationale, but maybe try this: YosysHQ/nextpnr#375 (comment). No idea how this can be related to Graphene :)

@dzygann
Copy link
Author

dzygann commented Jun 4, 2021

@dimakuv You are incredible. This did the trick :)

sudo strip --remove-section=.note.ABI-tag /usr/lib/x86_64-linux-gnu/libQt5Core.so.5

[P194630:T1:murmurd] debug: glibc register library /lib/x86_64-linux-gnu/libQt5Core.so.5 loaded at 0x2a0b48ad8000
[P194630:T1:murmurd] debug: adding a library for gdb: file:/lib/x86_64-linux-gnu/libQt5Core.so.5
[P194630:T1:murmurd] trace: ---- shim_openat(AT_FDCWD, "/lib/libstdc++.so.6", O_RDONLY|0x80000, 0000) = -2
[P194630:T1:murmurd] trace: ---- shim_openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libstdc++.so.6", O_RDONLY|0x80000, 0000) = 0x3
[P194630:T1:murmurd] trace: ---- shim_read(3, 0x2a0b4a0b9208, 0x340) ...
[P194630:T1:murmurd] trace: ---- return from shim_read(...) = 0x340
[P194630:T1:murmurd] trace: ---- shim_fstat(3, 0x2a0b4a0b90b0) = 0x0
[P194630:T1:murmurd] trace: ---- shim_mmap(0x0, 0x2000, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0x0) ...
[P194630:T1:murmurd] trace: ---- return from shim_mmap(...) = 0x2a0b48ad6000
[P194630:T1:murmurd] trace: ---- shim_mmap(0x0, 0x1e0800, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0x0) ...
[P194630:T1:murmurd] trace: ---- return from shim_mmap(...) = 0x2a0b488f5000
[P194630:T1:murmurd] trace: ---- shim_mprotect(0x2a0b4898b000, 0x13a000, PROT_NONE) ...
[P194630:T1:murmurd] trace: ---- return from shim_mprotect(...) = 0x0
[P194630:T1:murmurd] trace: ---- shim_mmap(0x2a0b4898b000, 0xf0000, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x96000) ...
[P194630:T1:murmurd] trace: ---- return from shim_mmap(...) = 0x2a0b4898b000
[P194630:T1:murmurd] trace: ---- shim_mmap(0x2a0b48a7b000, 0x49000, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x186000) ...
[P194630:T1:murmurd] trace: ---- return from shim_mmap(...) = 0x2a0b48a7b000
[P194630:T1:murmurd] trace: ---- shim_mmap(0x2a0b48ac5000, 0xe000, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1cf000) ...
[P194630:T1:murmurd] trace: ---- return from shim_mmap(...) = 0x2a0b48ac5000
[P194630:T1:murmurd] trace: ---- shim_mmap(0x2a0b48ad3000, 0x2800, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_FIXED, -1, 0x0) ...
[P194630:T1:murmurd] trace: ---- return from shim_mmap(...) = 0x2a0b48ad3000
[P194630:T1:murmurd] trace: ---- shim_close(3) = 0x0

Now I run into the next issue, but I will first investigate and open a new ticket if I need support.
Thank you so much for your support!

@dzygann dzygann closed this as completed Jun 4, 2021
@dimakuv
Copy link

dimakuv commented Jun 4, 2021

Cool.

So I found the real root cause analysis for this: microsoft/WSL#3023 (comment).

Long story short: Glibc checks the kernel version using three methods (reading vDSO magic numbers, reading uname(2) and reading /proc/sys/kernel/osrelease), if it finds the .note.ABI-tag ELF section (it contains the minimal required kernel version).

In Graphene, we probably don't have anything in our vDSO lib at all, we have some really old Linux version in uname, and /proc/sys/kernel/osrelease is not existing at all. We should fix this to report some meaningful Linux version inside Graphene.

Very interesting bug...

@boryspoplawski
Copy link
Contributor

In Graphene, we probably don't have anything in our vDSO lib at all, we have some really old Linux version in uname, and /proc/sys/kernel/osrelease is not existing at all. We should fix this to report some meaningful Linux version inside Graphene.

We do have a note section in vDSO, currently with hardcoded version 4.13.0 https://github.com/oscarlab/graphene/blob/master/LibOS/shim/src/vdso/vdso-note.S#L10

@dzygann
Copy link
Author

dzygann commented Jun 4, 2021

Should I reopen the ticket, so that you can deliver a fix for this issue?

@boryspoplawski
Copy link
Contributor

I've just checked and libQt5 from Ubuntu20.04 requires version 3.17.0 and we report 4.19.0 in our vdso... OTOH our uname emulation reports 3.10.0, so maybe this is the problem.

Any volunteers for trying if updating that works? :)

@boryspoplawski boryspoplawski reopened this Jun 4, 2021
@boryspoplawski
Copy link
Contributor

After checking it seems that we do not have that .note section in our vDSO. Seems like our recent Meson rewrite broke it. @woju please fix :)

@dimakuv dimakuv mentioned this issue Jul 22, 2021
Open
@dimakuv
Copy link

dimakuv commented Jul 22, 2021

Created #2572 per Borys's last comments. This issue got too long, the original problem is solved (well, circumvented).

@dimakuv dimakuv closed this as completed Jul 22, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@dimakuv @boryspoplawski @dzygann