Jemalloc compatibility fix

[ronrother]

Hi @wolfcw

I'm opening this PR to share a patch that potentially addresses #327 and #130. I agree with your opinion that libjemalloc should be more resilient during initialization (it does not seem to support reentrancy when calling clock_gettime), but for use cases where we can't really patch Jemalloc, it could be useful to have an alternative.
This patch does the following:

1. Interpositions calloc and free using static buffer, depending on a flag (only to enable dlsym calls);
2. Within init_pthread_cond_init_232, loads only the necessary functions to use it; [1]
3. Within clock_gettime, loads only real_clock_gettime and calls that if not initalized; [1];
4. calloc and free use the internal __libc_* functions if it didn't have a chance to load the real_ before using.

[1] that means it effectively does not initialize the library in these two functions.

Please let me know if there are any changes I can make to turn this into a mergeable fix. Even if you decide by not integrating it, I think it's good to have a reference point for anyone struggling with this issue.

PS: An alternative, simpler fix that works for some use cases, is to just syscall SYS_clock_gettime inside clock_gettime when not initialized, but then you also have to compile without FAKE_PTHREAD (since Jemalloc depends on that too).

[wolfcw]

Thanks!

I like the approach taken of adapting libfaketime's initialization procedures when used along jemalloc. I also think it's a good idea wrapping it in #ifdefs, given the interception of frequently used functions like calloc() and free().

Still, and before merging this, I wonder whether you considered adding another activating flag, e.g., an environment variable similar to FAKETIME_DONT_FAKE_MONOTONIC, to enable jemalloc compatibility mode (on top of the compile-time flag). Right now, users will have to compile and ld_preload two versions of libfaketime - one with jemalloc support and one without - if they want to use it on both types of applications.

Maybe it would be easier to use by setting an environment variable, e.g., FAKETIME_JEMALLOC_COMPAT=1 on a libfaketime version compiled with -DJEMALLOC_COMPAT, and simple omit that environment variable when jemalloc compatibility is not required? Or would that result in another reentrancy problem?

[ronrother]

Hey @wolfcw

I like your idea, it certainly simplifies operations and likely would still solve the problem. I will have a try at it over the next couple day.
Thanks for the feedback!

[wolfcw]

@ronrother: As mentioned in #327, this might actually shift the recursion problem from clock_gettime() into calloc()/dlsym(). Not sure yet whether this is glibc / jemalloc version specific.

[ronrother]

@wolfcw Yes, the solution in this PR is somewhat finicky. For instance, libfaketime needs to be preloaded before jemalloc (come first in the LD_PRELOAD order), otherwise jemalloc will interpose calloc() first. I've been thinking of a better solution, but couldn't come up with anything perfect so far.
One real dealbreaker is jemalloc's dependence on pthread_init_cond, because we have no choice there other than use dlsym to get a reference to the real function. For clock_gettime and the other time functions, I think you could get away with issuing a syscall to __sys_clock_gettime when not initalized, and not calling ftpl_init there).

[ronrother]

Closing this PR since this mitigation can't be safely integrated.