Macos python 3.11/3.12 calling (some) subprocesses from executor

[ryandvmartin]

Looking for some guidance on a strange error on Python 3.11/3.12 on MacOS with this executor. I experienced the issue using joblib but managed to reproduce here. The modified example is borrowed from the docs. Calling some programs with a subprocess inside the executor causes a strange python error:

Fatal Python error: init_sys_streams: can't initialize sys standard streams
Python runtime state: core initialized
OSError: [Errno 9] Bad file descriptor

Current thread 0x00000001ec17cc00 (most recent call first):
  <no Python frame>

Going back to Python 3.10 on the same system with the same code, it runs as expected. Code also runs fine on Windows/Linux Python 3.11/3.12.

The example below calls jupyter in a subprocess, but swapping this to python --version or gcc --version works just fine... (Seems to be these python "shims" like jupyter --version or flake8 --version that are failing...)

Reproducing code:

import os
from time import sleep
from loky import get_reusable_executor
from subprocess import run


def call_program(k):
    pid = os.getpid()
    sleep(0.01)
    print(run(["jupyter", "--version"]).check_returncode())
    return pid

executor = get_reusable_executor(max_workers=4, timeout=2)
res = executor.submit(call_program, 1)

On MacOS, python 3.11, 3.12 this raises with:

Fatal Python error: init_sys_streams: can't initialize sys standard streams
Python runtime state: core initialized
OSError: [Errno 9] Bad file descriptor

Current thread 0x00000001ec17cc00 (most recent call first):
  <no Python frame>
---------------------------------------------------------------------------
_RemoteTraceback                          Traceback (most recent call last)
_RemoteTraceback:
"""
Traceback (most recent call last):
  File "/Users/python/envs/test311/lib/python3.11/site-packages/loky/process_executor.py", line 463, in _process_worker
    r = call_item()
        ^^^^^^^^^^^
  File "/Users/python/envs/test311/lib/python3.11/site-packages/loky/process_executor.py", line 291, in __call__
    return self.fn(*self.args, **self.kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<ipython-input-3-56017b8caf15>", line 9, in call_program
  File "/Users/python/envs/test311/lib/python3.11/subprocess.py", line 502, in check_returncode
    raise CalledProcessError(self.returncode, self.args, self.stdout,
subprocess.CalledProcessError: Command '['jupyter', '--version']' returned non-zero exit status 1.
"""

Tracing with python -m trace -t test.py shows the error:

fork_exec.py(26):      for i in open_fds - keep_fds:
fork_exec.py(41):          os.execve(sys.executable, cmd, child_env)
Fatal Python error: init_sys_streams: can't initialize sys standard streams
Python runtime state: core initialized
OSError: [Errno 9] Bad file descriptor

Current thread 0x00000001ec17cc00 (most recent call first):
  <no Python frame>

Environments created with:
mamba create --name test310 python=3.10 loky ipython jupyter
mamba create --name test311 python=3.11 loky ipython jupyter
mamba create --name test311 python=3.12 loky ipython jupyter

Any thoughts or ideas? Am I even in the right place!?

Thanks :)

[ryandvmartin]

Hoping some more triage might help. I installed in dev mode and through some trials noticed that if I add 0 to this list of fds to keep open:

    # Make sure to keep stdout and stderr open for logging purpose
    keep_fds = {*keep_fds, 0, 1, 2}

My error goes away. Perhaps that gives some hints at where the root of the issue might be?

[ogrisel]

Thanks for the report and the hint. I can still reproduce with Python 3.13 on macOS.

I will try to give it a look.

[ogrisel]

For reference, 0 stands for stdin and the spawn start method from the standard library does not have the problem:

import os
from time import sleep
from loky import get_reusable_executor
import multiprocessing as mp
from concurrent.futures import ProcessPoolExecutor
from subprocess import run


def call_program(k):
    pid = os.getpid()
    sleep(0.01)
    print(run(["jupyter", "--version"]).check_returncode())
    return pid


if __name__ == "__main__":
    # executor = get_reusable_executor(max_workers=4, timeout=2)
    executor = ProcessPoolExecutor(max_workers=4, mp_context=mp.get_context("spawn"))
    res = executor.submit(call_program, 1)

[ogrisel]

I took a look at the docstring of _posixsubprocess.fork_exec which is used by default with the "spawn" context method of the standard library, and indeed stdin is not closed.

Since we are switching to _posixsubprocess.fork_exec as part of #429, this should also resolve this problem.

I will try to wrap you reproducer as a non-regression test.

[ryandvmartin]

Thank you for looking into/resolving this.

Is anything else needed to get the ball rolling for this fix to be pulled into joblib ? Otherwise this issue can be resolved.

[ogrisel]

We need to release loky and then joblib. The loky release should happen soon.

[ogrisel]

Closing this issue since #429 was merged with the fix and a non-regression test.