Multiprocessing IPC failing on Linux

[st-the-cloud]

I've been trying to adapt a Python program so that it runs cross-platform (original dev is on Windows). I kept encountering an issue where my subprocess (created via the multiprocessing module) would crash, and couldn't figure out why. I've stripped more and more away until I've isolated the issue to be caused by IPC; in the example I'll post I'm using a queue, but the same thing happens with a pipe. The code runs fine if I omit the call to 'queue.put()' (and it works fine when run unfrozen). The subprocess doesn't crash as soon as that call is made, it happens shortly after (under a second after), and I am unable to get any information about why its terminated. I've also experimented with different start modes (spawn, forkserver) and that doesn't resolve the issue either.

Here is my code:

import json
import time
import signal
import sys
import os
import platform
from pathlib import Path
import multiprocessing
from queue import Empty
import logging
import tempfile


class SignalHandler:
    shutdown_requested = False

    def __init__(self):
        signal.signal(signal.SIGINT, self.request_shutdown)
        signal.signal(signal.SIGTERM, self.request_shutdown)

    def request_shutdown(self, *args):
        self.shutdown_requested = True

    def can_run(self):
        return not self.shutdown_requested


def main():
    signal_handler = SignalHandler()
    queue = multiprocessing.Queue()

    loopy_process = multiprocessing.Process(target=loopy,
                                            args=(queue,))

    loopy_process.start()

    while signal_handler.can_run():
        print('before checking process alive')
        try:
            alive = loopy_process.is_alive()
        except Exception as e:
            print(f'error checking alive; {e}')
            alive = False
        if not alive:
            print('loopy crashed')
            break
        try:
            data = queue.get_nowait()
            print(data)
        except Exception as e:
            print(e)
        time.sleep(1)
        print('end of while loop')

    try:
        data = queue.get_nowait()
        print(data)
    except Exception as e:
        print(e)
    queue.close()

    if loopy_process.is_alive():
        loopy_process.kill()
        loopy_process.join()

    print(f'Terminating')


def loopy(queue):
    log_path = os.path.join(tempfile.gettempdir(), 'mp_debug.log')
    logger = multiprocessing.get_logger()
    logger.setLevel(logging.DEBUG)
    handler = logging.FileHandler(log_path)
    handler.setFormatter(logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s'))
    logger.addHandler(handler)
    logger.debug('Logging set up, about the enter while loop')
    while True:
        try:
            logger.debug('About to put into queue')
            queue.put('loopy running')
            logger.debug('Put it in the queue')
        except Exception as e:
            logger.error(f'Put failed; {e}')
        logger.debug('About to sleep')
        time.sleep(1)
        logger.debug('Slept')


if __name__ == "__main__":
    print('entered name=main')
    multiprocessing.freeze_support()

    try:
        main()
    except Exception as error:
        print(f"Service stopped: {error}")
    sys.exit(1)

The console output is as follows:

entered name=main
before checking process alive

end of while loop
before checking process alive
loopy crashed

Terminating

The log file contains:

2024-11-04 17:40:05,125 - multiprocessing - DEBUG - Logging set up, about the enter while loop
2024-11-04 17:40:05,125 - multiprocessing - DEBUG - About to put into queue
2024-11-04 17:40:05,125 - multiprocessing - DEBUG - Queue._start_thread()
2024-11-04 17:40:05,125 - multiprocessing - DEBUG - doing self._thread.start()
2024-11-04 17:40:05,125 - multiprocessing - DEBUG - starting thread to feed data to pipe
2024-11-04 17:40:05,126 - multiprocessing - DEBUG - ... done self._thread.start()
2024-11-04 17:40:05,126 - multiprocessing - DEBUG - Put it in the queue
2024-11-04 17:40:05,126 - multiprocessing - DEBUG - About to sleep

And finally I am freezing it using the following arguments:
pyinstaller --hidden-import asyncio --hidden-import uuid --hidden-import secrets --hidden-import queue test_service_mini.py

Environment details:
Pyinstaller 6.11.0
Python 3.12.3
OS: Oracle Linux Server 8.10 (EL8-based)
Linux is running on a VM via Oracle VM VirtualBox Manager, on a Windows 11 Pro 23H2 machine.

[rokm]

Hmm, seems to work as expected on my Fedora 41 system (built with python 3.12.7 from python3.12 package instead of default 3.13.0). Can you reproduce this across different distributions (and/or python versions), or is it OL8-specific?

[rokm]

Also seems to work with RHEL8 VM running under VirtualBox (on the afore-mentioned linux host); albeit that pulled python 3.12.5 for dnf install python3.12 instead of 3.12.3 that you reported...

[st-the-cloud]

Also seems to work with RHEL8 VM running under VirtualBox (on the afore-mentioned linux host); albeit that pulled python 3.12.5 for dnf install python3.12 instead of 3.12.3 that you reported...

Ahh so I've just tried it with 3.12.5 instead of 3.12.3 (everything else the same) and it works fine! Must be a bug specific to that version then maybe? Thanks for your help