Parallelization

[mizimmer90]

We need to figure out a standardized parallelization protocol

[Justin-J-Miller]

Wanted to re-up discussion on this. We've not historically had issues using multiprocessing.Pool(), but this launches processes as forks which can hang indefinitely. Starting with python 3.12, launching processes in this way throws a warning, and starting with python 3.14 the default will switch away from launching with fork to either spawn or forkserver.

We could do the following at the top of each script that launches things from Pool to switch everything from fork to spawn.

from multiprocessing import set_start_method
set_start_method("spawn")

Though I'm sure how big of a performance hit (both with memory and throughput) we might take with that.

[justinrporter]

I have not been keeping track of the changes to python’s parallelization tools in the last few years but I generally disliked pools for a variety of reasons (including the ones you allude to). However, it may or may not be a good use of time to overhaul the code base unless there is a pressing reason to…I guess what I’m saying is there is no special reason that pool was used at the time, except that it was easy.JustinOn Jul 25, 2024, at 9:57 AM, Justin Miller ***@***.***> wrote: Wanted to re-up discussion on this. We've not historically had issues using multiprocessing.Pool(), but this launches processes as forks which can hang indefinitely. Starting with python 3.12, launching processes in this way throws a warning, and starting with python 3.14 the default will switch away from launching with fork to either spawn or forkserver. We could do the following at the top of each script that launches things from Pool to switch everything from fork to spawn. from multiprocessing import set_start_method set_start_method("spawn") Though I'm sure how big of a performance hit (both with memory and throughput) we might take with that. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were assigned.Message ID: ***@***.***>