Docs suggest that loky.set_start_method exists

[sjdv1982]

but the actual API seems to be loky.backend.context.set_start_method

[ogrisel]

I agree, however, I don't really see the point of changing the start method in loky.

I think I would favor:

• not implying that this is public API by deleting the section mentioning set_start_method in the API doc;
• deprecating the old "loky_init_main" start method, which is probably useless nowadays.

Similarly, now that cloudpickle is based the C-implementation of pickle, it is almost always as fast as the "pickle" module and our documentation, the narrative of this example has become meaningless:

https://loky.readthedocs.io/en/stable/auto_examples/cloudpickle_wrapper.html

We could also likely deprecate loky.wrap_non_picklable_objects because I do not see any usecase for this hackish tool anymore.

We should accordingly update https://loky.readthedocs.io/en/stable/API.html#task-results-serialization

As this library is slower than the pickle module in the standard library, by default, loky uses it only to serialize objects which are detected to be in the main module.

• There is no speed tradeoff anymore;
• I cannot see in the code where we would restrict the use of cloudpickle to only things defined in __main__. I think we use it for everything, e.g. non-picklable locally defined functions:

Source code of local_module.py:

def make_local_function():
    def local_function():
        return "ok"
    return local_function

In [1]: import local_module

In [2]: f = local_module.make_local_function()

In [3]: f.__qualname__
Out[3]: 'make_local_function.<locals>.local_function'

In [4]: import pickle

In [5]: pickle.dumps(f)
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[5], line 1
----> 1 pickle.dumps(f)

AttributeError: Can't get local object 'make_local_function.<locals>.local_function'

In [6]: from loky import get_reusable_executor

In [7]: e = get_reusable_executor()

In [8]: e.submit(f).result()
Out[8]: 'ok'