Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feature] Serialise executors #260

Open
jan-janssen opened this issue Feb 7, 2024 · 3 comments
Open

[feature] Serialise executors #260

jan-janssen opened this issue Feb 7, 2024 · 3 comments

Comments

@jan-janssen
Copy link
Member

As the Executors in pympipool typically use zmq to connect the executor to the process which is executing the functions it should in principle be possible to serialise the executor - store the information of the executor - and then deserialise the same object again. Still it is not clear what happens to the messages already submitted and in addition it is also not clear if the server process is started by the executor or by the process executing the python functions.

@jan-janssen
Copy link
Member Author

Currently the Executor starts the zmq and the process which is executing the tasks is just connecting to it. This has to be changed to be able to serialise the Executor object - https://github.com/pyiron/pympipool/blob/main/pympipool/backend/serial.py

@jan-janssen
Copy link
Member Author

The flux --parent command allows flux to connect to a parent instance rather than creating a sub instance - https://flux-framework.readthedocs.io/projects/flux-core/en/latest/man1/flux.html#cmdoption-flux-p

@jan-janssen
Copy link
Member Author

Nested flux execution was implemented in #382

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant