diff --git a/README.md b/README.md index 1206b284..a1516702 100644 --- a/README.md +++ b/README.md @@ -1,110 +1,152 @@ -# pympipool - scale python functions over multiple compute nodes +# pympipool - up-scale python functions for high performance computing [![Unittests](https://github.com/pyiron/pympipool/actions/workflows/unittest-openmpi.yml/badge.svg)](https://github.com/pyiron/pympipool/actions/workflows/unittest-openmpi.yml) [![Coverage Status](https://coveralls.io/repos/github/pyiron/pympipool/badge.svg?branch=main)](https://coveralls.io/github/pyiron/pympipool?branch=main) -Up-scaling python functions for high performance computing (HPC) can be challenging. While the python standard library -provides interfaces for multiprocessing and asynchronous task execution, namely [`multiprocessing`](https://docs.python.org/3/library/multiprocessing.html) -and [`concurrent.futures`](https://docs.python.org/3/library/concurrent.futures.html#module-concurrent.futures) both are -limited to the execution on a single compute node. So a series of python libraries have been developed to address the -up-scaling of python functions for HPC. Starting in the datascience and machine learning community with solutions like -[dask](https://www.dask.org) over more HPC focused solutions like [parsl](http://parsl-project.org) up to Python bindings -for the message passing interface (MPI) named [mpi4py](https://mpi4py.readthedocs.io). Each of these solutions has their -advantages and disadvantages, in particular the mixing of MPI parallel python functions and serial python functions in -combined workflows remains challenging. - -To address these challenges `pympipool` is developed with three goals in mind: -* Reimplement the standard python library interfaces namely [`multiprocessing.pool.Pool`](https://docs.python.org/3/library/multiprocessing.html) -and [`concurrent.futures.Executor`](https://docs.python.org/3/library/concurrent.futures.html#module-concurrent.futures) -as closely as possible, to minimize the barrier of up-scaling an existing workflow to be used on HPC resources. -* Integrate MPI parallel python functions based on [mpi4py](https://mpi4py.readthedocs.io) on the same level as serial -python functions, so both can be combined in a single workflow. This allows the users to parallelize their workflows -one function at a time. Internally this is achieved by coupling a serial python process to a MPI parallel python process. -* Embrace [Jupyter](https://jupyter.org) notebooks for the interactive development of HPC workflows, as they allow the -users to document their though process right next to the python code and their results all within one document. - -# Features -As different users and different workflows have different requirements in terms of the level of parallelization, the -`pympipool` implements a series of five different interfaces: -* [`pympipool.Pool`](https://pympipool.readthedocs.io/en/latest/interfaces.html#pool): Following the -[`multiprocessing.pool.Pool`](https://docs.python.org/3/library/multiprocessing.html) the `pympipool.Pool` class -implements the `map()` and `starmap()` functions. Internally these connect to an MPI parallel subprocess running the -[`mpi4py.futures.MPIPoolExecutor`](https://mpi4py.readthedocs.io/en/stable/mpi4py.futures.html#mpipoolexecutor). -So by increasing the number of workers, by setting the `max_workers` parameter the `pympipool.Pool` can scale the -execution of serial python functions beyond a single compute node. For MPI parallel python functions the `pympipool.MPISpawnPool` -is derived from the `pympipool.Pool` and uses `MPI_Spawn()` to execute those. For more details see below. -* [`pympipool.Executor`](https://pympipool.readthedocs.io/en/latest/interfaces.html#executor): The easiest way to -execute MPI parallel python functions right next to serial python functions is the `pympipool.Executor`. It implements -the executor interface defined by the [`concurrent.futures.Executor`](https://docs.python.org/3/library/concurrent.futures.html#module-concurrent.futures). -So functions are submitted to the `pympipool.Executor` using the `submit()` function, which returns an -[`concurrent.futures.Future`](https://docs.python.org/3/library/concurrent.futures.html#future-objects) object. With -these [`concurrent.futures.Future`](https://docs.python.org/3/library/concurrent.futures.html#future-objects) objects -asynchronous workflows can be constructed which periodically check if the computation is completed `done()` and then -query the results using the `result()` function. The limitation of the `pympipool.Executor` is lack of load balancing, -each `pympipool.Executor` acts as a serial first in first out (FIFO) queue. So it is the task of the user to balance the -load of many different tasks over multiple `pympipool.Executor` instances. -* [`pympipool.HPCExecutor`](https://pympipool.readthedocs.io/en/latest/interfaces.html#hpcexecutor>): To address the -limitation of the `pympipool.Executor` that only a single task is executed at any time, the `pympipool.HPCExecutor` -provides a wrapper around multiple `pympipool.Executor` objects. It balances the queues of the individual -`pympipool.Executor` objects to maximize the throughput for the given resources. This functionality comes with an -additional overhead of another thread, acting as a broker between the task queue of the `pympipool.HPCExecutor` and the -individual `pympipool.Executor` objects. -* [`pympipool.PoolExecutor`](https://pympipool.readthedocs.io/en/latest/interfaces.html#poolexecutor): To combine the -functionality of the `pympipool.Pool` and the `pympipool.Executor` the `pympipool.PoolExecutor` again connects to the -[`mpi4py.futures.MPIPoolExecutor`](https://mpi4py.readthedocs.io/en/stable/mpi4py.futures.html#mpipoolexecutor). -Still in contrast to the `pympipool.Pool` it does not implement the `map()` and `starmap()` functions but rather the -`submit()` function based on the [`concurrent.futures.Executor`](https://docs.python.org/3/library/concurrent.futures.html#module-concurrent.futures) -interface. In this case the load balancing happens internally and the maximum number of workers `max_workers` defines -the maximum number of parallel tasks. But only serial python tasks can be executed in contrast to the `pympipool.Executor` -which can also execute MPI parallel python tasks. -* [`pympipool.MPISpawnPool`](https://pympipool.readthedocs.io/en/latest/interfaces.html#mpispawnpool): An alternative -way to support MPI parallel functions in addition to the `pympipool.Executor` is the `pympipool.MPISpawnPool`. Just like -the `pympipool.Pool` it supports the `map()` and `starmap()` functions. The additional `ranks_per_task` parameter -defines how many MPI ranks are used per task. All functions are executed with the same number of MPI ranks. The -limitation of this approach is that it uses `MPI_Spawn()` to create new MPI ranks for the execution of the individual -tasks. Consequently, this approach is not as scalable as the `pympipool.Executor` but it offers load balancing for a -large number of similar MPI parallel tasks. -* [`pympipool.SocketInterface`](https://pympipool.readthedocs.io/en/latest/interfaces.html#socketinterface): The key -functionality of the `pympipool` package is the coupling of a serial python process with an MPI parallel python process. -This happens in the background using a combination of the [zero message queue](https://zeromq.org) and -[cloudpickle](https://github.com/cloudpipe/cloudpickle) to communicate binary python objects. The `pympipool.SocketInterface` -is an abstraction of this interface, which is used in the other classes inside `pympipool` and might also be helpful for -other projects. - -In addition to using MPI to start a number of processes on different HPC computing resources, `pympipool` also supports -the [flux-framework](https://flux-framework.org) as additional backend. By setting the optional `enable_flux_backend` -parameter to `True` the flux backend can be enabled for the `pympipool.Pool`, `pympipool.Executor` and `pympipool.PoolExecutor`. -Other optional parameters include the selection of the working directory where the python function should be executed `cwd` -and the option to oversubscribe MPI tasks which is an [OpenMPI](https://www.open-mpi.org) specific feature which can be -enabled by setting `oversubscribe` to `True`. For more details on the `pympipool` classes and their application, the -extended documentation is linked below. +Up-scaling python functions for high performance computing (HPC) can be challenging. While the python standard library +provides interfaces for multiprocessing and asynchronous task execution, namely +[multiprocessing](https://docs.python.org/3/library/multiprocessing.html) and +[concurrent.futures](https://docs.python.org/3/library/concurrent.futures.html#module-concurrent.futures) both are +limited to the execution on a single compute node. So a series of python libraries have been developed to address the +up-scaling of python functions for HPC. Starting in the datascience and machine learning community with solutions +like [dask](https://www.dask.org) over more HPC focused solutions like +[fireworks](https://materialsproject.github.io/fireworks/) and [parsl](http://parsl-project.org) up to Python +bindings for the message passing interface (MPI) named [mpi4py](https://mpi4py.readthedocs.io). Each of these +solutions has their advantages and disadvantages, in particular scaling beyond serial python functions, including thread +based parallelism, MPI parallel python application or assignment of GPUs to individual python function remains +challenging. + +To address these challenges `pympipool` is developed with three goals in mind: + +* Extend the standard python library [`concurrent.futures.Executor`](https://docs.python.org/3/library/concurrent.futures.html#module-concurrent.futures) + interface, to minimize the barrier of up-scaling an existing workflow to be used on HPC resources. +* Integrate thread based parallelism, MPI parallel python functions based on [mpi4py](https://mpi4py.readthedocs.io) and + GPU assignment. This allows the users to accelerate their workflows one function at a time. +* Embrace [Jupyter](https://jupyter.org) notebooks for the interactive development of HPC workflows, as they allow the + users to document their though process right next to the python code and their results all within one document. + +## HPC Context +In contrast to frameworks like [dask](https://www.dask.org), [fireworks](https://materialsproject.github.io/fireworks/) +and [parsl](http://parsl-project.org) which can be used to submit a number of worker processes directly the the HPC +queuing system and then transfer tasks from either the login node or an interactive allocation to these worker processes +to accelerate the execution, [mpi4py](https://mpi4py.readthedocs.io) and `pympipool` follow a different +approach. Here the user creates their HPC allocation first and then [mpi4py](https://mpi4py.readthedocs.io) or +`pympipool` can be used to distribute the tasks within this allocation. The advantage of this approach is that +no central data storage is required as the workers and the scheduling task can communicate directly. + +## Examples +The following examples illustrates how `pympipool` can be used to distribute a series of MPI parallel function calls +within a queuing system allocation. `example.py`: +``` +from pympipool import Executor + +def calc(i): + from mpi4py import MPI + size = MPI.COMM_WORLD.Get_size() + rank = MPI.COMM_WORLD.Get_rank() + return i, size, rank + +with Executor(max_workers=2, cores_per_worker=2) as exe: + fs_0 = exe.submit(calc, 0) + fs_1 = exe.submit(calc, 1) + print(fs_0.result(), fs_1.result()) +``` +This example can be executed using:: +``` +python example.py +``` +Which returns:: +``` +>>> [(0, 2, 0), (0, 2, 1)], [(1, 2, 0), (1, 2, 1)] +``` +The important part in this example is that [mpi4py](https://mpi4py.readthedocs.io) is only used in the `calc()` +function, not in the python script, consequently it is not necessary to call the script with `mpiexec` but instead +a call with the regular python interpreter is sufficient. This highlights how `pympipool` allows the users to +parallelize one function at a time and not having to convert their whole workflow to use [mpi4py](https://mpi4py.readthedocs.io). +The same code can also be executed inside a jupyter notebook directly which enables an interactive development process. + +The standard [`concurrent.futures.Executor`](https://docs.python.org/3/library/concurrent.futures.html#module-concurrent.futures) +interface is extended by adding the option `cores_per_worker=2` to assign multiple MPI ranks to each function call. +To create two workers `max_workers=2` each with two cores each requires a total of four CPU cores to be available. +After submitting the function `calc()` with the corresponding parameter to the executor `exe.submit(calc, 0)` +a python [`concurrent.futures.Future`](https://docs.python.org/3/library/concurrent.futures.html#future-objects) is +returned. Consequently, the `pympipool.Executor` can be used as a drop-in replacement for the +[`concurrent.futures.Executor`](https://docs.python.org/3/library/concurrent.futures.html#module-concurrent.futures) +which allows the user to add parallelism to their workflow one function at a time. + +## Backends +Depending on the availability of different resource schedulers in your HPC environment the `pympipool.Executor` +uses a different backend, with the `pympipool.flux.PyFluxExecutor` being the preferred backend: + +* `pympipool.mpi.PyMpiExecutor`: The simplest executor of the three uses [mpi4py](https://mpi4py.readthedocs.io) as a + backend. This simplifies the installation on all operating systems including Windows. Still at the same time it limits + the up-scaling to a single compute node and serial or MPI parallel python functions. There is no support for thread + based parallelism or GPU assignment. This interface is primarily used for testing and developing or as a fall-back + solution. It is not recommended to use this interface in production. +* `pympipool.slurm.PySlurmExecutor`: The [SLURM workload manager](https://www.schedmd.com) is commonly used on HPC + systems to schedule and distribute tasks. `pympipool` provides a python interface for scheduling the execution of + python functions as SLURM job steps which are typically created using the `srun` command. This executor supports + serial python functions, thread based parallelism, MPI based parallelism and the assignment of GPUs to individual + python functions. When the [SLURM workload manager](https://www.schedmd.com) is installed on your HPC cluster this + interface can be a reasonable choice, still depending on the [SLURM workload manager](https://www.schedmd.com) + configuration in can be limited in terms of the fine-grained scheduling or the responsiveness when working with + hundreds of compute nodes in an individual allocation. +* `pympipool.flux.PyFluxExecutor`: The [flux framework]](https://flux-framework.org) is the preferred backend for + `pympipool`. Just like the `pympipool.slurm.PySlurmExecutor` it supports serial python functions, thread based + parallelism, MPI based parallelism and the assignment of GPUs to individual python functions. Still the advantages of + using the [flux framework](https://flux-framework.org) as a backend are the easy installation, the faster allocation + of resources as the resources are managed within the allocation and no central databases is used and the superior + level of fine-grained resource assignment which is typically not available on HPC resource schedulers. + +Each of these backends consists of two parts a broker and a worker. When a new tasks is submitted from the user it is +received by the broker and the broker identifies the first available worker. The worker then executes a task and returns +it to the broker, who returns it to the user. While there is only one broker per `pympipool.Executor` the number +of workers can be specified with the `max_workers` parameter. + +## Disclaimer +While we try to develop a stable and reliable software library, the development remains a opensource project under the +BSD 3-Clause License without any warranties:: +``` +BSD 3-Clause License + +Copyright (c) 2022, Jan Janssen +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +* Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +``` # Documentation -* [Installation](https://pympipool.readthedocs.io/en/latest/installation.html) - * [pypi-based installation](https://pympipool.readthedocs.io/en/latest/installation.html#pypi-based-installation) - * [conda-based installation](https://pympipool.readthedocs.io/en/latest/installation.html#conda-based-installation) -* [Interfaces](https://pympipool.readthedocs.io/en/latest/interfaces.html) - * [Pool](https://pympipool.readthedocs.io/en/latest/interfaces.html#pool) - * [Executor](https://pympipool.readthedocs.io/en/latest/interfaces.html#executor) - * [HPCExecutor](https://pympipool.readthedocs.io/en/latest/interfaces.html#hpcexecutor>) - * [ParallelExecutor](https://pympipool.readthedocs.io/en/latest/interfaces.html#poolexecutor) - * [MPISpawnPool](https://pympipool.readthedocs.io/en/latest/interfaces.html#mpispawnpool) - * [SocketInterface](https://pympipool.readthedocs.io/en/latest/interfaces.html#socketinterface) -* [Development](https://pympipool.readthedocs.io/en/latest/development.html) - -# License -`pympipool` is released under the BSD license https://github.com/pyiron/pympipool/blob/main/LICENSE . It is a spin-off -of the `pyiron` project https://github.com/pyiron/pyiron therefore if you use `pympipool` for calculation which result -in a scientific publication, please cite: - - @article{pyiron-paper, - title = {pyiron: An integrated development environment for computational materials science}, - journal = {Computational Materials Science}, - volume = {163}, - pages = {24 - 36}, - year = {2019}, - issn = {0927-0256}, - doi = {https://doi.org/10.1016/j.commatsci.2018.07.043}, - url = {http://www.sciencedirect.com/science/article/pii/S0927025618304786}, - author = {Jan Janssen and Sudarsan Surendralal and Yury Lysogorskiy and Mira Todorova and Tilmann Hickel and Ralf Drautz and Jörg Neugebauer}, - keywords = {Modelling workflow, Integrated development environment, Complex simulation protocols}, - } +* [Installation](https://pympipool.readthedocs.io/en/latest/installation.html) + * [Basic Installation](https://pympipool.readthedocs.io/en/latest/installation.html#basic-installation) + * [High Performance Computing](https://pympipool.readthedocs.io/en/latest/installation.html#high-performance-computing) +* [Examples](https://pympipool.readthedocs.io/en/latest/examples.html) + * [Compatibility](https://pympipool.readthedocs.io/en/latest/examples.html#compatibility) + * [Data Handling](https://pympipool.readthedocs.io/en/latest/examples.html#data-handling) + * [Up-Scaling](https://pympipool.readthedocs.io/en/latest/examples.html#up-scaling) +* [Development](https://pympipool.readthedocs.io/en/latest/development.html) + * [Contributions](https://pympipool.readthedocs.io/en/latest/development.html#contributions) + * [Integration](https://pympipool.readthedocs.io/en/latest/development.html#integration) + * [Alternative Projects](https://pympipool.readthedocs.io/en/latest/development.html#alternative-projects) +* [Module Index](https://pympipool.readthedocs.io/en/latest/py-modindex.html) \ No newline at end of file diff --git a/docs/source/installation.md b/docs/source/installation.md index 7461aa89..ccbb3069 100644 --- a/docs/source/installation.md +++ b/docs/source/installation.md @@ -32,7 +32,7 @@ following command: pip install pympipool ``` -## Integration with high performance computing +## High Performance Computing `pympipool` currently provides interfaces to the [SLURM workload manager](https://www.schedmd.com) and the [flux framework](https://flux-framework.org). With the [flux framework](https://flux-framework.org) being the recommended solution as it can be installed without root user rights and it can be integrated in existing resource