Skip to content

Commit

Permalink
More updates
Browse files Browse the repository at this point in the history
  • Loading branch information
jan-janssen committed Nov 20, 2024
1 parent 343bd98 commit 9600e9a
Show file tree
Hide file tree
Showing 6 changed files with 33 additions and 189 deletions.
147 changes: 0 additions & 147 deletions docs/development.md

This file was deleted.

14 changes: 7 additions & 7 deletions docs/trouble_shooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,13 +16,13 @@ local mode on computers with strict firewall rules.

## Message Passing Interface
To use the message passing interface (MPI) executorlib requires [mpi4py](https://mpi4py.readthedocs.io/) as optional
dependency. The installation of this and other optional dependencies is covered in the [installation section]().
dependency. The installation of this and other optional dependencies is covered in the [installation section](https://executorlib.readthedocs.io/en/latest/installation.html#mpi-support).

## Missing Dependencies
The default installation of executorlib only comes with a limited number of dependencies, especially the [zero message queue](https://zeromq.org)
and [cloudpickle](https://github.com/cloudpipe/cloudpickle). Additional features like [caching](), [HPC submission mode]()
and [HPC allocation mode]() require additional dependencies. The dependencies are explained in more detail in the
[installation section]().
and [cloudpickle](https://github.com/cloudpipe/cloudpickle). Additional features like [caching](https://executorlib.readthedocs.io/en/latest/installation.html#caching), [HPC submission mode](https://executorlib.readthedocs.io/en/latest/installation.html#hpc-submission-mode)
and [HPC allocation mode](https://executorlib.readthedocs.io/en/latest/installation.html#hpc-allocation-mode) require additional dependencies. The dependencies are explained in more detail in the
[installation section](https://executorlib.readthedocs.io/en/latest/installation.html#).

## Python Version
Executorlib supports all current Python version ranging from 3.9 to 3.13. Still some of the dependencies and especially
Expand All @@ -38,9 +38,9 @@ The resource dictionary parameter `resource_dict` can contain one or more of the
* `openmpi_oversubscribe` (bool): adds the `--oversubscribe` command line flag (OpenMPI and SLURM only) - default False
* `slurm_cmd_args` (list): Additional command line arguments for the srun call (SLURM only)

For the special case of the [HPC allocation mode]() the resource dictionary parameter `resource_dict` can also include
additional parameters define in the submission script of the [Python simple queuing system adatper (pysqa)](https://pysqa.readthedocs.io)
these include but are not limited to:
For the special case of the [HPC allocation mode](https://executorlib.readthedocs.io/en/latest/3-hpc-allocation.html)
the resource dictionary parameter `resource_dict` can also include additional parameters define in the submission script
of the [Python simple queuing system adatper (pysqa)](https://pysqa.readthedocs.io) these include but are not limited to:
* `run_time_max` (int): the maximum time the execution of the submitted Python function is allowed to take in seconds.
* `memory_max` (int): the maximum amount of memory the Python function is allowed to use in Gigabytes.
* `partition` (str): the partition of the queuing system the Python function is submitted to.
Expand Down
8 changes: 3 additions & 5 deletions notebooks/1-local.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -230,7 +230,7 @@
"metadata": {},
"source": [
"In addition, to the compute cores `cores`, the resource dictionary parameter `resource_dict` can also define the threads per core as `threads_per_core`, the GPUs per core as `gpus_per_core`, the working directory with `cwd`, the option to use the OpenMPI oversubscribe feature with `openmpi_oversubscribe` and finally for the [Simple Linux Utility for Resource \n",
"Management (SLURM)](https://slurm.schedmd.com) queuing system the option to provide additional command line arguments with the `slurm_cmd_args` parameter - [resource dictionary]()."
"Management (SLURM)](https://slurm.schedmd.com) queuing system the option to provide additional command line arguments with the `slurm_cmd_args` parameter - [resource dictionary](https://executorlib.readthedocs.io/en/latest/trouble_shooting.html#resource-dictionary)."
]
},
{
Expand Down Expand Up @@ -325,7 +325,7 @@
"source": [
"For most cases MPI based parallelism leads to higher computational efficiency in comparison to thread based parallelism, still the choice of parallelism depends on the specific Python function which should be executed in parallel. Careful benchmarks are required to achieve the optimal performance for a given computational architecture. \n",
"\n",
"Beyond MPI based parallelism and thread based parallelism the [HPC Submission Mode]() and the [HPC Allocation Mode]() also provide the option to assign GPUs to the execution of individual Python functions. "
"Beyond MPI based parallelism and thread based parallelism the [HPC Submission Mode](https://executorlib.readthedocs.io/en/latest/2-hpc-submission.html) and the [HPC Allocation Mode](https://executorlib.readthedocs.io/en/latest/3-hpc-allocation.html) also provide the option to assign GPUs to the execution of individual Python functions. "
]
},
{
Expand Down Expand Up @@ -457,9 +457,7 @@
}
],
"source": [
"with Executor(\n",
" backend=\"local\", resource_dict={\"cores\": 2}, block_allocation=True\n",
") as exe:\n",
"with Executor(backend=\"local\", resource_dict={\"cores\": 2}, block_allocation=True) as exe:\n",
" fs = exe.submit(calc_mpi, 3)\n",
" print(fs.result())"
]
Expand Down
Loading

0 comments on commit 9600e9a

Please sign in to comment.