Skip to content

Commit

Permalink
Add HPC submission
Browse files Browse the repository at this point in the history
  • Loading branch information
@jan-janssen
jan-janssen committed Nov 19, 2024
1 parent bca3c32 commit b8ab12f
Show file tree
Hide file tree
Showing 2 changed files with 191 additions and 81 deletions.
67 changes: 42 additions & 25 deletions notebooks/1-local.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -49,8 +49,8 @@
"output_type": "stream",
"text": [
"2\n",
"CPU times: user 73.1 ms, sys: 49.3 ms, total: 122 ms\n",
"Wall time: 804 ms\n"
"CPU times: user 100 ms, sys: 70.7 ms, total: 171 ms\n",
"Wall time: 1.94 s\n"
]
}
],
Expand Down Expand Up @@ -80,8 +80,8 @@
"output_type": "stream",
"text": [
"[4, 6, 8]\n",
"CPU times: user 44.3 ms, sys: 27 ms, total: 71.3 ms\n",
"Wall time: 1.35 s\n"
"CPU times: user 49.4 ms, sys: 29.2 ms, total: 78.7 ms\n",
"Wall time: 1.75 s\n"
]
}
],
Expand Down Expand Up @@ -111,8 +111,8 @@
"output_type": "stream",
"text": [
"[10, 12, 14]\n",
"CPU times: user 32.9 ms, sys: 25.2 ms, total: 58.1 ms\n",
"Wall time: 968 ms\n"
"CPU times: user 40.5 ms, sys: 28.1 ms, total: 68.6 ms\n",
"Wall time: 1.09 s\n"
]
}
],
Expand Down Expand Up @@ -204,14 +204,6 @@
"Another option is to set the resource dictionary parameter `resource_dict` during the initialization of the `Executor`. In this case it is internally set for every call of the `submit()` function, without the need to specify it again."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "dcc06777-ae51-469b-bead-63ce402efe04",
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": 7,
Expand Down Expand Up @@ -351,7 +343,7 @@
"metadata": {},
"source": [
"### Block Allocation\n",
"By default each submitted Python function is executed in a dedicated process. This gurantees that the execution of the submitted Python function starts with a fresh process. Still the initialization of the Python process takes time. Especially when the call of the Python function requires only limited computational resources it makes sense to reuse the existing Python process for the execution of multiple Python functions. In executorlib this functionality is enabled by setting the `block_allocation` parameter to `Ture`."
"By default each submitted Python function is executed in a dedicated process. This gurantees that the execution of the submitted Python function starts with a fresh process. Still the initialization of the Python process takes time. Especially when the call of the Python function requires only limited computational resources it makes sense to reuse the existing Python process for the execution of multiple Python functions. In executorlib this functionality is enabled by setting the `block_allocation` parameter to `Ture`. To limit the number of parallel Python processes when using block allocation it is recommended to set the `max_workers` parameter to restrict the number of available computing resources. "
]
},
{
Expand All @@ -365,14 +357,14 @@
"output_type": "stream",
"text": [
"2\n",
"CPU times: user 34.5 ms, sys: 32.1 ms, total: 66.6 ms\n",
"Wall time: 1.26 s\n"
"CPU times: user 37.1 ms, sys: 21.8 ms, total: 58.9 ms\n",
"Wall time: 1.09 s\n"
]
}
],
"source": [
"%%time\n",
"with Executor(backend=\"local\", block_allocation=True) as exe:\n",
"with Executor(max_workers=2, backend=\"local\", block_allocation=True) as exe:\n",
" future = exe.submit(sum, [1, 1])\n",
" print(future.result())"
]
Expand Down Expand Up @@ -431,16 +423,41 @@
"experience performance degradation.\n",
"\n",
" Local host: MacBook-Pro.local\n",
" System call: unlink(2) /var/folders/z7/3vhrmssx60v240x_ndq448h80000gn/T//ompi.MacBook-Pro.501/pid.14134/1/vader_segment.MacBook-Pro.501.765b0001.1\n",
" System call: unlink(2) /var/folders/z7/3vhrmssx60v240x_ndq448h80000gn/T//ompi.MacBook-Pro.501/pid.22031/1/vader_segment.MacBook-Pro.501.17620001.1\n",
" Error: No such file or directory (errno 2)\n",
"--------------------------------------------------------------------------\n",
"--------------------------------------------------------------------------\n",
"A system call failed during shared memory initialization that should\n",
"not have. It is likely that your MPI job will now either abort or\n",
"experience performance degradation.\n",
"\n",
" Local host: MacBook-Pro.local\n",
" System call: unlink(2) /var/folders/z7/3vhrmssx60v240x_ndq448h80000gn/T//ompi.MacBook-Pro.501/pid.22028/1/vader_segment.MacBook-Pro.501.17610001.1\n",
" Error: No such file or directory (errno 2)\n",
"--------------------------------------------------------------------------\n",
"--------------------------------------------------------------------------\n",
"A system call failed during shared memory initialization that should\n",
"not have. It is likely that your MPI job will now either abort or\n",
"experience performance degradation.\n",
"\n",
" Local host: MacBook-Pro.local\n",
" System call: unlink(2) /var/folders/z7/3vhrmssx60v240x_ndq448h80000gn/T//ompi.MacBook-Pro.501/pid.22030/1/vader_segment.MacBook-Pro.501.17630001.1\n",
" Error: No such file or directory (errno 2)\n",
"--------------------------------------------------------------------------\n",
"--------------------------------------------------------------------------\n",
"A system call failed during shared memory initialization that should\n",
"not have. It is likely that your MPI job will now either abort or\n",
"experience performance degradation.\n",
"\n",
" Local host: MacBook-Pro.local\n",
" System call: unlink(2) /var/folders/z7/3vhrmssx60v240x_ndq448h80000gn/T//ompi.MacBook-Pro.501/pid.22029/1/vader_segment.MacBook-Pro.501.17600001.1\n",
" Error: No such file or directory (errno 2)\n",
"--------------------------------------------------------------------------\n"
]
}
],
"source": [
"with Executor(\n",
" backend=\"local\", resource_dict={\"cores\": 2}, block_allocation=True\n",
") as exe:\n",
"with Executor(backend=\"local\", resource_dict={\"cores\": 2}, block_allocation=True) as exe:\n",
" fs = exe.submit(calc_mpi, 3)\n",
" print(fs.result())"
]
Expand Down Expand Up @@ -535,7 +552,7 @@
"output_type": "stream",
"text": [
"[2, 4, 6]\n",
"CPU times: user 526 ms, sys: 134 ms, total: 661 ms\n",
"CPU times: user 547 ms, sys: 161 ms, total: 708 ms\n",
"Wall time: 1.33 s\n"
]
}
Expand Down Expand Up @@ -568,8 +585,8 @@
"output_type": "stream",
"text": [
"[2, 4, 6]\n",
"CPU times: user 41.4 ms, sys: 31.7 ms, total: 73.1 ms\n",
"Wall time: 989 ms\n"
"CPU times: user 52.1 ms, sys: 41.1 ms, total: 93.2 ms\n",
"Wall time: 1.13 s\n"
]
}
],
Expand Down
Loading

0 comments on commit b8ab12f

Please sign in to comment.