Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Transport & Engine: AsyncTransport plugin #6626

Open
wants to merge 36 commits into
base: main
Choose a base branch
from
Open

Transport & Engine: AsyncTransport plugin #6626

Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
b59d999
♻️ Allow for file uploads/downloads to be async
chrisjsewell Jul 9, 2023
0c42841
Update test_execmanager.py
chrisjsewell Jul 9, 2023
6cb4d9e
Merge remote-tracking branch 'upstream/main' into async-run
chrisjsewell Jul 9, 2023
14cbd29
update run methods
chrisjsewell Jul 9, 2023
e9361bb
Merge branch 'main' into async-transport
khsrali Nov 18, 2024
6811098
async transport, the first implementation
khsrali Nov 18, 2024
5fdde51
asynchrounous counterparts are added to transport.py
khsrali Nov 18, 2024
ccc545e
Giovanni's review applied
khsrali Nov 19, 2024
f187fdc
adopted tests
khsrali Nov 19, 2024
565724d
docstring updated
khsrali Nov 20, 2024
6e350e7
added computer test for ssh_async
khsrali Nov 21, 2024
03ccc30
review applied
khsrali Nov 25, 2024
178bf7b
chnage from machine to machine_
khsrali Nov 27, 2024
cc0bc5c
review applied
khsrali Nov 29, 2024
3210c27
copy-remote adopted with behaviour of asyncssh
khsrali Dec 4, 2024
76aaf53
Merge branch 'main' into async-transport
khsrali Dec 4, 2024
65f0663
remove str() use from test_all_plugins
khsrali Dec 4, 2024
a809b98
copy() are now aligned with fresh development on asyncssh
khsrali Dec 5, 2024
38cfc24
fixed some stupid issues
khsrali Dec 5, 2024
799e0f8
plumpy hook pointing to async-run branch, now
khsrali Dec 5, 2024
665a163
Merge branch 'main' into async-transport
khsrali Dec 5, 2024
0837193
updated uv lock
khsrali Dec 5, 2024
1b96110
Fixing uv.lock file for the depedencies from a github repo
agoscinski Dec 5, 2024
3aa0031
Merge branch 'main' into async-transport
khsrali Dec 6, 2024
a68240c
fix conflicts
khsrali Dec 6, 2024
520e58e
fixed afew self blocking calls in copy_async()
khsrali Dec 10, 2024
22eb929
Merge branch 'main' into async-transport
khsrali Dec 11, 2024
90718f4
fix rtd
khsrali Dec 11, 2024
343cf9c
fix uv
khsrali Dec 11, 2024
482eeca
escape for bash on command
khsrali Dec 11, 2024
5e29e5b
fixed many warnings of rtd
khsrali Dec 11, 2024
a5ff84d
Merge branch 'main' into async-transport
khsrali Dec 11, 2024
1761d94
implement max_io_allowed
khsrali Dec 13, 2024
cf01ac0
Merge branch 'main' into async-transport
khsrali Dec 13, 2024
6627b21
update asyncssh dependency
khsrali Dec 13, 2024
2ebf945
plumpy dependency pin the exact commit
khsrali Dec 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion environment.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ dependencies:
- python~=3.9
- alembic~=1.2
- archive-path~=0.4.2
- asyncssh~=2.18.0
- circus~=0.18.0
- click-spinner~=0.1.8
- click~=8.1
Expand All @@ -22,7 +23,7 @@ dependencies:
- importlib-metadata~=6.0
- numpy~=1.21
- paramiko~=3.0
- plumpy~=0.22.3
- plumpy@ git+https://github.com/khsrali/plumpy.git@allow-async-upload-download#egg=plumpy
- pgsu~=0.3.0
- psutil~=5.6
- psycopg[binary]~=3.0
Expand Down
5 changes: 4 additions & 1 deletion pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ classifiers = [
dependencies = [
'alembic~=1.2',
'archive-path~=0.4.2',
'asyncssh~=2.18.0',
'circus~=0.18.0',
'click-spinner~=0.1.8',
'click~=8.1',
Expand All @@ -34,7 +35,7 @@ dependencies = [
'importlib-metadata~=6.0',
'numpy~=1.21',
'paramiko~=3.0',
'plumpy~=0.22.3',
'plumpy@git+https://github.com/khsrali/plumpy.git@allow-async-upload-download#egg=plumpy',
'pgsu~=0.3.0',
'psutil~=5.6',
'psycopg[binary]~=3.0',
Expand Down Expand Up @@ -174,6 +175,7 @@ requires-python = '>=3.9'
[project.entry-points.'aiida.transports']
'core.local' = 'aiida.transports.plugins.local:LocalTransport'
'core.ssh' = 'aiida.transports.plugins.ssh:SshTransport'
'core.ssh_async' = 'aiida.transports.plugins.ssh_async:AsyncSshTransport'
'core.ssh_auto' = 'aiida.transports.plugins.ssh_auto:SshAutoTransport'

[project.entry-points.'aiida.workflows']
Expand Down Expand Up @@ -305,6 +307,7 @@ module = 'tests.*'
ignore_missing_imports = true
module = [
'ase.*',
'asyncssh.*',
'bpython.*',
'bs4.*',
'CifFile.*',
Expand Down
3 changes: 2 additions & 1 deletion requirements/requirements-py-3.10.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ ase==3.22.1
asn1crypto==1.5.1
asttokens==2.2.1
async-generator==1.10
asyncssh~=2.18.0
attrs==23.1.0
babel==2.12.1
backcall==0.2.0
Expand Down Expand Up @@ -120,7 +121,7 @@ pillow==9.5.0
platformdirs==3.6.0
plotly==5.15.0
pluggy==1.0.0
plumpy==0.22.3
plumpy@git+https://github.com/khsrali/plumpy.git@allow-async-upload-download#egg=plumpy
prometheus-client==0.17.0
prompt-toolkit==3.0.38
psutil==5.9.5
Expand Down
3 changes: 2 additions & 1 deletion requirements/requirements-py-3.11.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ ase==3.22.1
asn1crypto==1.5.1
asttokens==2.2.1
async-generator==1.10
asyncssh~=2.18.0
attrs==23.1.0
babel==2.12.1
backcall==0.2.0
Expand Down Expand Up @@ -119,7 +120,7 @@ pillow==9.5.0
platformdirs==3.6.0
plotly==5.15.0
pluggy==1.0.0
plumpy==0.22.3
plumpy@git+https://github.com/khsrali/plumpy.git@allow-async-upload-download#egg=plumpy
prometheus-client==0.17.0
prompt-toolkit==3.0.38
psutil==5.9.5
Expand Down
3 changes: 2 additions & 1 deletion requirements/requirements-py-3.12.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ ase==3.22.1
asn1crypto==1.5.1
asttokens==2.4.0
async-generator==1.10
asyncssh~=2.18.0
attrs==23.1.0
babel==2.13.1
backcall==0.2.0
Expand Down Expand Up @@ -119,7 +120,7 @@ pillow==10.1.0
platformdirs==3.11.0
plotly==5.17.0
pluggy==1.3.0
plumpy==0.22.3
plumpy@git+https://github.com/khsrali/plumpy.git@allow-async-upload-download#egg=plumpy
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will you make a PR there so we can do a new release?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes! Please review here: aiidateam/plumpy#272

prometheus-client==0.17.1
prompt-toolkit==3.0.39
psutil==5.9.6
Expand Down
3 changes: 2 additions & 1 deletion requirements/requirements-py-3.9.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ ase==3.22.1
asn1crypto==1.5.1
asttokens==2.2.1
async-generator==1.10
asyncssh~=2.18.0
attrs==23.1.0
babel==2.12.1
backcall==0.2.0
Expand Down Expand Up @@ -122,7 +123,7 @@ pillow==9.5.0
platformdirs==3.6.0
plotly==5.15.0
pluggy==1.0.0
plumpy==0.22.3
plumpy@git+https://github.com/khsrali/plumpy.git@allow-async-upload-download#egg=plumpy
prometheus-client==0.17.0
prompt-toolkit==3.0.38
psutil==5.9.5
Expand Down
5 changes: 3 additions & 2 deletions src/aiida/calculations/monitors/base.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,13 @@

import tempfile
from pathlib import Path
from typing import Union

from aiida.orm import CalcJobNode
from aiida.transports import Transport
from aiida.transports import AsyncTransport, BlockingTransport


def always_kill(node: CalcJobNode, transport: Transport) -> str | None:
def always_kill(node: CalcJobNode, transport: Union['BlockingTransport', 'AsyncTransport']) -> str | None:
"""Retrieve and inspect files in working directory of job to determine whether the job should be killed.

This particular implementation is just for demonstration purposes and will kill the job as long as there is a
Expand Down
Loading
Loading