Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add generic SODAR ingest command #199

Merged
merged 72 commits into from
Jan 9, 2024
Merged
Changes from 1 commit
Commits
Show all changes
72 commits
Select commit Hold shift + click to select a range
95c76ab
feat: Common functions for interfacing with python-irodsclient
sellth Nov 7, 2023
6b72d9d
fix tests
sellth Nov 7, 2023
2636c8c
fix chksum test
sellth Nov 7, 2023
779d402
move all basic irods logic into new class iRODSCommon
sellth Nov 8, 2023
1f58f7e
adapt tests to new class
sellth Nov 8, 2023
da40f44
iRODSTransfer is now a child of iRODSCommon
sellth Nov 8, 2023
d8dbbbf
flake8
sellth Nov 8, 2023
064a1e6
custom irods_env_path support; pass along kwargs
sellth Nov 8, 2023
7837b2a
flake8
sellth Nov 8, 2023
32aaa5f
make internal variables private
sellth Nov 8, 2023
f36efff
cleanup
sellth Nov 8, 2023
523bc28
improved docstrings
sellth Nov 9, 2023
c62faf1
TransferJob computes file size during init
sellth Nov 17, 2023
f66cc42
flake8
sellth Nov 17, 2023
e2bda03
using attrs import instead of attr shadow
sellth Nov 17, 2023
9d480c9
show file counts in progress bar
sellth Nov 17, 2023
179aee5
add recursive put
sellth Nov 17, 2023
4b0e1d2
use proper context manager
sellth Nov 17, 2023
4bde914
add sync mode; check for existing files on remote
sellth Nov 23, 2023
538e9a8
do not clean up session too soon
sellth Nov 27, 2023
41c8e12
prepare TransferJob to also handle gets
sellth Nov 27, 2023
5df1d7e
add get method
sellth Nov 27, 2023
205c6e4
restructure session creation; check valid token every time
sellth Nov 28, 2023
27e54da
create irods sessions dynamically
sellth Nov 28, 2023
f1e5105
also check for timed out session token
sellth Nov 28, 2023
78e01a1
make session available as property
sellth Nov 28, 2023
e1625a8
linting
sellth Nov 28, 2023
1a58949
remove irods session context manager and multiple sessions
sellth Nov 28, 2023
d6f6300
improve logging during chksum
sellth Dec 8, 2023
cb5d10a
increase session timeout
sellth Dec 8, 2023
bab45a0
better error handling
sellth Dec 8, 2023
9daa75e
feat: generic ingest function based on irods pythonclient
sellth May 3, 2023
49c2c54
add -y option alias --yes
sellth May 10, 2023
33006a0
make flake8 happy
sellth May 10, 2023
1a5e0b4
extract iRODS functionality into separate file
sellth May 10, 2023
a6b8d5b
compute md5 always and generate temp file on upload
sellth May 10, 2023
70cb0d7
adjusted log levels
sellth May 11, 2023
4903537
add test for init_irods
sellth May 12, 2023
1fb4d4c
add test for get_irods_error
sellth May 12, 2023
a0ccf97
add test for transfer job builder
sellth May 12, 2023
06d0471
add iinit-like behaviour when asking for password
sellth May 17, 2023
b145301
test sodar ingest build file list
sellth May 19, 2023
ab7736f
removed print statements in favour of logger
sellth May 19, 2023
29ab738
make more clear that this is not about checksum mismatch
sellth May 19, 2023
ab0933f
re-add test for transfer job builder
sellth May 19, 2023
3c4004d
add smoke check test for sodar ingest
sellth May 23, 2023
64bf19b
remove unused code
sellth May 23, 2023
79e9b3f
isort
sellth May 23, 2023
721d69c
use more pathlib
sellth May 24, 2023
cb2dffd
more no-coverage regions
sellth May 24, 2023
d1a599b
check for missing API token
sellth May 25, 2023
9a6cb90
adjusted docstring
sellth May 25, 2023
db872b4
add documentation for sodar ingest command
sellth May 30, 2023
4e0ffee
check if file already exists in iRODS and skip
sellth May 31, 2023
2b24a8c
moved remote file check to wrapper
sellth May 31, 2023
48d33a5
Clean exit code when nothing to do.
sellth Jun 1, 2023
5650dd8
upgrade python-irodsclient version
sellth Jun 2, 2023
d3d0312
add support for exclude patterns
sellth Jul 14, 2023
31b6416
fix excludes
sellth Aug 2, 2023
35eea47
sort file list for output
sellth Aug 2, 2023
6b96f59
reworked hashsum logic
sellth Aug 2, 2023
040bf38
increased iRODS session timeout
sellth Aug 7, 2023
026d93a
don't re-create already existing collections
sellth Sep 26, 2023
7ee5eb4
re-write of command to use irods_common.py
sellth Dec 7, 2023
f179b49
fix infinite loop when only 1 collection is present
sellth Dec 7, 2023
4490dde
increase test coverage
sellth Dec 7, 2023
70908b7
accomodate final subcollection as target
sellth Dec 7, 2023
2fa772c
transfer jobs as tuple, not set
sellth Dec 8, 2023
e6e4308
better help text for --yes
sellth Dec 8, 2023
8ade629
Merge branch 'main' into ingest
sellth Jan 9, 2024
0e41791
describe more clearly what's happening
sellth Jan 9, 2024
8dde5b5
update docs
sellth Jan 9, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
also check for timed out session token
  • Loading branch information
sellth committed Nov 28, 2023
commit f1e510506ace7d95a598d7ad6de4a9a4f0ac57a0
8 changes: 7 additions & 1 deletion cubi_tk/irods_common.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
CAT_INVALID_AUTHENTICATION,
PAM_AUTH_PASSWORD_FAILED,
CAT_PASSWORD_EXPIRED,
CAT_INVALID_USER,
)
from irods.password_obfuscation import encode
from irods.session import NonAnonymousLoginWithoutPassword, iRODSSession
Expand Down Expand Up @@ -78,11 +79,16 @@ def _init_irods(self) -> iRODSSession:
try:
session = iRODSSession(irods_env_file=self.irods_env_path)
session.connection_timeout = 300
session.server_version
return session
except NonAnonymousLoginWithoutPassword as e: # pragma: no cover
logger.info(self.get_irods_error(e))
self._irods_login()
except (CAT_INVALID_AUTHENTICATION, CAT_PASSWORD_EXPIRED): # pragma: no cover
except (
CAT_INVALID_AUTHENTICATION,
CAT_INVALID_USER,
CAT_PASSWORD_EXPIRED,
): # pragma: no cover
logger.warning("Problem with your session token.")
self.irods_env_path.parent.joinpath(".irodsA").unlink()
self._irods_login()
Expand Down