Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Add generic SODAR ingest command #199

Merged
merged 72 commits into from
Jan 9, 2024
Merged

feat: Add generic SODAR ingest command #199

merged 72 commits into from
Jan 9, 2024

Conversation

sellth
Copy link
Contributor

@sellth sellth commented Nov 7, 2023 •
edited
Loading

re do of #168

This adds a generic sodar ingest command which tries to mimic the base functionality of irsync, boosted by the power of using SODAR's API. Features include:

  • no dependency on icommands; uses iinit session file if present, asks for password otherwise

  • multiple source locations, single target collection

  • recursive file matching

  • creation of matched sub-folders in iRODS

  • destination either as iRODS path or SODAR LZ UUID

  • uses SODAR API for UUID destination & for querying LZ collections

  • computes local checksums if missing

  • triggers remote iRODS checksum computation

  • scriptable; all user input can be skipped

  • has a cool progress bar

  • rebase after PR feat: Common functions for interfacing with python-irodsclient #202 is merged.

sellth added 28 commits November 7, 2023 13:40
@sellth
feat: Common functions for interfacing with python-irodsclient
95c76ab
@sellth
fix tests
6b72d9d
@sellth
fix chksum test
2636c8c
@sellth
move all basic irods logic into new class iRODSCommon
779d402
@sellth
adapt tests to new class
1f58f7e
@sellth
iRODSTransfer is now a child of iRODSCommon
da40f44
@sellth
flake8
d8dbbbf
@sellth
custom irods_env_path support; pass along kwargs
064a1e6
@sellth
flake8
7837b2a
@sellth
make internal variables private
32aaa5f
@sellth
cleanup
f36efff
@sellth
improved docstrings
523bc28
@sellth
TransferJob computes file size during init
c62faf1
@sellth
flake8
f66cc42
@sellth
using attrs import instead of attr shadow
e2bda03
@sellth
show file counts in progress bar
9d480c9
@sellth
add recursive put
179aee5
@sellth
use proper context manager
4b0e1d2
@sellth
add sync mode; check for existing files on remote
4bde914
@sellth
do not clean up session too soon
538e9a8
@sellth
prepare TransferJob to also handle gets
41c8e12
@sellth
add get method
5df1d7e
@sellth
restructure session creation; check valid token every time
205c6e4
@sellth
create irods sessions dynamically
27e54da
@sellth
also check for timed out session token
f1e5105
@sellth
make session available as property
78e01a1
@sellth
linting
e1625a8
@sellth
remove irods session context manager and multiple sessions
1a58949
sellth and others added 18 commits December 8, 2023 16:42
@sellth
check for missing API token
d1a599b
@sellth
adjusted docstring
9a6cb90
@sellth
add documentation for sodar ingest command
db872b4
@sellth
check if file already exists in iRODS and skip
4e0ffee
@sellth
moved remote file check to wrapper
2b24a8c
@sellth
Clean exit code when nothing to do.
48d33a5
@sellth
upgrade python-irodsclient version
5650dd8
@sellth
add support for exclude patterns
d3d0312
@sellth
fix excludes
31b6416
@sellth
sort file list for output
35eea47
@sellth
reworked hashsum logic
6b96f59 
- now checks for .md5 file on disk
- loads hash if found
- computes hash if not found and stores on disk
- checks iRODS for existing file using hash
@sellth
increased iRODS session timeout
040bf38
@sellth
don't re-create already existing collections
026d93a
@sellth
re-write of command to use irods_common.py
7ee5eb4
@sellth
fix infinite loop when only 1 collection is present
f179b49
@sellth
increase test coverage
4490dde
@sellth
accomodate final subcollection as target
70908b7
@sellth
transfer jobs as tuple, not set
2fa772c
@sellth sellth changed the title feat: SODAR ingest command based on irods-pythonclient feat: Add generic SODAR ingest command Dec 8, 2023
Nicolai-vKuegelgen
Nicolai-vKuegelgen approved these changes Jan 9, 2024
View reviewed changes
Copy link
Contributor

@Nicolai-vKuegelgen Nicolai-vKuegelgen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@sellth sellth enabled auto-merge (squash) January 9, 2024 14:33
@sellth sellth disabled auto-merge January 9, 2024 14:34
@sellth sellth enabled auto-merge (squash) January 9, 2024 14:35
@sellth sellth merged commit 8548c97 into main Jan 9, 2024
6 checks passed
@sellth sellth deleted the ingest branch January 9, 2024 14:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Nicolai-vKuegelgen Nicolai-vKuegelgen Nicolai-vKuegelgen approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@sellth @Nicolai-vKuegelgen