sda-admin cli

[nanjiangshu]

Related issue(s) and PR(s)
This PR closes #1007

Description
The command line tool that can talk to SDA API endpoints. It provides functionalities to list users and files, ingest and set accession IDs for files, and create or release datasets.

How to test

1. At the root of the repo, start the stack by running

make build-all
PR_NUMBER=$(/usr/bin/date '+%F')  docker compose -f .github/integration/sda-s3-integration.yml run integration_test

2. Set the ENVs by

export ACCESS_TOKEN=$(curl -s -k http://localhost:8080/tokens | jq -r '.[0]')
export API_HOST=http://localhost:8090

3. Build the executable in the folder sda-admin by

go build 

3. Test various functionalities

# list all users
./sda-admin list users  

# list all files for the user [email protected]
./sda-admin list files  

# list all files for the user [email protected]
./sda-admin list files -user [email protected] 

# trigger the ingestion of a given file 
./sda-admin file ingest --filepath requester_demo.org/data/file1.c4gh --user [email protected]  

# assign an accession ID to a file
./sda-admin file accession --filepath requester_demo.org/data/file1.c4gh  --user [email protected]   --accession-id my-id1 

# create a dataset from a list of accession IDs and the dataset ID
./sda-admin dataset create --dataset-id dataset001 my-id1 my-id2 

# # release a dataset for downloading
./sda-admin dataset release --dataset-id dataset001 

[jbygdell]

Didn't go through all but added a few pointers that needs to go in overall.
Secondly use the linting function often so you don't end up with simple mistakes.

[MalinAhlberg]

Really well done, @nanjiangshu!
I added some minor comments. Also, shouldn't .github/workflows/test.yml be updated so that the tests are run?

[viklund]

I think commands should either be <verb> <noun> or <noun> <verb>. Currently there's a mix of both.

I suggest not having list as a toplevel command. So refactor list files and list users into files list and users list.

[nanjiangshu]

Really well done, @nanjiangshu! I added some minor comments. Also, shouldn't .github/workflows/test.yml be updated so that the tests are run?

Good catch @MalinAhlberg , I added in the commit ab22a90

[nanjiangshu]

I think commands should either be <verb> <noun> or <noun> <verb>. Currently there's a mix of both.

I suggest not having list as a toplevel command. So refactor list files and list users into files list and users list.

Good suggestion @viklund . The syntax of the commands is now consistent with the format . I use file instead of files and set-accession instead of session.

sda-admin user list
sda-admin file list
sda-admin file ingest
sda-admin file set-accession
sda-admin dataset create
sda-admin dataset release  

[jbygdell]

• Add this folder to the code-linter action
• Add an option in the Makefile to build the app, both separate and as part of the build-all step
• Add an option in the Makefile to run the linter for this folder

[MalinAhlberg]

I tested this a bit, and I think it's a really nice way to work with the pipeline! Also, it seems to be a really good way to run into bugs, by being a maybe-not-so-organized user. I quickly ran into bug #1027, and also managed to run into something similar for verify/postgres. They keep logging even though I did not get any error from sda-admin:

postgres  | 2024-09-12 13:36:22.248 UTC [80] ERROR:  duplicate key value violates unique constraint "unique_checksum"
...
verify    | {"level":"error","msg":"SetVerified failed, reason: (pq: duplicate key value violates unique constraint \"unique_checksum\")","time":"2024-09-12T13:36:22Z"}

I'm not sure what I did wrong.
When trying to create a dataset with incorrect file IDs (which could happen if you for example mistype), no error is reported but I was not able to continue after that, all commands returned errors such as

Error: failed to [...], reason: failed to send request, reason: Post "http://localhost:8090/file/accession": EOF

[jbygdell]

I tested this a bit, and I think it's a really nice way to work with the pipeline! Also, it seems to be a really good way to run into bugs, by being a maybe-not-so-organized user. I quickly ran into bug #1027, and also managed to run into something similar for verify/postgres. They keep logging even though I did not get any error from sda-admin:

postgres  | 2024-09-12 13:36:22.248 UTC [80] ERROR:  duplicate key value violates unique constraint "unique_checksum"
...
verify    | {"level":"error","msg":"SetVerified failed, reason: (pq: duplicate key value violates unique constraint \"unique_checksum\")","time":"2024-09-12T13:36:22Z"}

All of that needs to be solved in the API. There are a few steps we need to double check.

[MalinAhlberg]

All of that needs to be solved in the API. There are a few steps we need to double check.

I created #1037 and #1038.

[jbygdell]

In order to test that the messages arrive in the correct stream (stderr or stdout) you can use this: /sda-admin -uri example.com -token fooBar help dataset 2>stderr.log 1>stdout.log

[viklund]

⭐