-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update download_utils.py #26
Conversation
Thanks @bsantan! kghub-downloader/kghub_downloader/download_utils.py Lines 112 to 113 in 16f85b0
local_name is mandatory. That's usually present, but what happens if there's just a url provided?
|
Note as per the KG construction call today: may be helpful to split the source-specific uniprot material into its own submodule to be called by download_utils.py |
@caufieldjh this is meant to check both if it exists and if it is specific to the uniprot case that I added. Note that after discussing with Harshad and Marcin, we think a better design may be to add this as a separate step in kg-microbe (perhaps as a GitHub workflow) that runs regularly and downloads all organism.json files to an s3 bucket, from which the kg download step would pull from. So this would remove some of these changes that also affect the download_from_yaml function here. |
Putting a pin on this PR from now. @caufieldjh , Marcin will reach out to you in the next few days to discuss a strategy involving S3 buckets. In short, we were thinking of running the Uniprot API to get JSON files (only the ones we need) and populating a S3 bucket. Add this S3 bucket url in the
|
Sure, that would be some great functionality to have, and possible with GH actions as long as it has access to the right credentials. |
Excellent! We'll need your guidance on this @caufieldjh ! Are there any examples already in the kg-hub universe? |
In terms of pushing artifacts to S3 during the Jenkins build, technically all/most of the builds already do that, they just do it through s3cmd. |
Closed in favor of #27 |
No description provided.