Fix docker build

[victorlin]

Description of proposed changes

Fix the Docker build by switching away from Pipenv and removing some unused dependencies.

Related issue(s)

Fixes #413

Testing

• Checks pass
• GenBank fetch and ingest using PR image is successful (aab1f5f, GitHub workflow, AWS Batch)
• Post-merge: add issue to use pip-compile: Lock dependencies using pip-compile #438

[tsibley]

+1 for ditching Pipenv!

[corneliusroemer]

Any progress here? Our CI is still broken 🙃

[victorlin]

@corneliusroemer not yet. I'll prioritize this for myself since I'm now realizing that ncov-ingest has been stuck using old versions of Nextclade and Augur. Since Nextclade 3 is now the default in the newer nextstrain/base images, we may have to bundle some Nextclade-related changes with this PR to at least use nextclade2 explicitly.

[joverlee521]

@corneliusroemer not yet. I'll prioritize this for myself since I'm now realizing that ncov-ingest has been stuck using old versions of Nextclade and Augur. Since Nextclade 3 is now the default in the newer nextstrain/base images, we may have to bundle some Nextclade-related changes with this PR to at least use nextclade2 explicitly.

ncov-ingest currently downloads Nextclade executable as part of the pipeline, so it's not using Nextclade version from the nextstrain/base image. @corneliusroemer is already migrating this to Nextclade V3 in #435.

[victorlin]

Oh thanks for clarifying! Is it common to have a Snakemake rule for env setup? I would expect it to be in the Dockerfile - that used to be the case but it was removed in 2ec2a6b. My guess is that Docker caching made the "latest" download outdated. Either an exact version needs to be specified or cache should be invalidated for the "latest" download (this is what we do for nextstrain/base).

[joverlee521]

Is it common to have a Snakemake rule for env setup?

I think this is left over from historical use of Nextclade before it was part of nextstrain/base. Previous conversation on this makes it seem like we can remove this from the Snakemake workflow itself and just use the Nextclade version available in the nextstrain/base image.

[corneliusroemer]

Yes, we can drop in the future if we have a new image, but right now it obviously doesn't work yet 😀

[victorlin]

I see it's on the task list for #375. Thanks for filling me in here! I'll still look into fixing the image build.

[victorlin]

I'm going to rebase this onto latest master, run GenBank fetch and ingest (on branch), then merge if everything succeeds.

EDIT: GenBank fetch and ingest running: GitHub workflow, AWS Batch

EDIT 2: Oops, I did the test wrong. Need to run the workflow on the PR image using aab1f5f, not the PR source branch. Running: GitHub workflow, AWS Batch

updated to use requirements.txt

[joverlee521]

Looks like the GitHub workflow was still using the nextstrain/ncov-ingest image rather than the branch image added in aab1f5f.

I just ran the GH Action for #437 and ran into an error on AWS Batch:

[batch] [2024-02-21T13:26:07-08:00] Traceback (most recent call last):
[batch] [2024-02-21T13:26:07-08:00]   File "/usr/local/lib/python3.10/site-packages/snakemake/__init__.py", line 610, in snakemake
[batch] [2024-02-21T13:26:07-08:00]     workflow = Workflow(
[batch] [2024-02-21T13:26:07-08:00]   File "/usr/local/lib/python3.10/site-packages/snakemake/workflow.py", line 235, in __init__
[batch] [2024-02-21T13:26:07-08:00]     self._sourcecache = SourceCache()
[batch] [2024-02-21T13:26:07-08:00]   File "/usr/local/lib/python3.10/site-packages/snakemake/sourcecache.py", line 353, in __init__
[batch] [2024-02-21T13:26:07-08:00]     os.makedirs(self.cache, exist_ok=True)
[batch] [2024-02-21T13:26:07-08:00]   File "/usr/local/lib/python3.10/os.py", line 215, in makedirs
[batch] [2024-02-21T13:26:07-08:00]     makedirs(head, exist_ok=exist_ok)
[batch] [2024-02-21T13:26:07-08:00]   File "/usr/local/lib/python3.10/os.py", line 215, in makedirs
[batch] [2024-02-21T13:26:07-08:00]     makedirs(head, exist_ok=exist_ok)
[batch] [2024-02-21T13:26:07-08:00]   File "/usr/local/lib/python3.10/os.py", line 225, in makedirs
[batch] [2024-02-21T13:26:07-08:00]     mkdir(name, mode)
[batch] [2024-02-21T13:26:07-08:00] PermissionError: [Errno 13] Permission denied: '/nextstrain/.cache/snakemake'

[joverlee521]

Ah, error comes from the removal of rm -rf ~/.cache, see details in fd2b922.

Will make a follow-up PR shortly.

[victorlin]

Oh sorry! In aab1f5f I changed fetch-and-ingest-genbank-master.yml instead of fetch-and-ingest-genbank-branch.yml 🤦

[joverlee521]

No worries @victorlin, the set up of GH Action workflows in this repo is a bit outdated.
I'm planning on doing updates to reduce potential confusion like this.