Create a Docker image for topic generator #505

coverbeck · 2024-10-04T21:05:54Z

Description
Creates a Docker image for the topic generator. We'll use that image for the scheduled task in Fargate.

Depends on companion workflow-actions PR

Review Instructions
See review steps in dockstore/workflow-actions#2

Issue
SEAB-6653

Security
If there are any concerns that require extra attention from the security team, highlight them here.

Please make sure that you've checked the following before submitting your pull request. Thanks!

Check that you pass the basic style checks and unit tests by running mvn clean install in the project that you have modified (until https://ucsc-cgl.atlassian.net/browse/SEAB-5300 adds multi-module support properly)
Ensure that the PR targets the correct branch. Check the milestone or fix version of the ticket.
If you are changing dependencies, check with dependabot to ensure you are not introducing new high/critical vulnerabilities
If this PR is for a user-facing feature, create and link a documentation ticket for this feature (usually in the same milestone as the linked issue). Style points if you create a documentation PR directly and link that instead.

coverbeck · 2024-10-04T21:07:53Z

Dockerfile

+RUN apt-get update \
+    && apt-get upgrade -y \
+    && apt-get install -y --no-install-recommends
+COPY topicgenerator/target/topicgenerator*[^s].jar /home


I suppose we could add other JARs from this repo in there, but the metrics aggregator and github delivery both require s3 permissions and documented to be run on the deployer, where, AFAIK, we don't have Docker installed. So for now, I'm only doing this one, but we could more in the future.

codecov · 2024-10-04T21:09:45Z

Codecov Report

Attention: Patch coverage is 28.57143% with 5 lines in your changes missing coverage. Please review.

Project coverage is 19.47%. Comparing base (b964bfb) to head (048dc9e).
Report is 1 commits behind head on develop.

Files with missing lines	Patch %	Lines
...opicgenerator/client/cli/TopicGeneratorClient.java	0.00%	5 Missing ⚠️

Additional details and impacted files

@@              Coverage Diff              @@
##             develop     #505      +/-   ##
=============================================
+ Coverage      15.91%   19.47%   +3.56%     
- Complexity       154      178      +24     
=============================================
  Files             57       57              
  Lines           2897     2901       +4     
  Branches         228      228              
=============================================
+ Hits             461      565     +104     
+ Misses          2400     2297     -103     
- Partials          36       39       +3

Flag	Coverage Δ
tooltester	`15.89% <0.00%> (-0.03%)`	⬇️
topicgenerator	`19.47% <28.57%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

coverbeck · 2024-10-11T17:25:41Z

topicgenerator/scripts/updateDockstoreTopicsFromDocker.sh

+cd /home
+
+echo "Creating a CSV file of AI topic candidates from Dockstore"
+java -jar $APP_JAR -c $CONFIG get-topic-candidates


This command doesn't exist yet, it's in PR #504.

coverbeck · 2024-10-14T23:39:43Z

topicgenerator/src/main/java/io/dockstore/topicgenerator/helper/AIModelType.java

-    CLAUDE_3_5_SONNET("anthropic.claude-3-5-sonnet-20240620-v1:0", 0.003, 0.015, 200000),
-    CLAUDE_3_HAIKU("anthropic.claude-3-haiku-20240307-v1:0", 0.00025, 0.00125, 200000),
+    CLAUDE_3_5_SONNET("us.anthropic.claude-3-5-sonnet-20240620-v1:0", 0.003, 0.015, 200000),
+    CLAUDE_3_HAIKU("us.anthropic.claude-3-haiku-20240307-v1:0", 0.00025, 0.00125, 200000),


I needed to change these to be cross-region inferences, so QA would work.

coverbeck · 2024-10-15T16:55:06Z

topicgenerator/src/main/java/io/dockstore/topicgenerator/client/cli/TopicGeneratorClient.java

-        final String mainScriptPath = NextflowUtilities.grabConfig(nextflowConfigFileContent).getString("manifest.mainScript", "main.nf");
+        final String mainScriptPath;
+        try {
+            mainScriptPath = NextflowUtilities.grabConfig(nextflowConfigFileContent).getString("manifest.mainScript", "main.nf");


First time I ran this, I believe I got a NextflowParsingException, which caused the program to exit early. Put this in a try catch.

Kathy thinks that it may have just been logging noise that I saw, so I could be wrong. But the method can throw a NextFlowParsingException, so we should catch it anyway.

kathy-t · 2024-10-15T20:33:33Z

topicgenerator/scripts/updateDockstoreTopicsFromDocker.sh

+
+cd /home
+
+echo "Generating topics for all AI topic candidates from Dockstore"


"for all AI topic candidates" -> for 500

topicgenerator/scripts/updateDockstoreTopicsFromDocker.sh

svonworl · 2024-10-15T22:10:55Z

Dockerfile

+RUN apt-get update \
+    && apt-get upgrade -y \
+    && apt-get install -y --no-install-recommends
+COPY topicgenerator/target/topicgenerator*[^s].jar /home/topic-generator.jar


What's the [^s] do?

After a build, there are 2 JARs in the target directory:

topicgenerator-1.16.0-SNAPSHOT-sources.jar
topicgenerator-1.16.0-SNAPSHOT.jar

The [^s] is what I came up with to skip the sources JAR.

kathy-t · 2024-10-18T21:01:15Z

Added two additional things to this PR:

In the maven.yml GitHub Action, added steps to build and push an image to quay.io/dockstore/dockstore-support if the branch being pushed to is develop. This is to support the auto generation of the SupportVersion property in the properties file and so that we don't need to do a tagged release every time we need a new image in QA.
Installed additional dependencies in the Docker image to support uploading files to S3

Re-requesting reviews

sonarcloud · 2024-10-18T21:07:43Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

coverbeck · 2024-10-21T05:15:16Z

.github/workflows/maven.yml

+          echo "S3_FOLDER=${S3_FOLDER}-$(echo $GITHUB_SHA | cut -c -7)" >> $GITHUB_ENV
+
+      # neat, quay itself uses manual github actions https://github.com/quay/quay/blob/master/.github/workflows/build-and-publish.yaml
+      - if: github.ref == 'refs/heads/develop'


At some point in the future, especially if we end up creating additional Docker images, we might want to consider moving this code to dockstore/workflows. But looks good now.

Agreed, I noticed we create images and upload checksums for images in dockstore/dockstore and dockstore-deploy as well, so it'll be good to create a re-usable workflow. I'll create a ticket for this or maybe this can be tackled when we implement uploading the image digest for this image

coverbeck · 2024-10-21T05:15:56Z

I can't approve this because I'm still the assignee -- not sure if that's intentional or not, so signaling my approval here.

coverbeck self-assigned this Oct 4, 2024

coverbeck commented Oct 4, 2024

View reviewed changes

denis-yuen approved these changes Oct 5, 2024

View reviewed changes

coverbeck force-pushed the feature/seab-6653/topicUpdater branch from 5ad260d to 7a4d732 Compare October 10, 2024 22:14

coverbeck commented Oct 11, 2024

View reviewed changes

coverbeck requested review from david4096, kathy-t and svonworl October 11, 2024 17:25

svonworl approved these changes Oct 12, 2024

View reviewed changes

coverbeck force-pushed the feature/seab-6653/topicUpdater branch from a686116 to ee11fd3 Compare October 14, 2024 17:33

coverbeck commented Oct 14, 2024

View reviewed changes

kathy-t and others added 18 commits October 15, 2024 09:17

Create csv file

1dc6db5

update artifacts

51d70a6

Update artifacts

92502dc

Clean up

ca5ed9b

Use endpoint with pagination

a7ab9ce

Update README

9719459

Remove test code

722519d

Create csv file

04d8560

Clean up

4d75566

Save work

fd79855

Rebase off develop

4455089

Clean up script

5663ce8

Undo bringing in code from draft PR

8c8d6a0

Undo bringing in code from draft PR

9f281ad

Why do I need this import; go figure

542902d

Use cross-region inference profile, I think.

6ea4210

Handle Nextflow JAR not present

bbf0da3

Clean up message

80ffaa0

Post-rebase fix.

d3c477a

coverbeck force-pushed the feature/seab-6653/topicUpdater branch from 52fb5fa to d3c477a Compare October 15, 2024 16:25

coverbeck commented Oct 15, 2024

View reviewed changes

coverbeck requested review from svonworl and denis-yuen October 15, 2024 16:56

kathy-t approved these changes Oct 15, 2024

View reviewed changes

david4096 approved these changes Oct 15, 2024

View reviewed changes

PR Feedback

8d7f369

svonworl approved these changes Oct 15, 2024

View reviewed changes

coverbeck and others added 10 commits October 15, 2024 16:34

Clarifying comment

81b5e7a

Create image on push

e422ab2

Remove redundant pull_request event

da4268c

Add OIDC authentication

7d09b13

No checksum

7eb18c9

Install aws cli

f782db3

Test branch matching

ce4ad53

Install jq

0ff65ac

Clean up

223cafc

Add newline

048dc9e

kathy-t requested review from david4096 and svonworl October 18, 2024 21:01

svonworl approved these changes Oct 19, 2024

View reviewed changes

coverbeck commented Oct 21, 2024

View reviewed changes

david4096 approved these changes Oct 21, 2024

View reviewed changes

kathy-t merged commit 0db8b4b into develop Oct 21, 2024
10 checks passed

kathy-t deleted the feature/seab-6653/topicUpdater branch October 21, 2024 19:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create a Docker image for topic generator #505

Create a Docker image for topic generator #505

coverbeck commented Oct 4, 2024 •

edited

Loading

coverbeck Oct 4, 2024

codecov bot commented Oct 4, 2024 •

edited

Loading

coverbeck Oct 11, 2024

coverbeck Oct 14, 2024

coverbeck Oct 15, 2024

kathy-t Oct 15, 2024

svonworl Oct 15, 2024

coverbeck Oct 15, 2024

kathy-t commented Oct 18, 2024

sonarcloud bot commented Oct 18, 2024

coverbeck Oct 21, 2024

kathy-t Oct 21, 2024

coverbeck commented Oct 21, 2024


		cd /home

		echo "Generating topics for all AI topic candidates from Dockstore"

Create a Docker image for topic generator #505

Create a Docker image for topic generator #505

Conversation

coverbeck commented Oct 4, 2024 • edited Loading

Choose a reason for hiding this comment

codecov bot commented Oct 4, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kathy-t commented Oct 18, 2024

sonarcloud bot commented Oct 18, 2024

Quality Gate passed

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coverbeck commented Oct 21, 2024

coverbeck commented Oct 4, 2024 •

edited

Loading

codecov bot commented Oct 4, 2024 •

edited

Loading