Packages

An input file is some archived file that a tool can act on.

A Tool is a set of scripts or executables and a dockerfile that can be run against input files.

A Package is a combination of Tool and Input File so that the tool will automatically be run against the specified files. Packages can be given a DOI and used in publications or for sharing.

To run a package:

User clicks run button; Job gets logged in DB and put on SQS queue.

Job listener catches job.

Create folder on EFS to hold files related to job. Should include username, job ID and (optionally) a user defined label.

Copy the Input data into the job folder.

Copy the required script files into the job folder.

Pull in the dockerfile for the given tool

Create the docker file/image/container.

Mount job folder into Docker container under /job.

Run scripts inside docker container:

Entrypoint script is defined by user and saved in dockerfile.

Input filenames are database and sent to entrypoint script as command line options.

Tool script can consume files inside the job folder and write new files to the job folder on efs.

When script terminates, terminate the docker container and free resources.

Potentially, remove all the original input files from job directory.

Update the Jobs database at various times to reflect the current status of the job (Finished, Running, Error, etc)