Skip to content

Commands

Jump to bottom
Alexis Lucattini edited this page Mar 26, 2021 · 12 revisions

Commands

References to project.yaml, tool.yaml, workflow.yaml, expression.yaml and schema.yaml in this document reference to files under ${CWL_ICA_REPO_PATH/config}

configure

Configure the path to the repository by adding the CWL_ICA_REPO_PATH env var
to your conda env activate.d directory. This variable is unset upon deactivate.
You MUST reactivate the environment after setting this value.
This command also creates configuration.yaml in the repo path if not already present.

Options:

  • repo-path
    • Path to the local clone of the cwl-ica git repository.
  • username
    • The username used when generating an author tag for a CWL tool, expression or workflow
    • This updates config/user.yaml
  • useremail
    • The email address used when generating an author tag for a CWL tool, expression or workflow
    • This updates config/user.yaml Usage: 
cwl-ica configure \
  --repo-path /abs/path/to/repository \
  --username "Alexis Lucattini" \ 
  --email "[email protected]"

project init

Initialise a project in project.yaml

This project must be registered in ICA, use ica projects list to view available projects.

Options:

  • project-id
    • Use ica projects list to view all available project names and thier ids.
  • project-name
    • Use ica projects list to view all available project names and their ids
    • Also used by other commands as the name of a project.
  • access-token
    • Sets this env var as ICA_ACCESS_TOKEN_<PROJECT_NAME> in the conda env activate.d directory. Can be updated with cwl-ica project-update --access-token=<new-access-token>
  • is-production
    • Setting this parameter will register the project as a production project. Production projects do not cannot be synced with cwl-ica tool-sync or cwl-ica workflow-sync and new ica workflow-versions are instead automatically created on pushes to the main branch.

Usage:

# A non production project
cwl-ica project-init \
  --project-id xxxx-yyyy... \
  --project-name my-registered-project \
  --access-token <project-access-token>

# A production project
cwl-ica project-init \
  --project-id xxxx-yyyy... \
  --project-name my-registered-project \
  --access-token <project-access-token> \
  --is-production

project update

Update a project

This project must be registered in ICA and in project.yaml.

This command is most-commonly used to update a project's expired access token, to rename a project in project.yaml or change a project's id in project.yaml OR to migrate a project from in development to production.

Options:

  • project-id
  • new-project-id (optional)
  • project-name
  • new-project-name (optional)
  • access-token (optional)
  • production boolean (optional)

Usage:

# Update an access token in activate.d directory.  
cwl-ica project-update \
  --project-id xxxx-yyyy... \
  --project-name my-registered-project \
  --access-token <access-token>

category-init

Create a category in category.yaml.
When registering tools and workflows, you may use the --categories command. This will be reflected in the catalogue AND on ICA. A tool or workflow may belong to multiple categories.

Options

  • category-name
  • category-description (optional)

Usage:

# Create a category
cwl-ica category-init \
  --category-name "name of category" \
  --category-description "whats the theme"

create-tool-from-template

Initialise a tool from the cwl templates.

Options

  • tool-name
  • tool-version
  • cwl-version (optional - default is v1.1)

Usage

cwl-ica create-tool-from-template \
  --tool-name name-of-tool \
  --tool-version version-of-tool

create-workflow-from-template

Initialise a workflow from the cwl templates

Options

  • workflow-name
  • workflow-version
  • cwl-version (optional - default is v1.1)

Usage

cwl-ica create-workflow-from-template \
  --workflow-name name-of-workflow \
  --workflow-version version-of-workflow

create-expressions-from-template

Initialise an expression from the cwl templates

Options

  • expression-name
  • expression-version
  • cwl-version (optional - default is v1.1)

Usage

cwl-ica create-expression-from-template \
  --expression-name name-of-expression \
  --expression-version version-of-expression

create-schema-from-template

Initialise a schema from the cwl templates

Options

  • schema-name
  • schema-version

Usage

cwl-ica create-schema-from-template \
  --schema-name name-of-schema \
  --schema-version version-of-schema

tool validate

Validates a CWL File of CommandLineTool prior to upload to ICA.

See [contributions][contributions] for more information / example of a valid cwltool.

The basics are:

  • tool must pass cwltool --strict --validate.
  • inputs, outputs, requirements and hints section should all be in [key-format].
  • id, label and doc must be present for tool doc.
  • label and doc must be present for each input and output of the tool.
  • A list of successCodes must be present.
  • An author description must be present.
  • Resource requirements must be specified in 'hints' section.

Options:

  • tool-path
    • Path to cwl tool. Must be inside ${CWL_ICA_REPO_PATH}.

Usage

cwl-ica tool-validate \
  --tool-path /path/to/tool.cwl

workflow validate

Validates a CWL Workflow file of Workflow prior to upload to ICA.

[contributions][contributions] for more information / example of a valid cwl workflow.

The basics are:

  • workflow must pass cwltool --strict --validate.
  • workflow in packed format must also pass cwltool --strict --validate.
  • inputs, outputs, steps, requirements section should all be in [key-format].
  • id, label and doc must be present for workflow doc.
  • label and doc must be present for each input and output and steps.
  • An author description must be present.
  • Each step must link to a separate file. No step is to be written 'in-place'.

Options

  • workflow-path:
    • Path to cwl workflow. Must be inside ${CWL_ICA_REPO_PATH}

Usage:

cwl-ica workflow-validate \
  --workflow-path /path/to/workflow.cwl

expression validate

Validates a CWL expression file of expression prior to upload to ICA.

[contributions][contributions] for more information / example of a valid cwl expression.

The basics are:

  • expression must pass cwltool --strict --validate.
  • expression in packed format must also pass cwltool --strict --validate.
  • expression must contain 'id', 'label' and 'doc attributes'
  • An author description must be present.
  • Resource requirements must be specified in 'hints' section.
  • label and doc must be present for each input and output of the expression.

Options

  • expression-path:
    • Path to cwl expression. Must be inside ${CWL_ICA_REPO_PATH}

Usage:

cwl-ica expression-validate \
  --expression-path /path/to/expression.cwl

schema validate

Validates a CWL schema file of Workflow prior to upload to ICA.

[contributions][contributions] for more information / example of a valid cwl schema. The basics are:

  • schema must pass cwltool --strict --validate.
  • schema in packed format must also pass cwltool --strict --validate.
  • inputs, outputs, steps, requirements section should all be in [key-format].
  • id, label and doc must be present for schema doc.
  • label and doc must be present for each input and output and steps.
  • An author description must be present.
  • Each step must link to a separate file. No step is to be written 'in-place'.

Options

  • schema-path:
    • Path to cwl schema. Must be inside ${CWL_ICA_REPO_PATH}

Usage:

cwl-ica schema-validate \
  --schema-path /path/to/workflow.cwl

tool init

Initialise / register a tool at tool.yaml and at projects.yaml (if --projects used).

Determine which projects the tool will be present in. Options

  • tool-path

    • Path to tool

  • projects (optional)

    • Comma separated list of ICA projects to add tool to.
    • Use 'all' to select all projects in projects.yaml.
    • For projects registered as 'production-projects' in projects.yaml. The tool will be registered but not synced. This is done only on a push to the main branch, the first seven characters of the commit id will be used as a suffix for the workflow version.

Usage

cwl-ica tool-init \
  --projects project[,additional-projects] \
  --tool-path /path/to/tool.cwl \
  --categories comma,separated,list

workflow init

Initialise / register a workflow at workflow.yaml and at projects.yaml (if --projects used).

Determine which projects the workflow will be present in. Options

  • workflow-path

    • Path to workflow

  • projects (optional)

    • Comma separated list of ICA projects to add workflow to.
    • Use 'all' to select all projects in projects.yaml.
    • For projects registered as 'production-projects' in projects.yaml. The tool will be registered but not synced. This is done only on a push to the main branch, the first seven characters of the commit id will be used as a suffix for the workflow version.

Usage

cwl-ica workflow-init \
  --projects project[,additional-projects] \
  --workflow-path /path/to/workflow.cwl \
  --categories comma,separated,list

expression init

Initialise / register an expression at expression.yaml

Options

  • expression-path
    • Path to expression

Usage

cwl-ica expression-init \
  --expression-path /path/to/expression.cwl

schema init

Initialise / register a schema at schema.yaml

Options

  • schema-path
    • Path to expression

Usage

cwl-ica schema-init \
  --schema-path /path/to/schema.cwl

tool add

Map an existing workflow and workflow version for a project to a tool path.
This tool may be already registered in tool.yaml

Options

  • tool-path
    • Path to tool
  • tool-name
    • Used as identifier in tool.yaml and projects.yaml.
  • tool-version
    • Used as an identifier in tool.yaml and projects.yaml
  • project
    • Project this workflow is bound to
  • workflow-id
    • The workflow id registered by ICA (must start with wfl.)
  • workflow-version
    • The workflow version registered by ICA

workflow add

Map an existing workflow and workflow version for a project to a workflow path This tool may be already registered in workflow.yaml

Options

  • workflow-path
    • Path to workflow
  • workflow-name
    • Used as identifier in workflow.yaml and projects.yaml.
  • workflow-version
    • Used as an identifier in workflow.yaml and projects.yaml
  • project
    • Project this workflow is bound to
  • workflow-id
    • The workflow id registered by ICA (must start with wfl.)
  • workflow-version
    • The workflow version registered by ICA

tool sync

Update the md5sum attribute in tool.yaml.
Sync a tool with ICA endpoint for a given project or for all projects.
This will first validate the tool using cwl-ica tool-validate before trying to sync.
This requires a confirmation for each project via user-input which can be overridden by using the --yes flag.

Options:

  • tool-path
    • The path to the CWL tool to sync to projects
  • projects (optional)
    • Comma separated list of ICA projects to add workflow to.
    • Use 'all' to select all projects in projects.yaml.
    • Production projects will NOT be synced

Usage:

cwl-ica tool-sync \
  --tool-path /path/to/tool.cwl \
  --projects project[,project2] \

workflow sync

Update the md5sum attribute in workflow.yaml.
Sync a workflow with ICA endpoint for a given project or for all projects.
This will first validate the workflow using cwl-ica workflow-validate before trying to sync.
This requires a confirmation for each project via user-input which can be overridden by using the --yes flag.

Options:

  • workflow-path
    • The path to the CWL workflow to sync to projects
  • projects (optional)
    • Comma separated list of ICA projects to add workflow to.
    • Use 'all' to select all projects in projects.yaml.
    • Production projects will NOT be synced

Usage:

cwl-ica workflow-sync \
  --workflow-path /path/to/workflow \
  --projects project[,project2]

add tool to project

Adds an existing tool to a project. One must then run cwl-ica tool-sync project in order to sync / register the project on ICA. The tool must be registered in tool.yaml.

Options:

  • tool-path
    • The path to the cwl tool
  • project
    • The name of the project to register the tool with

Usage:

cwl-ica add-tool-to-project \
  --tool-path /path/to/tool.cwl \
  --project project

add workflow to project

Adds an existing workflow to a project. One must then run cwl-ica workflow-sync project in order to sync / register the project on ICA. The workflow must be registered in workflow.yaml.

Options:

  • workflow-path
    • The path to the cwl workflow
  • project
    • The name of the project to register the workflow with

Usage:

cwl-ica add-workflow-to-project \
  --workflow-path /path/to/workflow.cwl \
  --project project

register tool run instance id

Register a run instance of a tool of the tool cpu / mem usage over time. This writes out a json file under runs/tools/tool-path.<wfr.id>.json This can provide a user with estimated run time metrics and suggest overrides as required.

Options:

  • tool-run-instance-id
    • The run instance id (must start with 'wfr.')
  • project
    • The project the run was launched on.

Usage:

cwl-ica register-tool-run-instance-id \
  --tool-run-instance-id wfr... \
  --project project

register workflow run instance id

Register a run instance of a workflow, collates the cpu / mem usage over time of each task run as well as a stacked area chart of the total resources used. As such can provide user with estimated run time metrics.
This writes out a json file under runs/workflows/workflow-path.<wfr-id>.json.

get workflow step ids

Get the ids of all steps in a workflow.
Prints two - tab delimited column of step id, step label.

The output ids can be used to set overrides for a cwl workflow.

Options:

  • workflow-path
    • Path to workflow.cwl

Usage:

cwl-ica get-workflow-step-ids \
  --workflow-path /path/to/workflow.cwl

list-tool-runs

Get a list of runs for each tool

Prints two - tab delimited column of workflow run id, project

The run id can be used in cwl-ica copy-tool-submission-template to replicate a run.

Options:

  • tool-path
    • Path to tool.cwl
  • projects
    • Restricts to projects run

Usage:

cwl-ica list-tool-runs \
  --tool-path /path/to/tool.cwl \
  --project (optional)

list-workflow-runs

Get a list of runs for each workflow

Prints two - tab delimited column of workflow run id, project

The run id can be used in cwl-ica copy-workflow-submission-template

Options:

  • workflow-path
    • Path to workflow.cwl
  • projects
    • Restricts to projects run

Usage:

cwl-ica list-workflow-runs \
  --workflow-path /path/to/workflow.cwl \
  --project (optional)

copy-tool-submission-template

Copy the tool submission template for a tool run

Options:

  • tool-path
    • Path to tool.cwl
  • project
    • project name
  • tool-run-instance-id
    • The run instance id to copy from
  • output-file (optional)
    • json stdout if not specified

Usage:

cwl-ica copy-tool-submission-template \
  --tool-path /path/to/tool.cwl \
  --project project \
  --tool-run-instance-id wfr... \
  --output-file input.json

copy-workflow-submission-template

Copy the workflow submission template for a workflow run

Options:

  • workflow-path
    • Path to a workflow.cwl
  • project
    • project name
  • workflow run instance id
    • The run instance id to copy from
  • output-file (optional)
    • json stdout if not specified

Usage:

cwl-ica copy-workflow-submission-template \
  --workflow-path /path/to/workflow.cwl \
  --project project \
  --workflow-run-instance-id wfr...
  --output-file input.json

GitHub Actions

There are five github actions subcommands that do not have to be remembered by the user. These run only on pull requests and pushes to the main branch.

  • github-actions tools-update
    • Updates all tools for all projects (including production projects)
    • Run on push to main branch
  • github-actions workflows-update
    • Updates all workflows for all projects (including production projects)
    • Run on push to main branch
  • build-catalogue
    • Updates the markdown workflows catalogue.
    • Run on push to main branch
  • check-pull-request
    • Validate each workflow in workflow.yaml
    • Validate each tool in tool.yaml
    • List / comment workflows that have already been updated on ICA
    • List / comment workflows that are going to be updated for each project via md5sum against md5sum on main branch.
    • Run on initialisation of pull request / commits to pull request to main branch
  • postman-generattor
    • Generate a postman json from a template that takes in a projects' workflows, workflow versions, and run ids and generates an importable postman json file.
    • Run on push to main branch
Clone this wiki locally