Skip to content

Commands

Jump to bottom
Sehrish edited this page Jul 27, 2021 · 12 revisions

References to tenant.yaml, project.yaml, tool.yaml, workflow.yaml, expression.yaml and schema.yaml in this document reference to files under ${CWL_ICA_REPO_PATH/config}

Commands

Configuration commands

configure repo

Configure the path to the repository by adding the CWL_ICA_REPO_PATH env var
to your conda env activate.d directory. This variable is unset upon deactivate.
You MUST reactivate the environment after setting this value.

Options:

  • repo-path
    • Path to the local clone of the cwl-ica git repository.

Usage:

cwl-ica configure-repo \
  --repo-path /abs/path/to/repository \

configure tenant

Configure the path to the repository,
Path to repo is set in ${CONDA_PREFIX}/etc/conda/activate.d, therefore You MUST reactivate the environment after running this command.
The repository directory must already exist.

Options:

  • tenant-id
    • The tenant-id to be mapped
  • tenant-name
    • The name of the tenant to be mapped
  • set-as-default (optional)
    • Set this tenant as the default tenant
    • You will need to deactivate and reactivate the conda environment for this to take effect.

Usage:

cwl-ica configure-tenant \
  --tenant-id "<tenant-id>" \
  --tenant-name "<tenant-name>" \
  --set-as-default

configure user

Create a user in user.yaml. This adds the username, and user's email and user orcid identifier inside <CWL_ICA_REPO_PATH>/config/user.yaml The username, user email and identifier are present in the label of each tool and workflow when --username is specified in create-*-from-template commands.

Options:

  • username
    • The users name, can be "FirstName LastName"
  • email
    • The email address of the user
  • identifier (optional)
    • The ORCID identifier for your user
  • set-as-default (optional)
    • Set this user as the default user
    • You will need to deactivate / reactivate the conda environment for this to take effect.

Usage:

cwl-ica configure-user \
  --username "Alexis Lucattini" \
  --email "[email protected]" \
  --identifier "https://orcid.org/0000-0001-9754-647X" \
  --set-as-default

ICA Initialisers

project init

Initialise a project in project.yaml

This project must be registered in ICA, use ica projects list to view available projects.

Add in a project to 'project.yaml' in the configuration folder. The project name does not necessarily need to match the name of the project in ica, although that may be convenient. One must specify an api-key by specifying the --project-api-key-name variable s.t when CWL_ICA_API_KEY_SH is run with the value of --project-api-key-name set as the env var PROJECT_API_KEY_PATH, will return an api-key for this project. Your api-key is not saved in a file, it is just used to generate a project-level access token. --project-abbr is used for when multiple projects are under the one tenant. One cannot use the same workflow name across multiple projects in the one tenant.

Options:

  • project-id
    • Use ica projects list to view all available project names and their ids.
  • project-name
    • Use ica projects list to view all available project names and their ids
    • Also used by other commands as the name of a project.
    • The project-name does not necessarily need to be the name of te
  • access-token
    • Sets this env var as ICA_ACCESS_TOKEN_<PROJECT_NAME> in the conda env activate.d directory. Can be updated with cwl-ica project-update --access-token=<new-access-token>
  • is-production
    • Setting this parameter will register the project as a production project. Production projects do not cannot be synced with cwl-ica tool-sync or cwl-ica workflow-sync and new ica workflow-versions are instead automatically created on pushes to the main branch.
  • tenancy-id
    • Used when your organisation is mapped over multiple tenancy ids. Best to put it in now as a set / forget
  • tenancy-name
    • Can be used instead of tenancy-id if cwl-ica configure-tenant has been set

Usage:

# A non production project
cwl-ica project-init \
  --project-id xxxx-yyyy... \
  --project-name my-registered-project \
  --access-token <project-access-token>

# A production project
cwl-ica project-init \
  --project-id xxxx-yyyy... \
  --project-name my-registered-project \
  --access-token <project-access-token> \
  estis-production

category init

Initialises a category in ${CWL_ICA_REPO_PATH}/config/category.yaml.
Categories are like tags and can be used as a filter to search for workflows / tools.
A tool / workflow may have multiple categories.

Options

  • category-name
  • category-description (optional)

Usage:

# Create a category
cwl-ica category-init \
  --category-name "name of category" \
  --category-description "whats the theme"

Set Default Commands

set default tenant

Set the default tenant by specifying the tenant name.
Tenant must be registered in ${CWL_ICA_REPO_PATH}/config/tenant.yaml.
You must deactivate and reactivate the conda environment in order for this setting to take effect.

Options

  • tenant-name

Usage

cwl-ica set-default-tenant \
  --tenant-name "umccr"

set default project

Set the default project by specifying the project name.
Project must be registered in ${CWL_ICA_REPO_PATH}/config/project.yaml You must deactivate and reactivate the conda environment in order for this setting to take effect.

Options

  • tenant-name

Usage

cwl-ica set-default-project \
  --project-name "development"

set default user

Set the default user by specifying the user name.
Project must be registered in ${CWL_ICA_REPO_PATH}/config/user.yaml You must deactivate and reactivate the conda environment in order for this setting to take effect.

Options

  • tenant-name

Usage

cwl-ica set-default-user \
  --user-name "Alexis Lucattini"

List Commands

list categories

Get a list of categories.

List all categories in ${CWL_ICA_REPO_PATH}/config/category.yaml.

Options:

None

Usage:

cwl-ica list-categories

list projects

Get a list of projects.

List all projects in ${CWL_ICA_REPO_PATH}/config/project.yaml.

Options:

None

Usage:

cwl-ica list-project

list tenants

Get a list of tenants.

List all tenants in ${CWL_ICA_REPO_PATH}/config/tenant.yaml.

Options:

None

Usage:

cwl-ica list-tenant

list users

Get a list of users.

List all users in ${CWL_ICA_REPO_PATH}/config/user.yaml.

Options:

None

Usage:

cwl-ica list-user

CWL Creation Commands

create tool from template

Initialise a tool from the cwl templates.

We initialise a tool with all of the bells / schema and whistles. This creates a file under <CWL_ICA_REPO_PATH>/tools/<tool_name>/<tool_version>/<tool_name>__<tool_version>.cwl

The tool will have the bare minimum inputs and is ready for you to edit. This command does NOT register the tool under tool.yaml, should you change your mind, you can easily just delete the file. Remember that for each input and output that you MUST add in a label and doc attribute.

For readability purposes we recommend that you place the 'id' attribute of each input and output as the yaml key. Remember, that the username isn't necessarily the creator of the software, but the person building the CWLTool file.

The --username must be first registered in ${CWL_ICA_REPO_PATH}/config/user.yaml. You may use the command "cwl-ica configure-user" to do that.

We make sure that --tool-name and --tool-version are appropriate names for folders / files, no special characters or spaces.

Make sure that the --tool-version argument is in x.y.z format. If additional patch / metadata is required use x.y.z__<patch_string>.

Options

  • tool-name
  • tool-version
  • username

Usage

cwl-ica create-tool-from-template \
  --tool-name name-of-tool \
  --tool-version version-of-tool \
  --username "Alexis Lucattini"

create workflow from template

We initialise a workflow with all of the bells / schema and whistles. This creates a file under ${CWL_ICA_REPO_PATH}/workflows/<workflow_name>/<workflow_version>/<workflow_name>__<workflow_version>.cwl

The workflow will have the bare minimum inputs and is ready for you to edit. This command does NOT register the tool under workflow.yaml, should you change your mind, you can easily just delete the file. Remember that for each input and output that you add in a label attribute and a doc attribute.

For readability purposes we recommend that you place the 'id' attribute of each input and output as the yaml key. Remember, that the username isn't necessarily the creator of the software, but the person building the CWLWorkflow file.

The --username must be first registered in ${CWL_ICA_REPO_PATH}/config/user.yaml. You may use the command "cwl-ica configure-user" to do that.

We make sure that --workflow-name and --workflow-version are appropriate names for folders / files, no special characters or spaces.

Make sure that the --workflow-version argument is in x.y.z format. If additional patch / metadata information is required, then use x.y.z__<patch_string>.

Options

  • workflow-name
  • workflow-version
  • username

Usage

cwl-ica create-workflow-from-template \
  --workflow-name name-of-workflow \
  --workflow-version version-of-workflow \
  --username "Alexis Lucattini"

create expressions from template

We initialise a expression with all of the bells / schema and whistles. This creates a file under ${CWL_ICA_REPO_PATH}/expressions/<expression_name>/<expression_version>/<expression_name>__<expression_version>.cwl

The expression will have the bare minimum inputs and is ready for you to edit. This command does NOT register the expression under expression.yaml, should you change your mind, you can easily just delete the file. Remember that for each input and output that you add in a label attribute and a doc attribute.

For readability purposes we recommend that you place the 'id' attribute of each input and output as the yaml key. Remember, that the username isn't necessarily the creator of the software, but the person building the CWLExpression file.

The --username must be first registered in <CWL_ICA_REPO_PATH>/config/user.yaml. You may use the command "cwl-ica configure-user" to do that.

We make sure that --expression-name and --expression-version are appropriate names for folders / files, no special characters or spaces.

Make sure that the --expression-version argument is in x.y.z format. If additional patch is required use x.y.z__<patch_string>.

Options

  • expression-name
  • expression-version
  • username

Usage

cwl-ica create-expression-from-template \
  --expression-name name-of-expression \
  --expression-version version-of-expression \
  --username "Alexis Lucattini"

create schema from template

We initialise a schema with all of the bells / schema and whistles. This creates a file under <CWL_ICA_REPO_PATH>/schema/<schema_name>/<schema_version>/<schema_name>__<schema_version>.cwl

The schema will have the bare minimum inputs and is ready for you to edit. This command does NOT register the schema under schema.yaml, should you change your mind, you can easily just delete the file. Remember that for each input and output that you add in a label attribute and a doc attribute.

For readability purposes we recommend that you place the 'id' attribute of each input and output as the yaml key.

We make sure that --schema-name and --schema-version are appropriate names for folders / files, no special characters or spaces.

Make sure that the --schema-version argument is in x.y.z format. If additional patch is required use x.y.z__<patch_string>.

Options

  • schema-name
  • schema-version

Usage

cwl-ica create-schema-from-template \
  --schema-name name-of-schema \
  --schema-version version-of-schema

Validation Commands

tool validate

Validate a CWL File of CommandLineTool.
This must be done prior to registering the tool with the "cwl-ica tool-init" command. The CWL tool must be in the tools/ directory under ${CWL_ICA_REPO_PATH}

See [contributions][contributions] for more information / example of a valid cwltool.

The basics are:

  • tool must pass cwltool --strict --validate.
  • inputs, outputs, requirements and hints section should all be in [key-format].
  • id, label and doc must be present for tool doc.
  • label and doc must be present for each input and output of the tool.
  • A list of successCodes must be present.
  • An author description must be present.
  • Resource requirements must be specified in 'hints' section.

Options:

  • tool-path
    • Path to cwl tool. Must be inside ${CWL_ICA_REPO_PATH}.

Usage

cwl-ica tool-validate \
  --tool-path /path/to/tool.cwl

workflow validate

Validate a CWL File of CommandLineTool. This must be done prior to registering the workflow with the "cwl-ica workflow-init" command. The CWL workflow must be in the workflows/ directory under ${CWL_ICA_REPO_PATH}

[contributions][contributions] for more information / example of a valid cwl workflow.

The basics are:

  • workflow must pass cwltool --strict --validate.
  • workflow in packed format must also pass cwltool --strict --validate.
  • inputs, outputs, steps, requirements section should all be in [key-format].
  • id, label and doc must be present for workflow doc.
  • label and doc must be present for each input and output and steps.
  • An author description must be present.
  • Each step must link to a separate file. No step is to be written 'in-place'.

Options

  • workflow-path:
    • Path to cwl workflow. Must be inside ${CWL_ICA_REPO_PATH}

Usage:

cwl-ica workflow-validate \
  --workflow-path /path/to/workflow.cwl

expression validate

Validate a CWL File of CommandLineTool. This must be done prior to registering the workflow with the "cwl-ica workflow-init" command. The CWL workflow must be in the workflows/ directory under ${CWL_ICA_REPO_PATH}.

[contributions][contributions] for more information / example of a valid cwl expression.

The basics are:

  • expression must pass cwltool --strict --validate.
  • expression in packed format must also pass cwltool --strict --validate.
  • expression must contain 'id', 'label' and 'doc attributes'
  • An author description must be present.
  • Resource requirements must be specified in 'hints' section.
  • label and doc must be present for each input and output of the expression.

Options

  • expression-path:
    • Path to cwl expression. Must be inside ${CWL_ICA_REPO_PATH}

Usage:

cwl-ica expression-validate \
  --expression-path /path/to/expression.cwl

schema validate

Validate a CWL File of CommandLineExpression.
This must be done prior to registering the schema with the "cwl-ica schema-init" command. The CWL schema must be in the schemas/ directory under ${CWL_ICA_REPO_PATH}.

[contributions][contributions] for more information / example of a valid cwl schema. The basics are:

  • schema must pass cwltool --strict --validate.
  • schema in packed format must also pass cwltool --strict --validate.
  • inputs, outputs, steps, requirements section should all be in [key-format].
  • id, label and doc must be present for schema doc.
  • label and doc must be present for each input and output and steps.
  • An author description must be present.
  • Each step must link to a separate file. No step is to be written 'in-place'.

Options

  • schema-path:
    • Path to cwl schema. Must be inside ${CWL_ICA_REPO_PATH}

Usage:

cwl-ica schema-validate \
  --schema-path /path/to/workflow.cwl

Initialisers

tool init

Initialise / register a tool at tool.yaml and at projects.yaml (if --projects used).

Add in a tool to tool.yaml in the configuration folder.
The should have been created with cwl-ica create-tool-from-template and then validated with tool-validate. We will also run a validation on the tool.
For each project listed, a new workflow id is created if no other version of this tool exists. A workflow version is also created for non-production projects.

When --projects=all, --tenants is checked and only all projects across specified tenants are updated.
If --tenants=all or --tenants is not defined and ${CWL_ICA_DEFAULT_TENANT}
is not defined then all projects across all tenants are updated.

Determine which projects the tool will be present in. Options

  • tool-path

    • Path to tool

  • projects (optional)

    • Comma separated list of ICA projects to add tool to.
    • Use 'all' to select all projects in projects.yaml.
    • For projects registered as 'production-projects' in projects.yaml. The tool will be registered but not synced. This is done only on a push to the main branch, the first seven characters of the commit id will be used as a suffix for the workflow version.

  • tenants (optional)

    • List of tenants to filter by

  • categories (optional)

    • List of categories to add to the workflow

Usage

cwl-ica tool-init \
  --projects project[,additional-projects] \
  --tool-path /path/to/tool.cwl \
  --categories comma,separated,list

workflow init

Add in a workflow to 'workflow.yaml' in the configuration folder. The should have been created with cwl-ica create-workflow-from-template and then validated with workflow-validate. We will also run a validation on the workflow. For each project listed, a new workflow id is created if no other version of this workflow exists. A workflow version is also created for non-production projects.

When --projects=all, --tenants is checked and only all projects across specified tenants are updated.
If --tenants=all or --tenants is not defined and ${CWL_ICA_DEFAULT_TENANT}
is not defined then all projects across all tenants are updated.

Determine which projects the workflow will be present in.

Options

  • workflow-path

    • Path to workflow

  • projects (optional)

    • Comma separated list of ICA projects to add workflow to.
    • Use 'all' to select all projects in projects.yaml.
    • For projects registered as 'production-projects' in projects.yaml. The tool will be registered but not synced. This is done only on a push to the main branch, the first seven characters of the commit id will be used as a suffix for the workflow version.

  • tenants (optional)

    • List of tenants to filter by

  • categories (optional)

    • List of categories to add to the workflow

Usage

cwl-ica workflow-init \
  --projects project[,additional-projects] \
  --tenants
  --workflow-path /path/to/workflow.cwl \
  --categories comma,separated,list

expression init

Add in an expression to 'expression.yaml' in the configuration folder. This should have been created with cwl-ica create-expression-from-template and then validated with cwl-ica expression-validate. We will also run a validation on the expression.

Options

  • expression-path
    • Path to expression

Usage

cwl-ica expression-init \
  --expression-path /path/to/expression.cwl

schema init

Initialise / register a schema at schema.yaml

Options

  • schema-path
    • Path to expression

Usage

cwl-ica schema-init \
  --schema-path /path/to/schema.cwl

Sync commands

tool sync

Sync a tool to tool.yaml in the configuration folder The tool should have been created with cwl-ica create-tool-from-template and then validated with tool-validate. The tool must exist in tool.yaml. Please use cwl-ica tool-init to add this tool to tool.yaml. We will also run a validation on the tool before syncing. For each project listed, the workflow version is patched with this version.

When --projects=all, --tenants is checked and only all projects across specified tenants are updated.
If --tenants=all or --tenants is not defined and ${CWL_ICA_DEFAULT_TENANT}
is not defined then all projects across all tenants are updated.

Options:

  • tool-path
    • The path to the CWL tool to sync to projects
  • projects (optional)
    • Comma separated list of ICA projects to add workflow to.
    • Use 'all' to select all projects in projects.yaml.
    • Production projects will NOT be synced
  • tenants (optional)
    • List of tenants, used only if --projects=all.

Usage:

cwl-ica tool-sync \
  --tool-path /path/to/tool.cwl \
  --projects project[,project2] \

workflow sync

Sync a workflow to 'workflow.yaml' in the configuration folder The workflow should have been created with cwl-ica create-workflow-from-template and then validated with workflow-validate. The workflow must exist in workflow.yaml. Please use cwl-ica workflow-init to add this workflow to workflow.yaml. We will also run a validation on the workflow before syncing. For each project listed, the workflow version is patched with this version. When --projects=all, --tenants is checked. If --tenants=all or --tenants is not defined and ${CWL_ICA_DEFAULT_TENANT} is not defined then all projects across all tenants are updated.

Options:

  • workflow-path
    • The path to the CWL workflow to sync to projects
  • projects (optional)
    • Comma separated list of ICA projects to add workflow to.
    • Use 'all' to select all projects in projects.yaml.
    • Production projects will NOT be synced
  • tenants (optional)
    • List of tenants, used only if --projects=all.

Usage:

cwl-ica workflow-sync \
  --workflow-path /path/to/workflow \
  --projects project[,project2]

expression sync

Sync a expression to expression.yaml in the configuration folder The expression should have been created with cwl-ica create-expression-from-template and then validated with expression-validate. The expression must exist in expression.yaml. Please use cwl-ica expression-init to add this expression to expression.yaml. We will also run a validation on the expression before syncing.

Options:

  • expression-path
    • The path to the CWL expression to sync

Usage:

cwl-ica expression-sync \
  --expression-path /path/to/expression.cwl \

schema sync

Sync a expression to schema.yaml in the configuration folder The schema should have been created with cwl-ica create-schema-from-template and then validated with schema-validate. The schema must exist in schema.yaml. Please use cwl-ica schema-init to add this schema to schema.yaml. We will also run a validation on the schema before syncing.

Options:

  • schema-path
    • The path to the CWL schema to sync

Usage:

cwl-ica schema-sync \
  --schema-path /path/to/schema.cwl \

Add to project commands

add tool to project

Adds an existing tool to a project. The tool must be registered in tool.yaml.

Please use cwl-ica tool-init to add this tool to tool.yaml. We will also run a validation on the tool before syncing. For each project listed, the workflow version is patched with this version.

Options:

  • tool-path
    • The path to the cwl tool
  • project
    • The name of the project to register the tool with

Usage:

cwl-ica add-tool-to-project \
  --tool-path /path/to/tool.cwl \
  --project project

add workflow to project

Add an existing workflow from workflow.yaml to a project The workflow must exist in workflow.yaml. Please use cwl-ica workflow-init to add this workflow to workflow.yaml. We will also run a validation on the workflow before syncing. For each project listed, the workflow version is patched with this version.

Options:

  • workflow-path
    • The path to the cwl workflow
  • project
    • The name of the project to register the workflow with

Usage:

cwl-ica add-workflow-to-project \
  --workflow-path /path/to/workflow.cwl \
  --project project

Add linked project

Add a linked project to an existing project in project.yaml.
This will update the acl attribute of all existing workflows and workflow versions to include the linked project.

Subsequent workflows and workflow versions will be added to the linked project

Options:

  • src-project
    • The name of the project in project.yaml
  • target-project
    • The project ID of the linked project

Add to category commands

Adds an existing category to an existing tool or workflow.

Add category to tool

Add an existing category to an existing tool

Options

  • --tool-name
    • Name of an exisiting tool
  • --category-name
    • Name of existing category

Usage

cwl-ica add-category-to-tool \
  --tool-name  existing-tool\
  --category-name existing-category

Add category to workflow

Add an existing category to an existing worklfow

Options

  • --workflow-name
    • Name of an exisiting workflow
  • --category-name
    • Name of existing category

Usage

cwl-ica add-category-to-workflow \
  --workflow-name  existing-workflow\
  --category-name existing-category

Add maintainer commands

This command can be used to acknowledge a user as a maintainer of following resources

add maintainer to expression

Acknowledge a user as a maintainer of a CWL expression

Options

  • --expression-path
    • Path to an existing expression file
  • --username
    • Username of a maintainer

Usage

cwl-ica add-maintainer-to-expression \
  --expression-path /path/to/expression.cwl\
  --username="maintainer_name"

add maintainer to tool

Acknowledge a user as a maintainer of a CWL tool

Options

  • --tool-path
    • Path to an existing tool file
  • --username
    • Username of a maintainer

Usage

cwl-ica add-maintainer-to-tool \
  --tool-path /path/to/tool.cwl\
  --username="maintainer_name"

add maintainer to workflow

Acknowledge a user as a maintainer of a CWL workflow

Options

  • --workflow-path
    • Path to an existing workflow file
  • --username
    • Username of a maintainer

Usage

cwl-ica add-maintainer-to-workflow \
  --workflow-path /path/to/workflow.cwl\
  --username="maintainer_name"

Coming soon 🚧

register tool run instance id

Register a run instance of a tool of the tool cpu / mem usage over time. This writes out a json file under runs/tools/tool-path.<wfr.id>.json This can provide a user with estimated run time metrics and suggest overrides as required.

Options:

  • tool-run-instance-id
    • The run instance id (must start with 'wfr.')
  • project
    • The project the run was launched on.

Usage:

cwl-ica register-tool-run-instance-id \
  --tool-run-instance-id wfr... \
  --project project

register workflow run instance id

Register a run instance of a workflow, collates the cpu / mem usage over time of each task run as well as a stacked area chart of the total resources used. As such can provide user with estimated run time metrics.
This writes out a json file under runs/workflows/workflow-path.<wfr-id>.json.

get workflow step ids

Get the ids of all steps in a workflow.
Prints two - tab delimited column of step id, step label.

The output ids can be used to set overrides for a cwl workflow.

Options:

  • workflow-path
    • Path to workflow.cwl

Usage:

cwl-ica get-workflow-step-ids \
  --workflow-path /path/to/workflow.cwl

copy-tool-submission-template

Copy the tool submission template for a tool run

Options:

  • tool-path
    • Path to tool.cwl
  • project
    • project name
  • tool-run-instance-id
    • The run instance id to copy from
  • output-file (optional)
    • json stdout if not specified

Usage:

cwl-ica copy-tool-submission-template \
  --tool-path /path/to/tool.cwl \
  --project project \
  --tool-run-instance-id wfr... \
  --output-file input.json

copy-workflow-submission-template

Copy the workflow submission template for a workflow run

Options:

  • workflow-path
    • Path to a workflow.cwl
  • project
    • project name
  • workflow run instance id
    • The run instance id to copy from
  • output-file (optional)
    • json stdout if not specified

Usage:

cwl-ica copy-workflow-submission-template \
  --workflow-path /path/to/workflow.cwl \
  --project project \
  --workflow-run-instance-id wfr...
  --output-file input.json

GitHub Actions

There are five github actions subcommands that do not have to be remembered by the user. These run only on pull requests and pushes to the main branch.

  • github-actions tools-update
    • Updates all tools for all projects (including production projects)
    • Run on push to main branch
  • github-actions workflows-update
    • Updates all workflows for all projects (including production projects)
    • Run on push to main branch
  • build-catalogue
    • Updates the markdown workflows catalogue.
    • Run on push to main branch
  • check-pull-request
    • Validate each workflow in workflow.yaml
    • Validate each tool in tool.yaml
    • List / comment workflows that have already been updated on ICA
    • List / comment workflows that are going to be updated for each project via md5sum against md5sum on main branch.
    • Run on initialisation of pull request / commits to pull request to main branch
  • postman-generattor
    • Generate a postman json from a template that takes in a projects' workflows, workflow versions, and run ids and generates an importable postman json file.
    • Run on push to main branch
Clone this wiki locally