Getting_Started

Getting Started

• Installation
• Configuration
• Initialise a project
• Building your first tool
• Registering your tool
• Building your first workflow
• Registering your workflow
• Syncing your tools / workflows with ICA.
• Show others how to run your workflow / tool
• Optimising your workflow
• Running a workflow

Installation

Install the latest release by heading to the [releases page][releases] and downloading the latest installation script.

You will need the following prerequisites:

• conda
• jq (optional but very handy for many shortcuts)

After downloading the installation script run:

bash install-cwl-ica-latest.sh

The installation script will then create a conda environment called cwl-ica.

You will need to activate this environment with the following command:

conda activate cwl-ica

Configuration

You will need to also clone this repo to your computer:

If you are a member of the UMCCR domain or have forked this repo s.t you own it, use the following command:

# subout the umccr repository for your own in the case of forking
git clone [email protected]:umccr/cwl-ica.git

You will now need to run the configuration command so your conda environment knows where your local clone is:

# Ensure your cwl-ica conda env has first been activated with "conda activate cwl-ica"
cwl-ica configure --repo-path /path/to/local-repository-clone

Great job! You've now configured your project.

You will need to reactivate your environment in order to complete the configuration with:

conda deactivate
conda activate cwl-ica

Initialise a project

First check out the list of registered projects in the repo with:

cat "${CWL_ICA_REPO_PATH}/config/projects.yaml"

If you would like to add a project run the following command:

cwl-ica project-init \
  --project-id xxxx-yyyy... \
  --project-name my-registered-project \
  --access-token <project-access-token>

To determine the project id, and project name you will need to run ica projects list.

To obtain the access token, you will need to log in to relevant project context and then running the following command:

ica tokens create --api-key="<your api key>"

This will place the token in "${CONDA_PREFIX}/etc/conda/activate.d" directory.

Building your first tool

First we use the cwl-ica create-tool-from-template command to create a file that we can expand on to build our first tool.

This will automatically create an id, label and doc for us, along with the author metadata namespaces for us to fill in.

The following command will create a tool under tools/tabix/0.2.6/tabix-0.2.6.cwl

cwl-ica create-tool-from-template \
  --tool-name tabix \
  --tool-version 0.2.6

Fill out the rest of the tool and then validate it. You should also test the tool locally (if possible).

cwl-ica tool-validate can be the most labourious part of the process but for good reason. No one else will use your tool if it's not documented properly.

Check out [contributions][contributions.md] or our [examples][examples] section for more help on mastering your first cwl tool.

Registering your tool

Now you've validated your tool, it's time to "register" it. This will:

1. Create an entry in tool.yaml for this cwl tool.
2. Create a workflow ID, and workflow version for the tool on ICA.
3. Keep the tool up-to-date on ICA.
4. Create a user-friendly markdown document when pushed to the main branch.

You can register your tool with cwl-ica tool-init. If you have an existing tool on ICA, you may connect it to this tool with cwl-ica tool-add. Likewise, if you have an existing tool with a new project, you may connect this tool to your new ICA project with cwl-ica add-tool-to-project.

Building your first workflow

If you've made it to this stage, congratulations! You've built a suite of tools and ready to stitch them together as a workflow.

Initialise the workflow through cwl-ica create-workflow-from-template. You will need to also 'validate' your workflow with cwl-ica workflow-validate.
This may be pretty tedious and is easier if you've first 'validated' all of your tools.

Once you've successfully run cwl-ica workflow-validate, it's time to register your workflow.

Registering your workflow

Like a tool registry, registering your workflow will also keep it in sync on ICA, and create a user-friendly markdown document on the workflow when pushed to the main branch.

You can register your tool with cwl-ica workflow-init. If you have an existing workflow on ICA, you may connect it to this workflow with cwl-ica workflow-add. Likewise, if you have an existing workflow with a new project, you may connect this workflow to your new ICA project with cwl-ica add-workflow-to-project.

Syncing your tools / workflows with ICA.

For non-production projects, tools and workflows will sync with the registered workflow id and workflow version on each push to the main branch. You may also 'sync' your tool with the following commands:

cwl-ica tool-sync or cwl-ica workflow-sync.

For production tools / workflows, you will need to first push to the main branch which will create a new version suffix based on the git commit of the merge.

Since all pushes to the main branch are required to be a pull-request, it is recommended the workflow has been first been fully tested in a non-production project.

Show others how to run your workflow / tool

Registering a run instance of your workflow / tool will guide others how your workflow should be set up to run.
For tools, this means a plot showing the cpu and mem usage over time along with the duration of the tool length.
For workflows, this means a stacked bar chat of the cpu / mem usage over time along with the duration of the workflow.

To register a run instance use either:

cwl-ica register-tool-run-instance-id or cwl-ica register-workflow-run-instance-id.

Optimising your workflow

One can use the overrides setting to optimise the cpu and mem usage (or even change the docker container used by a step in a workflow or tool).

In order to view the step ids of a workflow, run cwl-ica get-workflow-step-ids.

Use these in the overrides settings to adjust the engine parameters for this step of a workflow.

Running a workflow

There are two recommended ways of running your workflow.

If you are unsure on how to run a workflow, check out the ica catalogue page. Which should have all of the documentation that you need.

1. Through postman

   On each push to the main branch, the GitHub actions builds a postman-json for each project.
   This can be imported into postman and allows a user to navigate through all registered workflows and to launch new workflows from the templates of registered runs for each workflow.
   Once imported, you will need to update ICA_ACCESS_TOKEN in the variable in the top directory with a token that belongs to the relevant project.

2. Copying a tool submission template

   One may also use the cwl-ica register-tool-run-instance-id or cwl-ica register-workflow-run-instance-id to set up template to submit from.
   Then edit the submission json and run via the following command.
   This can then be launched via the ica workflows versions launch <workflow-id> <workflow-version> input.json