Skip to content

Latest commit

 

History

History
73 lines (40 loc) · 6.57 KB

execute-the-app-package-using-airflow.md

File metadata and controls

73 lines (40 loc) · 6.57 KB
Raw

Execute the App Package using Airflow

Use the Airflow UI to execute your application package. To do this you will use an Airflow DAG (Directed Acyclic Graph), a process that manages the execution of an application or workflow, that can run any Application Package by specifying the two inputs described below.

NOTE: In the future, each Application Package or workflow (series of Application Packages) will be able to have its own DAG in Airflow with a custom input form.

Follow the steps below:

  1. Log into Airflow UI using the link below.
  • Link to Airflow UI: Airflow UI
  • NOTE: There isn’t a Username and Password for each User yet. Ask for the temporary Username and Password.

The login for Airflow is not the same as your Unity login. Please ask a Unity team member for the login.

  1. Select cwl_dag from the Airflow DAG-list screen as shown in the figure below.

Choose cwl_dag

  1. In the upper right is a “▶” button to trigger (execute) this DAG as shown in the figure below.
  • NOTE: The page for “cwl_dag” will show a history of runs in a bar-chart at the left, and several informational panels in the middle.

Press the Play button in the upper right

  1. This will open up the Trigger DAG page with five inputs as shown below as shown below

DAG execution page (Trigger DAG)

  1. Paste the URL to the “unity-example-application” workflow.cwl file into CWL workflow field:
  • NOTE: Review Step 2 under Optional Steps: Modify the Example Application Package to find out how to retrieve the URL.
  • Here’s the CWL workflow for this exercise: http://awslbdockstorestack-lb-1429770210.us-west-2.elb.amazonaws.com:9998/api/ga4gh/trs/v2/tools/%23workflow%2Fdockstore.org%2Fmike-gangl%2Funity-example-application/versions/5/PLAIN-CWL/descriptor/%2Fworkflow.cwl
  1. Paste either the URL of the “input file” or the JSON string content directly into the CWL workflow parameters field:
  • Here’s the URL of the CWL workflow parameters for this exercise: https://raw.githubusercontent.com/unity-sds/unity-example-application/main/test/ogc_app/unity-example-app.test.template.json
  • See the figure below for example:

  • If you need to modify the JSON file:
  1. Copy the unity-example-app.test.template.json locally
  2. replace <PROJECT> with your project (e.g. emit, asips, unity)
  3. replace <venue> with your venue (e.g. dev, int, ops)
  4. copy and paste the new json (with your project, venue substitutions) into the required “CWL workflow parameters” box
  • See the figure below for example:

  1. Leave the remaining three fields with their default values for this exercise as shown in the figure below (container memory: 4Gi, CPUs: 4, and container storage: 10Gi)
  • NOTE: These fields allow User to modify the computing resources available to a Job.

  1. Press the Trigger button to execute a job as shown in figure below

  1. You will return to the DAG detail page and you’ll see a new light-green bar representing the new job as shown in the figure below.

  1. If you click the new light-green bar, or the squares under it, you will see the main information panel update with status details on your new job as shown in the figure below.
  • NOTE: Wait until the job finishes. If any of the squares turn red, that step has failed. You can see more by going through the panels such as Graph or Audit Log.

\