SPS release 24.3

CHANGELOG.md

@@ -10,14 +10,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ## Tags
 
 - SPS Version 2.2.0
-- OGC API Version 1.1.0
-- OGC Python Client Version 1.1.0
+- OGC API Version 2.0.0
+- OGC Python Client Version 2.0.0
 
 ## Repositories
 
 - unity-sps: <https://github.com/unity-sds/unity-sps/releases/tag/2.2.0>
-- unity-sps-ogc-processes-api: <https://github.com/unity-sds/unity-sps-ogc-processes-api/releases/tag/1.1.0>
-- unity-sps-ogc-processes-api-client-python: <https://github.com/unity-sds/unity-sps-ogc-processes-api-client-python/releases/tag/1.1.0>
+- unity-sps-ogc-processes-api: <https://github.com/unity-sds/unity-sps-ogc-processes-api/releases/tag/2.0.0>
+- unity-sps-ogc-processes-api-client-python: <https://github.com/unity-sds/unity-sps-ogc-processes-api-client-python/releases/tag/2.0.0>
 
 ## Epics
 
@@ -27,8 +27,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   - [[New Feature]: Increase ephemeral disk space for Airflow workers](https://github.com/unity-sds/unity-sps/issues/152)
   - [[New Feature]: Enable users to select the EC2 type to execute a workload](https://github.com/unity-sds/unity-sps/issues/153)
   - [[New Feature]: Set the DAG run status to "failed" if the main worker task failed](https://github.com/unity-sds/unity-sps/issues/189)
+  - [[New Feature]: Demonstrate use of ECR within an Airflow DAG (https://github.com/unity-sds/unity-sps/issues/186)
 - EPIC: `Airflow/WPS-T Integration`
   - [[New Feature]: Create test to deploy, execute and undeploy the CWL DAG](https://github.com/unity-sds/unity-sps/issues/131)
+  - [[New Feature]: Enable execution of OGC data processing requests with arbitrary parameter values](https://github.com/unity-sds/unity-sps/issues/129)
 - EPIC: `Production Venue Deployments`
   - [[New Feature]: Airflow HTTPD Proxy development and configuration](https://github.com/unity-sds/unity-sps/issues/125)
   - [[New Feature]: Expose SPS health check endpoints](https://github.com/unity-sds/unity-sps/issues/127)
@@ -37,6 +39,25 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   - [[New Feature]: Review the SPS GitBook documentation](https://github.com/unity-sds/unity-sps/issues/118)
   - [[New Feature]: Store SPS Terraform state on S3](https://github.com/unity-sds/unity-sps/issues/132)
   - [[New Feature]: Parametrize the SPS Integration Tests](https://github.com/unity-sds/unity-sps/issues/155)
+  - [[New Feature] Upgrade SPS to latest version of Airflow 2.10.0](https://github.com/unity-sds/unity-sps/issues/195)
+
+## Docker Containers
+
+- ghcr.io/unity-sds/unity-sps/sps-airflow:2.2.0
+- ghcr.io/unity-sds/unity-sps/sps-docker-cwl:2.2.0
+- ghcr.io/unity-sds/unity-sps-ogc-processes-api/unity-sps-ogc-processes-api:2.0.0
+
+## Documentation
+
+- For Administrators:
+  - [SPS Deployment with Terraform](https://app.gitbook.com/o/xZRqGQeQXJ0RP4VMj7Lq/s/UMIRhLdbRQTvMWop8Il9/developer-docs/science-processing/docs/admin-guide/sps-deployment-with-terraform)
+  - [Interacting with an Existing SPS Deployment](https://app.gitbook.com/o/xZRqGQeQXJ0RP4VMj7Lq/s/UMIRhLdbRQTvMWop8Il9/developer-docs/science-processing/docs/admin-guide/interacting-with-an-existing-sps-deployment)
+  - [SPS Airflow Custom Docker Image Build Instructions](https://app.gitbook.com/o/xZRqGQeQXJ0RP4VMj7Lq/s/UMIRhLdbRQTvMWop8Il9/developer-docs/science-processing/docs/admin-guide/sps-airflow-custom-docker-image-build-instructions)
+  - [SPS Post Deployment Operations](https://app.gitbook.com/o/xZRqGQeQXJ0RP4VMj7Lq/s/UMIRhLdbRQTvMWop8Il9/developer-docs/science-processing/docs/admin-guide/sps-post-deployment-operations)
+- For Deverlopers:
+  - [Tutorial: Deploy, Execute, and Undeploy a Process using the OGC API - Processes](https://app.gitbook.com/o/xZRqGQeQXJ0RP4VMj7Lq/s/UMIRhLdbRQTvMWop8Il9/developer-docs/science-processing/docs/developers-guide/tutorial-deploy-execute-and-undeploy-a-process-using-the-ogc-api-processes)
+- For Users:
+  - [Tutorial: Register and Execute a CWL Workflow](https://app.gitbook.com/o/xZRqGQeQXJ0RP4VMj7Lq/s/UMIRhLdbRQTvMWop8Il9/developer-docs/science-processing/docs/users-guide/tutorial-register-and-execute-a-cwl-workflow)
 
 # [Unity Release 24.2] - 2024-07-01
 
@@ -70,6 +91,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   - [[Task] Add TESTING.md file to SPS repo](https://github.com/unity-sds/unity-sps/issues/99)
 - EPIC: `SPS Infrastructure`
   - [[New Feature] Store SPS Terraform state on S3](https://github.com/unity-sds/unity-sps/issues/132)
+- EPIC: `SPS Security`
+  - [[Bug]: Upgrade EKS 1.27 AMIs](https://github.com/unity-sds/unity-sps/issues/159)
+  - [[Bug]: Upgrade to EKS 1.29 AMIs](https://github.com/unity-sds/unity-sps/issues/206)
 
 ## Docker Containers
 

airflow/dags/cwl_dag.py

@@ -26,7 +26,7 @@
 POD_NAMESPACE = "sps"
 POD_LABEL = "cwl_task"
 # SPS_DOCKER_CWL_IMAGE = "ghcr.io/unity-sds/unity-sps/sps-docker-cwl:2.1.0"
-SPS_DOCKER_CWL_IMAGE = "ghcr.io/unity-sds/unity-sps/sps-docker-cwl:2.2.0-beta-3"
+SPS_DOCKER_CWL_IMAGE = "ghcr.io/unity-sds/unity-sps/sps-docker-cwl:2.2.0"
 
 NODE_POOL_DEFAULT = "airflow-kubernetes-pod-operator"
 NODE_POOL_HIGH_WORKLOAD = "airflow-kubernetes-pod-operator-high-workload"

airflow/dags/docker_cwl_pod.yaml

@@ -21,7 +21,7 @@ spec:
 
   containers:
   - name: cwl-docker
-    image: ghcr.io/unity-sds/unity-sps/sps-docker-cwl:2.2.0-beta-1
+    image: ghcr.io/unity-sds/unity-sps/sps-docker-cwl:2.2.0
     imagePullPolicy: Always
     command: ["/usr/share/cwl/docker_cwl_entrypoint.sh"]
     securityContext:

airflow/dags/sbg_L1_to_L2_e2e_cwl_step_by_step_dag.py

@@ -23,7 +23,7 @@
 # The Kubernetes namespace within which the Pod is run (it must already exist)
 POD_NAMESPACE = "sps"
 POD_LABEL = "sbg_task"
-SPS_DOCKER_CWL_IMAGE = "ghcr.io/unity-sds/unity-sps/sps-docker-cwl:2.2.0-beta-1"
+SPS_DOCKER_CWL_IMAGE = "ghcr.io/unity-sds/unity-sps/sps-docker-cwl:2.2.0"
 
 # The path of the working directory where the CWL workflow is executed
 # (aka the starting directory for cwl-runner).

airflow/dags/sbg_preprocess_cwl_dag.py

@@ -17,7 +17,7 @@
 # The Kubernetes namespace within which the Pod is run (it must already exist)
 POD_NAMESPACE = "sps"
 POD_LABEL = "sbg_preprocess_task"
-SPS_DOCKER_CWL_IMAGE = "ghcr.io/unity-sds/unity-sps/sps-docker-cwl:2.2.0-beta-1"
+SPS_DOCKER_CWL_IMAGE = "ghcr.io/unity-sds/unity-sps/sps-docker-cwl:2.2.0"
 
 # The path of the working directory where the CWL workflow is executed
 # (aka the starting directory for cwl-runner).

ogc-application-packages/cwl_dag.json

@@ -0,0 +1,74 @@
+{
+  "executionUnit": {
+    "image": "ghcr.io/unity-sds/unity-sps/sps-docker-cwl:2.1.0",
+    "type": "docker"
+  },
+  "processDescription": {
+    "description": "This process executes any CWL workflow.",
+    "id": "cwl_dag",
+    "inputs": {
+      "cwl_args": {
+        "description": "The URL of the CWL workflow's YAML parameters file",
+        "maxOccurs": 1,
+        "minOccurs": 1,
+        "schema": {
+          "format": "uri",
+          "type": "string"
+        },
+        "title": "CWL Workflow Parameters URL"
+      },
+      "cwl_workflow": {
+        "description": "The URL of the CWL workflow",
+        "maxOccurs": 1,
+        "minOccurs": 1,
+        "schema": {
+          "format": "uri",
+          "type": "string"
+        },
+        "title": "CWL Workflow URL"
+      },
+      "request_cpu": {
+        "description": "The number of CPU cores requested for the job",
+        "maxOccurs": 1,
+        "minOccurs": 1,
+        "schema": {
+          "type": "string"
+        },
+        "title": "Requested CPU"
+      },
+      "request_memory": {
+        "default": "8Gi",
+        "description": "The amount of memory requested for the job",
+        "maxOccurs": 1,
+        "minOccurs": 1,
+        "schema": {
+          "type": "string"
+        },
+        "title": "Requested Memory"
+      },
+      "request_storage": {
+        "description": "The amount of storage requested for the job",
+        "maxOccurs": 1,
+        "minOccurs": 1,
+        "schema": {
+          "type": "string"
+        },
+        "title": "Requested Storage"
+      }
+    },
+    "jobControlOptions": [
+      "async-execute"
+    ],
+    "outputs": {
+      "result": {
+        "description": "The result of the SBG Preprocess Workflow execution",
+        "schema": {
+          "$ref": "some-ref"
+        },
+        "title": "Process Result"
+      }
+    },
+    "title": "Generic CWL Process",
+    "version": "1.0.0"
+  }
+}

ogc-application-packages/karpenter_test.json

@@ -0,0 +1,36 @@
+{
+  "executionUnit": {
+    "image": "busybox",
+    "type": "docker"
+  },
+  "processDescription": {
+    "description": "This process tests Karpenter node provisioning with different instance types.",
+    "id": "karpenter_test",
+    "inputs": {
+      "placeholder": {
+        "default": 1,
+        "description": "A placeholder parameter",
+        "maxOccurs": 1,
+        "minOccurs": 1,
+        "schema": {
+          "type": "integer"
+        },
+        "title": "Placeholder"
+      }
+    },
+    "jobControlOptions": [
+      "async-execute"
+    ],
+    "outputs": {
+      "result": {
+        "description": "The result of the Karpenter test execution",
+        "schema": {
+          "$ref": "some-ref"
+        },
+        "title": "Process Result"
+      }
+    },
+    "title": "Karpenter Test Process",
+    "version": "1.0.0"
+  }
+}

ogc-application-packages/sbg_preprocess_cwl_dag.json

@@ -0,0 +1,48 @@
+{
+  "executionUnit": {
+    "image": "ghcr.io/unity-sds/unity-sps/sps-docker-cwl:2.2.0",
+    "type": "docker"
+  },
+  "processDescription": {
+    "description": "This process executes the SBG Preprocess Workflow using CWL.",
+    "id": "sbg_preprocess_cwl_dag",
+    "inputs": {
+      "cwl_args": {
+        "default": "https://raw.githubusercontent.com/unity-sds/sbg-workflows/main/preprocess/sbg-preprocess-workflow.dev.yml",
+        "description": "The SBG Pre-process YAML parameters URL",
+        "maxOccurs": 1,
+        "minOccurs": 1,
+        "schema": {
+          "format": "uri",
+          "type": "string"
+        },
+        "title": "CWL Workflow Parameters"
+      },
+      "cwl_workflow": {
+        "default": "https://raw.githubusercontent.com/unity-sds/sbg-workflows/main/preprocess/sbg-preprocess-workflow.cwl",
+        "description": "The SBG Pre-process CWL workflow URL",
+        "maxOccurs": 1,
+        "minOccurs": 1,
+        "schema": {
+          "format": "uri",
+          "type": "string"
+        },
+        "title": "CWL Workflow"
+      }
+    },
+    "jobControlOptions": [
+      "async-execute"
+    ],
+    "outputs": {
+      "result": {
+        "description": "The result of the SBG Preprocess Workflow execution",
+        "schema": {
+          "$ref": "some-ref"
+        },
+        "title": "Process Result"
+      }
+    },
+    "title": "SBG Preprocess CWL Workflow",
+    "version": "1.0.0"
+  }
+}

pyproject.toml

@@ -4,9 +4,12 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "unity-sps"
-version = "2.1.0"
+version = "2.2.0"
 authors = [
     { name = "Drew Meyers", email = "[email protected]" },
+    { name = "Luca Cinquini", email = "[email protected]" },
+    { name = "Nikki Tebaldi", email = "[email protected]" },
+    { name = "Brad Lunsford", email = "[email protected]" }
 ]
 description = "The science processing service area of Unity."
 classifiers = [
@@ -37,11 +40,13 @@ test = [
     "pytest-bdd==7.1.1",
     "pytest-mock==3.12.0",
     "requests==2.31.0",
-    "apache-airflow==2.9.1",
+    "apache-airflow==2.10.0",
     "kubernetes==29.0.0",
     "boto3==1.34.46",
     "backoff==2.2.1",
-    "apache-airflow-providers-cncf-kubernetes==8.2.0"
+    "apache-airflow-providers-cncf-kubernetes==8.4.1",
+    "unity_sps_ogc_processes_api_python_client @ git+https://github.com/unity-sds/[email protected]"
+
 ]
 experiment = []
 lambda-airflow-dag-trigger = [