Merge pull request #430 from stackhpc/feat/fatimage-auto-upload

Add workflow for fat image uploads to client sites

Author: bertiethorpe

Diff for: .github/bin/get-s3-image.sh

@@ -0,0 +1,26 @@
+#!/bin/bash
+
+#####
+# This script looks for an image in OpenStack and if not found, downloads from
+# S3 bucket, and then uploads to OpenStack
+#####
+
+set -ex
+
+image_name=$1
+bucket_name=$2
+echo "Checking if image $image_name exists in OpenStack"
+image_exists=$(openstack image list --name "$image_name" -f value -c Name)
+
+if [ -n "$image_exists" ]; then
+    echo "Image $image_name already exists in OpenStack."
+else
+    echo "Image $image_name not found in OpenStack. Getting it from S3."
+
+    wget https://object.arcus.openstack.hpc.cam.ac.uk/swift/v1/AUTH_3a06571936a0424bb40bc5c672c4ccb1/$bucket_name/$image_name --progress=dot:giga
+
+    echo "Uploading image $image_name to OpenStack..."
+    openstack image create --file $image_name --disk-format qcow2 $image_name --progress
+
+    echo "Image $image_name has been uploaded to OpenStack."
+fi

Diff for: .github/workflows/upload-release-image.yml.sample

@@ -0,0 +1,66 @@
+# This workflow can be used to fetch images published by StackHPC and upload them to a client's OpenStack. 
+# The workflow takes two inputs:
+# - image name
+# - s3 bucket name
+# and first checks to see if the image exists in the target OpenStack. If the image doesn't exist, it is downloaded 
+# from StackHPC's public S3 bucket and then uploaded to the target OpenStack.
+#
+# To use this workflow in a downstream ansible-slurm-appliance repository simply copy it into .github/workflows
+# and give it an appropriate name, e.g.
+# cp .github/workflows/upload-s3-image.yml.sample .github/workflows/upload-s3-image.yml
+#
+# In order for the workflow to access the target OpenStack, an application credential clouds.yaml file must be 
+# added as a repository secret named OS_CLOUD_YAML.
+# Details on the contents of the clouds.yaml file can be found at https://docs.openstack.org/keystone/latest/user/application_credentials.html
+
+name: Upload release images to client sites from s3
+on:
+  workflow_dispatch:
+    inputs:
+      image_name:
+        type: string
+        description: Image name from: (https://object.arcus.openstack.hpc.cam.ac.uk/swift/v1/AUTH_3a06571936a0424bb40bc5c672c4ccb1/{BUCKET_NAME})
+        required: true
+      bucket_name:
+        type: choice
+        required: true
+        description: Bucket name
+        options:
+          - openhpc-images
+          # - openhpc-images-prerelease
+
+jobs:
+  image_upload:
+    runs-on: ubuntu-22.04
+    concurrency: ${{ github.ref }}
+    env: 
+      OS_CLOUD: openstack
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Write clouds.yaml
+        run: |
+          mkdir -p ~/.config/openstack/
+          echo "${{ secrets.OS_CLOUD_YAML }}" > ~/.config/openstack/clouds.yaml
+        shell: bash
+
+      - name: Upload latest image if missing
+        run: |
+          python3 -m venv venv
+          . venv/bin/activate
+          pip install -U pip
+          pip install $(grep -o 'python-openstackclient[><=0-9\.]*' requirements.txt)
+          bash .github/bin/get-s3-image.sh ${{ inputs.image_name }} ${{ inputs.bucket_name }}
+
+      - name: Cleanup OpenStack Image (on error or cancellation)
+        if: cancelled()
+        run: |
+          . venv/bin/activate
+          image_hanging=$(openstack image list --name ${{ inputs.image_name }} -f value -c ID -c Status | grep -v ' active$' | awk '{print $1}')
+          if [ -n "$image_hanging" ]; then
+            echo "Cleaning up OpenStack image with ID: $image_hanging"
+            openstack image delete $image_hanging
+          else
+            echo "No image ID found, skipping cleanup."
+          fi
+        shell: bash

Diff for: README.md

@@ -149,4 +149,6 @@ Please see the [monitoring-and-logging.README.md](docs/monitoring-and-logging.RE
 
 The `.github` directory contains a set of sample workflows which can be used by downstream site-specific configuration repositories to simplify ongoing maintainence tasks. These include:
 
-- An [upgrade check](.github/workflows/upgrade-check.yml.sample) workflow which automatically checks this upstream stackhpc/ansible-slurm-appliance repo for new releases and proposes a pull request to the downstream site-specific repo when a new release is published.
+- An [upgrade check](.github/workflows/upgrade-check.yml.sample) workflow which automatically checks this upstream stackhpc/ansible-slurm-appliance repo for new releases and proposes a pull request to the downstream site-specific repo when a new release is published.
+
+- An [image upload](.github/workflows/upload-s3-image.yml.sample) workflow which takes an image name, downloads it from StackHPC's public S3 bucket if available, and uploads it to the target OpenStack cloud.