Enable main check tool to run KaaS tests (#490)

Enable master check tool to run KaaS tests resolves #485 ---- Signed-off-by: Matthias Büchse <[email protected]>
SovereignCloudStack · Feb 29, 2024 · fa0de3c · fa0de3c
1 parent d34db90
commit fa0de3c
Show file tree

Hide file tree

Showing 5 changed files with 83 additions and 58 deletions.
diff --git a/.github/workflows/scs-compliance-check.yml b/.github/workflows/scs-compliance-check.yml
@@ -33,9 +33,7 @@ jobs:
                 password: ${{ secrets[inputs.secret_name] }}
           EOF
       - name: "Run scs-compliance-check"
-        run: "cd /scs-compliance && ./scs-compliance-check.py scs-compatible-${{ inputs.layer }}.yaml --version ${{ inputs.version }} -o result.yaml"
-        env:
-          OS_CLOUD: ${{ inputs.cloud }}
+        run: "cd /scs-compliance && ./scs-compliance-check.py scs-compatible-${{ inputs.layer }}.yaml --version ${{ inputs.version }} -o result.yaml -s ${{ inputs.cloud }} -a os_cloud=${{ inputs.cloud }}"
       - name: "Upload results"
         uses: actions/upload-artifact@v3
         with:

diff --git a/Standards/scs-0003-v1-sovereign-cloud-standards-yaml.md b/Standards/scs-0003-v1-sovereign-cloud-standards-yaml.md
@@ -89,19 +89,25 @@ The certification YAML _MUST_ contain the following keys:
 
 The certification YAML _MAY_ contain the following keys:
 
-### Prerequisite
+| Key                 | Type               | Description                                                   |
+| ------------------- | ------------------ | ------------------------------------------------------------- |
+| `prerequisite`      | Map                | Descriptor for the prerequisite certificate scope (see below) |
+| `variables`         | Array of String    | Lists variables that may occur in check tool descriptors      |
+
+The main check tool will expect an assignment for these variables (which is specific to the subject under test), and every occurrence of the variable in the check tool descriptor will be substituted accordingly.
+
+### Prerequisite descriptor
 
 A certificate within a certain level (above SCS-compatible) can only be granted if a valid corresponding certificate of the level below is presented,
 where corresponding means: of the same layer. The latter certificate is said to be a prerequisite for the former.
 
 We implement this logic by allowing for the designation of a certificate scope as a prerequisite;
 then a certificate of that prerequisite scope has to be presented before the certificate of the scope in question can be granted.
 
-| Key                 | Type   | Description                                                   | Example                                                                                                                           |
-| ------------------- | ------ | ------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- |
-| `prerequisite`      | Map    | Descriptor for the prerequisite certificate scope, if any     |                                                                                                                                   |
-| `prerequisite.name` | String | Full name of the certificate scope                            | _SCS Compatible IaaS_                                                                                                             |
-| `prerequisite.url`  | String | Valid URL to the latest raw version of the certificate scope  | _[scs-compatible-iaas.yaml](https://raw.githubusercontent.com/SovereignCloudStack/standards/main/Tests/scs-compatible-iaas.yaml)_ |
+| Key    | Type   | Description                                                   | Example                                                                                                                           |
+| ------ | ------ | ------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- |
+| `name` | String | Full name of the certificate scope                            | _SCS Compatible IaaS_                                                                                                             |
+| `url`  | String | Valid URL to the latest raw version of the certificate scope  | _[scs-compatible-iaas.yaml](https://raw.githubusercontent.com/SovereignCloudStack/standards/main/Tests/scs-compatible-iaas.yaml)_ |
 
 ### Version descriptor
 
@@ -138,10 +144,15 @@ Every list of standards consists of several standards that – altogether – de
 | Key               | Type   | Description                                                                                                                              | Example                |
 | ----------------- | ------ | ---------------------------------------------------------------------------------------------------------------------------------------- | ---------------------- |
 | `executable`      | String | Valid local filename (relative to the path of scs-compliance-check.py) of a script that verifies compliance with the particular standard | _image-md-check.py_    |
-| `args`            | String | _Optional_ list of arguments to be passed to the `check_tool`. Preferably none needed.                                                   | `-v`                   |
+| `env`             | Map    | _Optional_ key-value map of environment variables (values may use variables)                                                             | OS_CLOUD: {os_cloud}   |
+| `args`            | String | _Optional_ command-line arguments to be passed to the `check_tool` (may use variables)                                                   | `-v -k {kubeconfig}`   |
 | `condition`       | String | _Optionally_ overrides the per-standard condition (`mandatory` or `optional`)                                                            | _optional_             |
 | `classification`  | String | One of: `light` (_default_), `medium`, `heavy`; describes the resource usage of the script; used to select an appropiate test interval   | _heavy_                |
 
+As mentioned, variables may be used within `env` and `args`; they are enclosed in single braces, like so: `{var}`.
+If a brace is desired, it needs to be doubled: `{{` will be turned into `{`. When the main check tool is run,
+each occurrence of a variable will be substituted for according to the variable assignment for the subject under test.
+
 _Note_: the `executable` could in principle also be given via a URL; however, this is not yet supported due to security considerations.
 
 ### Basic Example
@@ -152,6 +163,8 @@ url: https://raw.githubusercontent.com/SovereignCloudStack/Docs/main/Certificati
 prerequisite:
   name: SCS Compatible IaaS
   url: https://raw.githubusercontent.com/SovereignCloudStack/standards/main/Tests/scs-compatible-iaas.yaml
+variables:
+  - os_cloud
 versions:
   - version: v5 # This version is in a draft state and work in progress
     # No stabilized_at: date set yet
@@ -161,12 +174,14 @@ versions:
         condition: mandatory # is default and can be left out
         check_tools:
           - executable: flavor-name-check.py
+            env:
+              OS_CLOUD: "{os_cloud}"
       - name: Image metadata
         url: https://raw.githubusercontent.com/SovereignCloudStack/Docs/main/Standards/SCS-0004-v1-image-metadata.md
         condition: mandatory
         check_tools:
           - executable: image-md-check.py
-            args: -v
+            args: -c {os_cloud} -v
           - executable: image-md-check2.py
             condition: optional
   - version: v4 # This is the upcoming version with a given target date. No further changes should be done to this set of standards
@@ -185,8 +200,6 @@ versions:
     obsoleted_at: 2021-11-01
     standards:
       - name: ....
-kaas:
-  - ...
 ```
 
 ## Process

diff --git a/Tests/scs-compatible-iaas.yaml b/Tests/scs-compatible-iaas.yaml
@@ -1,5 +1,7 @@
 name: SCS Compatible IaaS
 url: https://raw.githubusercontent.com/SovereignCloudStack/standards/main/Tests/scs-compatible-iaas.yaml
+variables:
+  - os_cloud
 versions:
   - version: v4
     stabilized_at: 2024-02-28
@@ -11,28 +13,28 @@ versions:
         url: https://raw.githubusercontent.com/SovereignCloudStack/standards/main/Standards/scs-0100-v3-flavor-naming.md
         check_tools:
           - executable: ./iaas/flavor-naming/flavor-names-openstack.py
-            args: "--mand=./iaas/scs-0100-v3-flavors.yaml"
+            args: -c {os_cloud} --mand=./iaas/scs-0100-v3-flavors.yaml
             # Note: --v2plus would outlaw the v1 flavor names. Don't do this yet.
       - name: Entropy
         url: https://raw.githubusercontent.com/SovereignCloudStack/standards/main/Standards/scs-0101-v1-entropy.md
         check_tools:
           - executable: ./iaas/entropy/entropy-check.py
-            args: -d
+            args: -c {os_cloud} -d
       - name: Image metadata
         url: https://raw.githubusercontent.com/SovereignCloudStack/standards/main/Standards/scs-0102-v1-image-metadata.md
         check_tools:
           - executable: ./iaas/image-metadata/image-md-check.py
-            args: -s -v
+            args: -c {os_cloud} -s -v
       - name: Standard flavors
         url: https://raw.githubusercontent.com/SovereignCloudStack/standards/main/Standards/scs-0103-v1-standard-flavors.md
         check_tools:
           - executable: ./iaas/standard-flavors/flavors-openstack.py
-            args: -d "./iaas/scs-0103-v1-flavors.yaml"
+            args: -c {os_cloud} -d ./iaas/scs-0103-v1-flavors.yaml
       - name: Standard images
         url: https://raw.githubusercontent.com/SovereignCloudStack/standards/main/Standards/scs-0104-v1-standard-images.md
         check_tools:
           - executable: ./iaas/standard-images/images-openstack.py
-            args: -d "./iaas/scs-0104-v1-images.yaml"
+            args: -c {os_cloud} -d ./iaas/scs-0104-v1-images.yaml
   - version: v3
     stabilized_at: 2023-06-15
     obsoleted_at: 2024-04-30
@@ -41,13 +43,13 @@ versions:
         url: https://raw.githubusercontent.com/SovereignCloudStack/standards/main/Standards/scs-0100-v3-flavor-naming.md
         check_tools:
           - executable: ./iaas/flavor-naming/flavor-names-openstack.py
-            args: "--v3"
+            args: --v3 -c {os_cloud}
             # Note: "--v3 --v2plus" would outlaw the v1 flavor names. Don't do this yet.
       - name: Image metadata
         url: https://raw.githubusercontent.com/SovereignCloudStack/standards/main/Standards/scs-0102-v1-image-metadata.md
         check_tools:
           - executable: ./iaas/image-metadata/image-md-check.py
-            args: -v
+            args: -v -c {os_cloud}
       - name: OpenStack Powered Compute v2022.11
         url: https://opendev.org/openinfra/interop/src/branch/master/guidelines/2022.11.json
   - version: v2
@@ -58,11 +60,12 @@ versions:
         url: https://raw.githubusercontent.com/SovereignCloudStack/standards/main/Standards/scs-0100-v2-flavor-naming.md
         check_tools:
           - executable: ./iaas/flavor-naming/flavor-names-openstack.py
+            args: -c {os_cloud}
       - name: Image metadata
         url: https://raw.githubusercontent.com/SovereignCloudStack/standards/main/Standards/scs-0102-v1-image-metadata.md
         check_tools:
           - executable: ./iaas/image-metadata/image-md-check.py
-            args: -v
+            args: -v -c {os_cloud}
       - name: OpenStack Powered Compute v2022.11
         url: https://opendev.org/openinfra/interop/src/branch/master/guidelines/2022.11.json
   - version: v1
@@ -73,11 +76,11 @@ versions:
         url: https://raw.githubusercontent.com/SovereignCloudStack/standards/main/Standards/scs-0100-v1-flavor-naming.md
         check_tools:
           - executable: ./iaas/flavor-naming/flavor-names-openstack.py
-            args: "--v1prefer"
+            args: -c {os_cloud} --v1prefer
       - name: Image metadata
         url: https://raw.githubusercontent.com/SovereignCloudStack/standards/main/Standards/scs-0102-v1-image-metadata.md
         check_tools:
           - executable: ./iaas/image-metadata/image-md-check.py
-            args: -v
+            args: -c {os_cloud} -v
       - name: OpenStack Powered Compute v2020.11
         url: https://opendev.org/openinfra/interop/src/branch/master/guidelines/2020.11.json