Add document type Supplement for Implementation and Testing Notes (#443)

* Initial attempt of a supplemental document regarding implementation and testing of scs-0100-v3

Signed-off-by: Matthias Büchse <[email protected]>

* Syntax error in front matter

Signed-off-by: Matthias Büchse <[email protected]>

* minor improvements

Signed-off-by: Matthias Büchse <[email protected]>

* Adapt ADR check script

Signed-off-by: Matthias Büchse <[email protected]>

* Add supplement to entropy standard

Signed-off-by: Matthias Büchse <[email protected]>

* Correction: mention that attributes have to be set

Signed-off-by: Matthias Büchse <[email protected]>

* Bugfix: subsection on manual tests was two levels too deep

Signed-off-by: Matthias Büchse <[email protected]>

* Minor improvements

Signed-off-by: Matthias Büchse <[email protected]>

* Relax testing notes to accommodate previous commit

Signed-off-by: Matthias Büchse <[email protected]>

* Update docstring of entropy-check.py

Signed-off-by: Matthias Büchse <[email protected]>

* State examples for flavor properties that are not immediately discoverable

Signed-off-by: Matthias Büchse <[email protected]>

* Extend scs-0001 in regard to Supplement type

Signed-off-by: Matthias Büchse <[email protected]>

* Update the section on Operation Tooling

Signed-off-by: Matthias Büchse <[email protected]>

* Use relative URIs

Signed-off-by: Matthias Büchse <[email protected]>

* Use absolute URIs, including host

Signed-off-by: Matthias Büchse <[email protected]>

* Update chk_adr to cope with templates and proposals better

Signed-off-by: Matthias Büchse <[email protected]>

* Replace scs- with SCS- as flavor prefix.

Signed-off-by: Kurt Garloff <[email protected]>

---------

Signed-off-by: Matthias Büchse <[email protected]>
Signed-off-by: Kurt Garloff <[email protected]>
Co-authored-by: Kurt Garloff <[email protected]>

Standards/scs-0001-v1-sovereign-cloud-standards.md

@@ -53,6 +53,12 @@ The SCS document format formally integrates
 the documentation of such decisions
 as documents of type `Decision Record`.
 
+#### Supplement
+
+A supplement extends a Standard with additional information, such as implementation and testing notes,
+that is merely informative, but not authoritative, and that may be subject to change more frequently
+than the standard itself.
+
 ### Document format
 
 The SCS documents are provided in GitHub flavored markdown.
@@ -82,6 +88,9 @@ The file name of an SCS document is formed using the following pattern:
 For a document with the number 190, with a major version number 2 and a slugified title `flavor-naming`,
 the resulting file name would be `scs-0190-v2-flavor-naming.md`.
 
+Supplements deviate from this pattern in that they employ a `w` instead of a `v` in front of the version
+number, and each supplement uses the same document number as the main document it is extending.
+
 The second digit in `XXXX` describes the track where the document belongs:
 
 | Track | Number |
@@ -97,9 +106,10 @@ embedded in the markdown header.
 
 | Field name      | Requirement                                                                | Description                                                                           |
 | --------------- | -------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- |
-| `type`          | REQUIRED                                                                   | one of `Procedural`, `Standard`, or `Decision Record`                                 |
+| `type`          | REQUIRED                                                                   | one of `Procedural`, `Standard`, `Decision Record`, or `Supplement`                   |
 | `status`        | REQUIRED                                                                   | one of `Proposal`, `Draft`, `Stable`, `Deprecated`, or `Rejected`                     |
 | `track`         | REQUIRED                                                                   | one of `Global`, `IaaS`, `KaaS`, `IAM`, `Ops`                                         |
+| `supplements`   | REQUIRED precisely when `type` is `Supplement`                             | list of documents that are extended by this document (e.g., multiple major versions)  |
 | `obsoleted_at`  | REQUIRED if `status` is `Deprecated`                                       | ISO formatted date indicating the date after which the deprecation is in effect       |
 | `stabilized_at` | REQUIRED if `status` was ever `Stable`                                     | ISO formatted date indicating the date after which the document was considered stable |
 | `rejected_at`   | REQUIRED if `status` is `Rejected`                                         | ISO formatted date indicating the date on which the document was rejected             |
@@ -150,6 +160,8 @@ where the group which has to form the consensus depends on the `track` of the do
 - IAM: The team working on identity and access management topics
 - Global: The entire SCS community
 
+Supplements may be kept in Draft state, because they are not authoritative.
+
 ### Proposal phase
 
 #### Proposal of a new document
@@ -161,14 +173,19 @@ against the [standards repository in the SovereignCloudStack organisation][scs-s
 The pull request MUST add exactly one SCS document,
 in the `Standards` folder.
 In the proposal phase,
-the document number MUST be replaced with `xxxx` in the file name.
+the document number MUST be replaced with `xxxx` in the file name,
+except for a Supplement, which uses the document number of the document it refers to.
 The major version MUST be 1.
 
 For a document with a slugified title `flavor-naming`,
-the file name would for instance be `scs-xxxx-v1-flavor-naming.md`.
+the file name would for instance be `scs-xxxx-v1-flavor-naming.md`;
+for a Supplement of `scs-0100-v3-flavor-naming.md`,
+the file name might be `scs-0100-w1-flavor-naming-implementation-testing.md` (note the `w1`!).
 
 The metadata MUST indicate the intended `track` and `type` of the document,
-and the `status` MUST be set to `Proposal`.
+and the `status` MUST be set to `Proposal`;
+for a Supplement, the `supplements` field MUST be set
+to a list of documents (usually containing one element).
 
 Upon acceptance by the group of people identified by the `track`,
 a number is assigned

Standards/scs-0100-v3-flavor-naming.md

@@ -259,27 +259,6 @@ offer the short variants for usability and easier portability, even beyond the m
 You must not extend the SCS naming scheme with your own extensions; you are encouraged however
 to suggest extensions that we can discuss and add to the official scheme.
 
-## Conformance Tests
-
-There is a script in [`flavor-name-check.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/flavor-naming/flavor-name-check.py)
-which can be used to decode, validate and construct flavor names.
-[`flavor-name-describe.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/flavor-naming/flavor-name-describe.py) outputs a human-readable decoding of the SCS flavor names.
-These scripts must stay in sync with the specification text.
-
-Ensure you have your OpenStack tooling (`python3-openstackclient`, `OS_CLOUD`) setup and call
-`tools/flavor-name-check.py -c $(openstack flavor list -f value -c Name)` to get a report
-on the flavor list compliance of the cloud environment.
-
-The script `flavor-names-openstack.py` talks to the OpenStack API of the
-cloud specified by the `OS_CLOUD` environment and queries properties and checks
-the names for standards compliance.
-It goes beyond the above example in checking that the discoverable
-features of flavors (vCPUs, RAM, Disk) match what the flavor names claim.
-This is used for SCS-compatible compliance testing.
-
-The functionality of the `flavor-name-check.py` script is also
-(partially) exposed via the web page <https://flavors.scs.community/>.
-
 ## Extensions
 
 Extensions provide a possibility for providers that offer a very differentiated set

Standards/scs-0100-w1-flavor-naming-implementation-testing.md

@@ -0,0 +1,73 @@
+---
+title: "SCS Flavor Naming Standard: Implementation and Testing Notes"
+type: Supplement
+track: IaaS
+status: Proposal
+supplements:
+  - scs-0100-v1-flavor-naming.md
+  - scs-0100-v2-flavor-naming.md
+  - scs-0100-v3-flavor-naming.md
+---
+
+## Introduction
+
+The three major versions of the standard that exist so far are very similar, and deliberately so.
+Therefore, the procedures needed to implement or test them are very similar as well. Yet, this document
+will only cover v3, because v1 and v2 are already obsolete by the time of writing.
+
+## Implementation Notes
+
+Every flavor whose name starts with `SCS-` must conform with the naming scheme laid down in the standard.
+
+### Operational Tooling
+
+#### Syntax Check
+
+The [test suite](https://github.com/SovereignCloudStack/standards/tree/main/Tests/iaas/flavor-naming)
+comes with a handy
+[command-line utility](https://github.com/SovereignCloudStack/standards/tree/main/Tests/iaas/flavor-naming/cli.py)
+that can be used to validate flavor names, to
+interactively construct a flavor name via a questionnaire, and to generate prose descriptions for given
+flavor names. See the
+[README](https://github.com/SovereignCloudStack/standards/tree/main/Tests/iaas/flavor-naming/README.md)
+for more details.
+
+The functionality of this script is also (partially) exposed via the web page
+<https://flavors.scs.community/>.
+
+With the OpenStack tooling (`python3-openstackclient`, `OS_CLOUD`) in place, you can call
+`cli.py -v parse v3 $(openstack flavor list -f value -c Name)` to get a report
+on the syntax compliance of the flavor names of the cloud environment.
+
+#### Flavor Creation
+
+The [OpenStack Flavor Manager](https://github.com/osism/openstack-flavor-manager) will create a whole set
+of flavors in one go, given a YAML description of this set.
+
+## Automated Tests
+
+### Errors
+
+The following items MUST be detected and reported as an error:
+
+- any syntax error in a name starting with `SCS-`,
+- any mismatch between any immediately discoverable property of a flavor (currently, CPU, RAM and disk size)
+  and the meaning of its name (which is usually a lower bound), such as the CPU generation or hypervisor.
+
+In addition, the following items MAY be reported as an error:
+
+- any mismatch between any non-immediately discoverable property of flavor and the meaning of its name.
+
+### Warnings
+
+None so far.
+
+### Implementation
+
+The script [`flavor-names-openstack.py`](https://github.com/SovereignCloudStack/standards/tree/main/Tests/iaas/flavor-naming/flavor-names-openstack.py)
+talks to the OpenStack API of the cloud specified by the `OS_CLOUD` environment and queries properties and
+checks the names for standards compliance.
+
+## Manual Tests
+
+To be determined.

Standards/scs-0101-v1-entropy.md

@@ -128,13 +128,5 @@ The user may choose to use the `virtio-rng` device via `rngd`.
 Compute nodes must use CPUs that offer instructions for accessing
 entropy (such as RDSEED or RDRAND on x86_64 or RNDR on arm64), and
 these instructions may not be filtered by the hypervisor.
-If this requirement cannot be verified directly, then at least the
-following two conditions must be satisfied in a virtual instance:
-
-1. The special file `/proc/sys/kernel/random/entropy_avail` must contain
-the value 256 (pinned since kernel 5.18).
-
-2. The number of FIPS 140-2 failures must not exceed 3 out of 1000 blocks
-tested, as determined by `cat /dev/random | rngtest -c 1000` .
 
 Compute nodes may provide a HRNG via `rngd`.

Standards/scs-0101-w1-entropy-implementation-testing.md

@@ -0,0 +1,64 @@
+---
+title: "SCS Entropy: Implementation and Testing Notes"
+type: Supplement
+track: IaaS
+status: Proposal
+supplements:
+  - scs-0101-v1-entropy.md
+---
+
+## Implementation
+
+We presume that almost nothing has to be done (or indeed can be done), as
+long as the CPUs and VM images are reasonably recent; only the flavor and
+image attributes have to be set:
+
+- flavor: `hw_rng:allowed=True` ,
+- image: `hw_rng_model: virtio` .
+
+## Automated Tests
+
+### Images Sample
+
+Some checks need to be performed on a live instance. For these checks, it is
+necessary to choose a sample of VM images to test on.
+
+For the time being, the sample MUST contain at least one public image reported
+by OpenStack. This may be extended in the future.
+
+### Errors
+
+For every image in the chosen sample, the following items MUST be detected and
+reported as an error:
+
+- the service `rngd` is not running,
+- the special file `/proc/sys/kernel/random/entropy_avail` does not contain
+  the value 256 (pinned since kernel 5.18),
+- the number of FIPS 140-2 failures exceeds 3 out of 1000 blocks
+  tested, as determined by `cat /dev/random | rngtest -c 1000` .
+
+Note: The latter two items act as surrogates for the following item, which
+cannot be detected directly:
+
+- CPU instructions for accessing entropy are not available to the VMs.
+
+### Warnings
+
+The following items MUST be detected and reported as a warning:
+
+- any flavor missing the attribute `hw_rng:allowed=True`,
+- any image missing the attribute `hw_rng_model: virtio`,
+
+Note that the requirement regarding the kernel patch level will not be
+checked, because of two reasons: (a) we already check the file `entropy_avail`
+(see subsection on Errors), and (b) users can always choose a recent image,
+as ensured by the image metadata standard.
+
+### Implementation
+
+The script [`entropy-check.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/entropy/entropy-check.py)
+connects to OpenStack and performs the checks described in this section.
+
+## Manual Tests
+
+None.

Tests/chk_adrs.py

@@ -69,6 +69,20 @@ def emit(self, s):
         print(f"ERROR: {s}", file=sys.stderr)
         self.errors += 1
 
+    def check_name(self, name):
+        if not name.startswith('scs-'):
+            return
+        components = name.split('-')
+        if len(components) < 4:
+            self.emit(f"document name must have at least four components separated by '-': {name}")
+            return
+        doc_no = components[1]
+        v_no = components[2][1:]
+        if len(doc_no) != 4 or not (doc_no.isnumeric() or doc_no.lower() == 'xxxx'):
+            self.emit(f"document code must have format NNNN, found {components[1]}")
+        if components[2][:1] not in ("v", "w") or not (v_no.isnumeric() or v_no.upper() == 'N'):
+            self.emit(f"document version must have format vN or wN, found: {components[2]}")
+
     def check_names(self, mds):
         """Check the list `mds` of md file names for name collisions"""
         # count the occurrences of the prefixes of length 12, e.g., scs-0001-v1-
@@ -79,10 +93,17 @@ def check_names(self, mds):
             self.emit(f"duplicates found: {', '.join(duplicates)}")
 
     def check_front_matter(self, fn, front):
-        """Check the dict `front` of front matter; `fn` is for context in error messages"""
+        """Check the dict `front` of front matter
+
+        The argument `fn` is mainly for context in error messages, but also to distinguish document types.
+        """
         if front is None:
             self.emit(f"in {fn}: is missing front matter altogether")
             return
+        # so far, only check primary documents, not supplemental ones
+        if fn[9] != 'v':
+            print(f"skipping non-primary {fn}", file=sys.stderr)
+            return
         # check each field in isolation
         errors = [
             key
@@ -103,6 +124,15 @@ def check_front_matter(self, fn, front):
             self.emit(f"in {fn}: status is Rejected, but rejected_at date is missing")
 
 
+def _load_front_matter(path):
+    with open(path, "rb") as flo:
+        loader = yaml.SafeLoader(flo)
+        try:
+            return loader.get_data()
+        finally:
+            loader.dispose()
+
+
 def main(argv):
     if len(argv) != 2:
         raise RuntimeError("must specify exactly one argument, PATH")
@@ -113,16 +143,10 @@ def main(argv):
         if fn.startswith("scs-") and fn.endswith(".md")
     ])
     checker = Checker()
-    checker.check_names(mds)
-    # now load each file and check front matter
     for fn in mds:
-        with open(os.path.join(path, fn), "rb") as flo:
-            loader = yaml.SafeLoader(flo)
-            try:
-                front = loader.get_data()
-            finally:
-                loader.dispose()
-            checker.check_front_matter(fn, front)
+        checker.check_name(fn)
+        checker.check_front_matter(fn, _load_front_matter(os.path.join(path, fn)))
+    checker.check_names(mds)
     return checker.errors
 
 

Tests/iaas/entropy/entropy-check.py

@@ -9,7 +9,7 @@
 restrictions); for further information, see the log messages on various channels:
     CRITICAL  for problems preventing the test to complete,
     ERROR     for violations of requirements,
-    INFO      for violations of recommendations,
+    WARNING   for violations of recommendations,
     DEBUG     for background information and problems that don't hinder the test.
 """
 from collections import Counter