Adding tag or commit links (#98)

Updates for tagging/commit hash identification

.github/workflows/github-actions.yaml

@@ -5,7 +5,7 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: ["3.6", "3.10"]
+        python-version: ["3.7", "3.10"]
     steps:
       - run: echo "The job was automatically triggered by a ${{ github.event_name }} event."
       - run: echo "This job is now running on a ${{ runner.os }} server hosted by GitHub!"

.github/workflows/lint.yaml

@@ -7,7 +7,7 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: ["3.6", "3.10"]
+        python-version: ["3.7", "3.10"]
     steps:
     - uses: actions/checkout@v3
     - name: Set up Python ${{ matrix.python-version }}
@@ -21,4 +21,4 @@ jobs:
         if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
     - name: Lint with flake8
       run: |
-        flake8 . --count --ignore=E203,W503 --max-complexity=15 --max-line-length=88 --show-source --statistics
+        flake8 . --count --ignore=E203,W503 --max-complexity=15 --max-line-length=88 --show-source --statistics

CONTRIBUTING.md

@@ -78,15 +78,8 @@ this document, but it's a useful skill and there are [many](https://thoughtbot.c
 
 ## Identifying Tests
 
-All SR-IOV tests must be identified via a UUID representing a formal, globally unique identifier for said test. Should the test
-specification or reference implementation change, a new UUID is required. This provides traceability and an unambiguous way to
+All SR-IOV tests must be identified via either a tag or a commit hash, representing a globally unique identifier for said test. This provides traceability and an unambiguous way to
 reference specific tests. For this reason, the "common name" a test receives is not considered its formal reference, and may
-remain even if a UUID changes. For specifics on use of UUIDs, see the root level `README.md`.
+remain even if a test changes. Semantic versioning of tags should be observed to create relevant links to unique tests. For more information, see the root level `README.md`, [Releases](https://docs.github.com/en/repositories/releasing-projects-on-github/managing-releases-in-a-repository), and [Semantic Versioning](https://semver.org/).
 
-The recommended best practice is to use the CLI tool `uuidgen` to generate a UUID, as follows:
-```
-$ uuidgen
-3df99cbc-ec2f-406c-b344-39ff32b440f0
-```
-
-As this identifier is used to identify a specific, unique test definition, the UUID may need to be changed during a pull request. A change to the UUID is required whenever the functionality of a test case changes. This is inclusive of bug fixes, which may lead to differing behavior. A UUID may remain if the difference is purely documentation, and does not change the interpretation of the text. An example of this is a simple spelling or formatting change. Anything more consequential should be considered for a UUID change, implying the test case has changed. This should be a main consideration when reviewing and approving pull requests. Changes which either alter implied functionality or change the reference implementation, but do not receive an updated UUID, will not be acceptable.
+As this identifier is used to identify a specific, unique test definition, a change to the tag is required whenever the functionality/content of a test case changes, inclusive of bug fixes. Practically, this means that when changes are merged to main the tag and release should change. Maintaining proper versioning should be a main consideration of the repository maintainers after accepting a pull request and merging to main.

README.md

@@ -113,12 +113,13 @@ Each test case has its own folder. Under this folder there are two files: `test_
 
 In order for the HTML test report to be generated properly, the test case name line should start with "Test Case Name: ", or what is defined by `tests_name_field` in the config.yaml file. The script will try to match `tests_name_field` to locate the test case name.
 
-To satisfy the requirement of a unique identifier, the `README.md` of any SR-IOV test case must contain a distinct UUID. This will be used to formally reference a specific test specification. In order for the html test report to be generated properly, the UUID line should start with "UUID: ", or what is defined by the `tests_id_field` in the config.yaml file. Additionally, the reference implementation of a test should also be identified with a corresponding UUID, preferably as a comment on the first line. For example:
+To satisfy the requirement of a unique identifier, semantic versioning should be observed when tagging releases. This will be used to formally reference a specific test specification. Where tags are not applicable, or in a case where a specific case has not yet been tagged, one may use the specific commit hash to reference a test case. For example, the following links are both valid references to the same test case specification:
 ```
-# UUID: defcb0a3-1d73-45f9-9438-f19c6fba8a8c
-...
-```
-This will allow for clear identification of when a test specification and reference implementation may diverge. See `CONTRIBUTING.md` for more information on test case identification, and the generation of new UUIDs.
+https://github.com/redhat-partner-solutions/rhel-sriov-test/tree/v0.0.1/sriov/tests/SR_IOV_MultipleVFCreation_withMTU/README.md
+https://github.com/redhat-partner-solutions/rhel-sriov-test/tree/85e9c78a0ea1fcf978f79f9b8402e46b6078690f/sriov/tests/SR_IOV_MultipleVFCreation_withMTU/README.md
+``` 
+The benefits of tagging and releases are that they allow for easy reference to/distribution of a complete test suite. When a change causes a new release, the test suite will be referenced by a new semantic version number. If one desires to run tests on a more adhoc basis, use a subset of tests, or perhaps run different versions of tests from different releases then a list of tests can be created with specific links (utilizing releases, commit hashes, or a combination of either). In order for the html test report to be generated properly, the tests should be run from the git repo, which will get either the tag or, if not tagged, the specific commit hash for a link.
+This will allow for clear identification of when a test specification and reference implementation may diverge. See `CONTRIBUTING.md` for more information on test case identification.
 
 ## Common Code
 

sriov/tests/SR_IOV_BondVF/README.md

@@ -1,5 +1,4 @@
 ## Test Case Name: SR-IOV.BondVF
-# a1df2b99-97af-4290-b55a-2aa777ced821
 
 ### Objective(s): Test and ensure that VFs kernel bond across PFs works as expected.
 

sriov/tests/SR_IOV_BondVF/test_SR_IOV_BondVF.py

@@ -1,4 +1,3 @@
-# UUID: a1df2b99-97af-4290-b55a-2aa777ced821
 import pytest
 from sriov.common.utils import (
     create_vfs,

sriov/tests/SR_IOV_BondVF_DPDK/README.md

@@ -1,5 +1,4 @@
 ## Test Case Name: SR-IOV.BondVF.DPDK
-# ada901f7-fc3f-40dd-8c13-6c07f00badfe
 
 ### Objective(s): Test and ensure that VFs DPDK bond across PFs works as expected.
 

sriov/tests/SR_IOV_BondVF_DPDK/test_SR_IOV_BondVF_DPDK.py

@@ -1,4 +1,3 @@
-# UUID: ada901f7-fc3f-40dd-8c13-6c07f00badfe
 import pytest
 from time import sleep
 from sriov.common.utils import (

sriov/tests/SR_IOV_InterVF/README.md

@@ -1,6 +1,5 @@
 
 ## Test Case Name: SR-IOV.InterVF
-# 5d23e6c6-7954-48de-ab5a-d8da6b7b45c5
 
 ### Objective(s): Test and ensure that VFs (virtual function) on the same PF can communicate
 

sriov/tests/SR_IOV_InterVF/test_SR_IOV_InterVF.py

@@ -1,4 +1,3 @@
-# UUID: 5d23e6c6-7954-48de-ab5a-d8da6b7b45c5
 import pytest
 from sriov.common.utils import create_vfs, execute_and_assert
 

sriov/tests/SR_IOV_InterVF_DPDK/README.md

@@ -1,6 +1,5 @@
 
 ## Test Case Name: SR-IOV.InterVF.DPDK
-# da929907-c54a-485d-8cc9-1ece9348d69f
 
 ### Objective(s): Test and ensure that VFs (virtual function) bound to DPDK driver can communicate with VF on the same PF.
 

sriov/tests/SR_IOV_InterVF_DPDK/test_SR_IOV_InterVF_DPDK.py

@@ -1,4 +1,3 @@
-# UUID: da929907-c54a-485d-8cc9-1ece9348d69f
 import pytest
 import time
 from sriov.common.utils import (

sriov/tests/SR_IOV_MTU/README.md

@@ -1,6 +1,5 @@
 
 ## Test Case Name: SR-IOV.MTU
-# UUID: 98e9389d-fc1b-4ea3-b0be-cfa5529dfdbe
 
 ### Objective(s): Test and ensure that VF (virtual function) MTU functions as intended.
 

sriov/tests/SR_IOV_MTU/test_SR_IOV_MTU.py

@@ -1,4 +1,3 @@
-# UUID: 98e9389d-fc1b-4ea3-b0be-cfa5529dfdbe
 import re
 from sriov.common.utils import (
     create_vfs,

sriov/tests/SR_IOV_MultipleVFCreation_withMAC_DPDK/README.md

@@ -1,5 +1,4 @@
 # Test Case Name: SR-IOV.MultipleVFCreation.withMAC.DPDK
-# UUID: 935c6576-ca12-4592-8b5d-f5a76560be81
 
 ### Objective(s): A robustness test to ensure that, from a clean start, VFs (virtual functions) provision MAC address functions with DPDK
 

sriov/tests/SR_IOV_MultipleVFCreation_withMAC_DPDK/test_SR_IOV_MultipleVFCreation_withMAC_DPDK.py

@@ -1,4 +1,3 @@
-# UUID: 935c6576-ca12-4592-8b5d-f5a76560be81
 from sriov.common.utils import (
     execute_and_assert,
     create_vfs,

sriov/tests/SR_IOV_MultipleVFCreation_withMTU/README.md

@@ -1,5 +1,4 @@
 # Test Case Name: SR-IOV.MultipleVFCreation.withMTU
-# UUID: 67f7961c-1967-41fa-a2f1-dc1f73ef37d1
 
 ### Objective(s): A robustness test to ensure that, from a clean start, the maximum number of VFs (virtual functions) can be created and the maximum MTU (maximum transmission unit) configured for each VF
 

sriov/tests/SR_IOV_MultipleVFCreation_withMTU/test_SR_IOV_MultipleVFCreation_withMTU.py

@@ -1,4 +1,3 @@
-# UUID: 67f7961c-1967-41fa-a2f1-dc1f73ef37d1
 from sriov.common.utils import (
     execute_and_assert,
     create_vfs,

sriov/tests/SR_IOV_Permutation/README.md

@@ -1,6 +1,5 @@
 
 ## Test Case Name: SR-IOV.Permutation
-# UUID: dc4d0d06-1544-45f4-87c0-01015b11355a
 
 ### Objective(s): Test VFs (virtual function) configuration with different properties combined
 

sriov/tests/SR_IOV_Permutation/test_SR_IOV_Permutation.py

@@ -1,4 +1,3 @@
-# UUID: dc4d0d06-1544-45f4-87c0-01015b11355a
 import pytest
 from sriov.common.utils import (
     create_vfs,

sriov/tests/SR_IOV_Permutation_DPDK/README.md

@@ -1,6 +1,5 @@
 
 ## Test Case Name: SR-IOV.Permutation
-# UUID: 52d5e6ca-8a98-4151-9b2e-7a2946073325
 
 ### Objective(s): Test VFs (virtual function) bound to DPDK driver with different properties combined
 

sriov/tests/SR_IOV_Permutation_DPDK/test_SR_IOV_Permutation_DPDK.py

@@ -1,4 +1,3 @@
-# UUID: 52d5e6ca-8a98-4151-9b2e-7a2946073325
 import pytest
 from time import sleep
 from sriov.common.utils import (

sriov/tests/SR_IOV_QinQ/README.md

@@ -1,6 +1,5 @@
 
 ## Test Case Name: SR-IOV.QinQ
-# UUID: ddcb6e81-8433-4dfb-bea9-0351d20f8ac2
 
 ### Objective(s): Test and ensure that QinQ on VF works with kernel drive.
 

sriov/tests/SR_IOV_QinQ/test_SR_IOV_QinQ.py

@@ -1,4 +1,3 @@
-# UUID: ddcb6e81-8433-4dfb-bea9-0351d20f8ac2
 from sriov.common.utils import create_vfs, execute_and_assert, start_tmux, stop_tmux
 
 

sriov/tests/SR_IOV_Spoof_Mac/README.md

@@ -1,6 +1,5 @@
 
 ## Test Case Name: SR-IOV.Spoof.Mac
-# UUID: 66627809-09ff-4c41-a732-35b2f6f9528c
 
 ### Objective(s): Test and ensure that VF (virtual function) spoof check and custom mac can be set at the same time.
 

sriov/tests/SR_IOV_Spoof_Mac/test_SR_IOV_Spoof_Mac.py

@@ -1,4 +1,3 @@
-# UUID: 66627809-09ff-4c41-a732-35b2f6f9528c
 import pytest
 from sriov.common.utils import create_vfs, execute_and_assert, start_tmux, stop_tmux
 

sriov/tests/SR_IOV_TrustMode/README.md

@@ -1,6 +1,5 @@
 
 ## Test Case Name: SR-IOV.TrustMode
-# UUID: 28be87c8-0b05-4eac-8ce4-6e5ebe95efc8
 
 ### Objective(s): Test and ensure that VF (virtual function) trust mode functions as intended.
 

sriov/tests/SR_IOV_TrustMode/test_SR_IOV_TrustMode.py

@@ -1,4 +1,3 @@
-# UUID: 28be87c8-0b05-4eac-8ce4-6e5ebe95efc8
 from sriov.common.utils import create_vfs, execute_and_assert
 
 

sriov/tests/SR_IOV_macAddress/README.md

@@ -1,6 +1,5 @@
 
 ## Test Case Name: SR-IOV.macAddress
-# UUID: 93ccf4b8-21e6-4148-848b-0f9eefdb20d6
 
 ### Objective(s): Test and ensure that VF (virtual function) MAC address functions as intended.
 

sriov/tests/SR_IOV_macAddress/test_SR_IOV_macAddress.py

@@ -1,4 +1,3 @@
-# UUID: 93ccf4b8-21e6-4148-848b-0f9eefdb20d6
 from sriov.common.utils import (
     create_vfs,
     execute_and_assert,

sriov/tests/SR_IOV_macAddressDuplicate/README.md

@@ -1,6 +1,5 @@
 
 ## Test Case Name: SR-IOV.macAddressDuplicate
-# UUID: 67e90840-5f2f-4423-a80d-828e99d43e55
 
 ### Objective(s): Test and ensure that duplicate mac address across VFs on the same PF is permitted.
 

sriov/tests/SR_IOV_macAddressDuplicate/test_SR_IOV_macAddressDuplicate.py

@@ -1,4 +1,3 @@
-# UUID: 67e90840-5f2f-4423-a80d-828e99d43e55
 from sriov.common.utils import create_vfs, execute_and_assert, verify_vf_address
 
 

sriov/tests/SR_IOV_macAddress_DPDK/README.md

@@ -1,5 +1,4 @@
 ## Test Case Name: SR-IOV.macAddress.DPDK
-# UUID: d786bfca-219c-4b9f-ad9b-59fa1bc8cbba
 
 ### Objective(s): Test and ensure that VF (virtual function) MAC address functions as intended when bound to the DPDK driver.
 

sriov/tests/SR_IOV_macAddress_DPDK/test_SR_IOV_macAddress_DPDK.py

@@ -1,4 +1,3 @@
-# UUID: d786bfca-219c-4b9f-ad9b-59fa1bc8cbba
 from sriov.common.utils import (
     create_vfs,
     get_pci_address,

sriov/tests/conftest.py

@@ -1,3 +1,4 @@
+import git
 import os
 import pytest
 from pytest_html import extras
@@ -144,48 +145,69 @@ def pytest_html_report_title(report) -> None:
     report.title = "SR-IOV Test Report"
 
 
+def parse_file_for_field(file_path, field) -> str:
+    lines = []
+    field_str = ""
+    with open(file_path) as f:
+        lines = f.readlines()
+    for line in lines:
+        field_index = line.find(field)
+        if field_index != -1:
+            field_str = (line[field_index + len(field):]).strip()
+        if field_str:
+            break
+    return field_str
+
+
 @pytest.fixture(autouse=True)
 def _report_extras(extra, request, settings, monkeypatch) -> None:
-    lines = []
     monkeypatch.chdir(request.fspath.dirname)
 
     try:
         # This is assuming the current working directory contains the test
         # specification.
-        with open(settings.config["tests_doc_file"]) as f:
-            lines = f.readlines()
-
-        case_name = ""
-        case_id = ""
-        for line in lines:
-            case_index = line.find(settings.config["tests_name_field"])
-            id_index = line.find(settings.config["tests_id_field"])
-            if case_index != -1:
-                case_name = (
-                    line[case_index + len(settings.config["tests_name_field"]):]
-                ).strip()
-            if id_index != -1:
-                case_id = (
-                    line[id_index + len(settings.config["tests_id_field"]):]
-                ).strip()
-            if case_name and case_id:
-                break
+        case_name = parse_file_for_field(
+            settings.config["tests_doc_file"], settings.config["tests_name_field"]
+        )
 
         if case_name != "":
+            sha = ""
+            git_tag = ""
+            try:
+                repo = git.Repo(search_parent_directories=True)
+                sha = repo.head.commit
+                # This will assume that there is one tag per commit.
+                for tag in repo.tags:
+                    if tag.commit == sha:
+                        git_tag = tag
+                        break
+            except Exception:
+                pass
+
             test_dir = os.path.dirname(request.module.__file__).split(os.sep)[-1]
-            link = (
-                settings.config["github_tests_path"]
-                + "/"
-                + test_dir
-                + "/"
-                + settings.config["tests_doc_file"]
-            )
+            if git_tag:
+                link = (
+                    settings.config["github_tests_path"].replace("main", str(git_tag))
+                    + "/"
+                    + test_dir
+                    + "/"
+                    + settings.config["tests_doc_file"]
+                )
+            elif sha:
+                link = (
+                    settings.config["github_tests_path"].replace("main", sha.hexsha)
+                    + "/"
+                    + test_dir
+                    + "/"
+                    + settings.config["tests_doc_file"]
+                )
+            else:
+                case_name = "No tag or commit hash: No Link to"
+                link = "#"
+
             extra.append(
                 extras.html(
-                    '<p>Local Test Case ID: '
-                    + case_id
-                    + '</p>'
-                    + '<p>Link to the test specification: <a href="'
+                    '<p>Link to the test specification: <a href="'
                     + link
                     + '">'
                     + case_name