Adding UUIDs: (#88)

CONTRIBUTING.md

@@ -75,3 +75,18 @@ this document, but it's a useful skill and there are [many](https://thoughtbot.c
 - Favor readability over brevity in both naming and structure
 - Document the _why_ with comments, and the _what_ with clear code
 - When in doubt, follow the [PEP 8](https://peps.python.org/pep-0008/) style guide
+
+## Identifying Tests
+
+All SR-IOV tests must be identified via a UUID representing a formal, globally unique identifier for said test. Should the test
+specification or reference implementation change, a new UUID is required. This provides traceability and an unambiguous way to
+reference specific tests. For this reason, the "common name" a test receives is not considered its formal reference, and may
+remain even if a UUID changes. For specifics on use of UUIDs, see the root level `README.md`.
+
+The recommended best practice is to use the CLI tool `uuidgen` to generate a UUID, as follows:
+```
+$ uuidgen
+3df99cbc-ec2f-406c-b344-39ff32b440f0
+```
+
+As this identifier is used to identify a specific, unique test definition, the UUID may need to be changed during a pull request. A change to the UUID is required whenever the functionality of a test case changes. This is inclusive of bug fixes, which may lead to differing behavior. A UUID may remain if the difference is purely documentation, and does not change the interpretation of the text. An example of this is a simple spelling or formatting change. Anything more consequential should be considered for a UUID change, implying the test case has changed. This should be a main consideration when reviewing and approving pull requests. Changes which either alter implied functionality or change the reference implementation, but do not receive an updated UUID, will not be acceptable.

README.md

@@ -46,10 +46,10 @@ dut:
   pmd_cpus: "30,32,34"      # cpu list used for the testpmd
   interface:
     pf1:
-      name: "ens7f3"        # first PF interface name 
+      name: "ens7f3"        # first PF interface name
       pci: "0000:ca:00.3"   # first PF PCI address
     vf1:
-      name: "ens7f3v0"      # first VF interface name 
+      name: "ens7f3v0"      # first VF interface name
       pci: "0000:ca:19.0"   # first VF PCI address
 trafficgen:
   host:                     # TrafficGen ip address
@@ -111,11 +111,18 @@ After the test is completed, report.html will be generated under the current wor
 
 Each test case has its own folder. Under this folder there are two files: `test_<testcase>.py` and `README.md`. The `README.md` under each test case folder contains the test case description. `test_<testcase>.py` is an reference implementation of this test case.
 
-In order for the html test report to be generated properly, the test case name line should start with "Test Case Name: ", or what is defined by `tests_name_field` in the config.yaml file. The script will try to match `tests_name_field` to locate the test case name.
+In order for the HTML test report to be generated properly, the test case name line should start with "Test Case Name: ", or what is defined by `tests_name_field` in the config.yaml file. The script will try to match `tests_name_field` to locate the test case name.
+
+To satisfy the requirement of a unique identifier, the `README.md` of any SR-IOV test case must contain a distinct UUID. This will be used to formally reference a specific test specification. In order for the html test report to be generated properly, the UUID line should start with "UUID: ", or what is defined by the `tests_id_field` in the config.yaml file. Additionally, the reference implementation of a test should also be identified with a corresponding UUID, preferably as a comment on the first line. For example:
+```
+# UUID: defcb0a3-1d73-45f9-9438-f19c6fba8a8c
+...
+```
+This will allow for clear identification of when a test specification and reference implementation may diverge. See `CONTRIBUTING.md` for more information on test case identification, and the generation of new UUIDs.
 
 ## Common Code
 
-The common code shared by all test cases is under the `sriov/common`. 
+The common code shared by all test cases is under the `sriov/common`.
 
 The common code has its own test cases. The majority of the common code test cases are under the `tests/common/` folder. Pytest is used to execute these test cases. Because a valid `config.yaml` file is expected by pytest to establish ssh connections and execute these test cases, they are considered an e2e test.
 

sriov/tests/SR_IOV_BondVF/README.md

@@ -1,4 +1,5 @@
 ## Test Case Name: SR-IOV.BondVF
+# a1df2b99-97af-4290-b55a-2aa777ced821
 
 ### Objective(s): Test and ensure that VFs kernel bond across PFs works as expected.
 
@@ -59,4 +60,3 @@ ip link set ${PF_1} vf 0 state disable
 * On trafficgen, delete the ping tmux session
 
 * On DUT, in the testpmd tmux window, enter “quit\n”, then delete testpmd tmux session, and reset the VFs
-

sriov/tests/SR_IOV_BondVF/test_SR_IOV_BondVF.py

@@ -1,3 +1,4 @@
+# UUID: a1df2b99-97af-4290-b55a-2aa777ced821
 import pytest
 from sriov.common.utils import (
     create_vfs,

sriov/tests/SR_IOV_BondVF_DPDK/README.md

@@ -1,4 +1,5 @@
 ## Test Case Name: SR-IOV.BondVF.DPDK
+# ada901f7-fc3f-40dd-8c13-6c07f00badfe
 
 ### Objective(s): Test and ensure that VFs DPDK bond across PFs works as expected.
 
@@ -54,4 +55,3 @@ ip link set ${PF_1} vf 0 state disable
 * On trafficgen, delete the ping tmux session
 
 * On DUT, in the testpmd tmux window, enter “quit\n”, then delete testpmd tmux session, and reset the VFs
-

sriov/tests/SR_IOV_BondVF_DPDK/test_SR_IOV_BondVF_DPDK.py

@@ -1,3 +1,4 @@
+# UUID: ada901f7-fc3f-40dd-8c13-6c07f00badfe
 import pytest
 from time import sleep
 from sriov.common.utils import (

sriov/tests/SR_IOV_InterVF/README.md

@@ -1,5 +1,6 @@
 
 ## Test Case Name: SR-IOV.InterVF
+# 5d23e6c6-7954-48de-ab5a-d8da6b7b45c5
 
 ### Objective(s): Test and ensure that VFs (virtual function) on the same PF can communicate
 
@@ -34,7 +35,7 @@ ip netns exec ns0 arp -s ${DUT_IP_1} ${DUT_VF1_MAC}
 ip netns exec ns1 arp -s ${DUT_IP_0} ${DUT_VF0_MAC}
 ```
 
-* From the ns0 ping ns1 address. Assert exit code of ping equal to 0, 
+* From the ns0 ping ns1 address. Assert exit code of ping equal to 0,
 ```
 ip netns exec ns0 ping ${DUT_IP_1}/24
 ```
@@ -47,6 +48,3 @@ ip netns exec ns0 ping ${DUT_IP_1}/24
 ```
 echo 0 > /sys/class/net/${DUT_PF}/device/sriov_numvfs
 ```
-
-
-

sriov/tests/SR_IOV_InterVF/test_SR_IOV_InterVF.py

@@ -1,3 +1,4 @@
+# UUID: 5d23e6c6-7954-48de-ab5a-d8da6b7b45c5
 import pytest
 from sriov.common.utils import create_vfs, execute_and_assert
 

sriov/tests/SR_IOV_InterVF_DPDK/README.md

@@ -1,5 +1,6 @@
 
 ## Test Case Name: SR-IOV.InterVF.DPDK
+# da929907-c54a-485d-8cc9-1ece9348d69f
 
 ### Objective(s): Test and ensure that VFs (virtual function) bound to DPDK driver can communicate with VF on the same PF.
 
@@ -50,6 +51,3 @@ testpmd> quit
 ```
 echo 0 > /sys/class/net/${DUT_PF}/device/sriov_numvfs
 ```
-
-
-

sriov/tests/SR_IOV_InterVF_DPDK/test_SR_IOV_InterVF_DPDK.py

@@ -1,3 +1,4 @@
+# UUID: da929907-c54a-485d-8cc9-1ece9348d69f
 import pytest
 import time
 from sriov.common.utils import (

sriov/tests/SR_IOV_MTU/README.md

@@ -1,5 +1,6 @@
 
 ## Test Case Name: SR-IOV.MTU
+# UUID: 98e9389d-fc1b-4ea3-b0be-cfa5529dfdbe
 
 ### Objective(s): Test and ensure that VF (virtual function) MTU functions as intended.
 
@@ -36,6 +37,3 @@ ping -W 1 -c -s $((MTU-28)) -M do ${TGEN_IP}
 ```
 echo 0 > /sys/class/net/${DUT_PF}/device/sriov_numvfs
 ```
-
-
-

sriov/tests/SR_IOV_MTU/test_SR_IOV_MTU.py

@@ -1,3 +1,4 @@
+# UUID: 98e9389d-fc1b-4ea3-b0be-cfa5529dfdbe
 import re
 from sriov.common.utils import (
     create_vfs,

sriov/tests/SR_IOV_MultipleVFCreation_withMAC_DPDK/README.md

@@ -1,4 +1,5 @@
 # Test Case Name: SR-IOV.MultipleVFCreation.withMAC.DPDK
+# UUID: 935c6576-ca12-4592-8b5d-f5a76560be81
 
 ### Objective(s): A robustness test to ensure that, from a clean start, VFs (virtual functions) provision MAC address functions with DPDK
 

sriov/tests/SR_IOV_MultipleVFCreation_withMAC_DPDK/test_SR_IOV_MultipleVFCreation_withMAC_DPDK.py

@@ -1,3 +1,4 @@
+# UUID: 935c6576-ca12-4592-8b5d-f5a76560be81
 from sriov.common.utils import (
     execute_and_assert,
     create_vfs,

sriov/tests/SR_IOV_MultipleVFCreation_withMTU/README.md

@@ -1,4 +1,5 @@
 # Test Case Name: SR-IOV.MultipleVFCreation.withMTU
+# UUID: 67f7961c-1967-41fa-a2f1-dc1f73ef37d1
 
 ### Objective(s): A robustness test to ensure that, from a clean start, the maximum number of VFs (virtual functions) can be created and the maximum MTU (maximum transmission unit) configured for each VF
 
@@ -16,7 +17,7 @@ echo 0 > /sys/class/net/$PF/device/sriov_numvfs
 echo $(cat /sys/class/net/$PF/device/sriov_totalvfs) > /sys/class/net/$PF/device/sriov_numvfs
 ```
 
-* Checkpoint: 
+* Checkpoint:
  - Check the number of VFs created under ```$PF``` in ```sriov_numvfs```
  - Check that none of the VFs are created with all zero MAC addresses
  - Assert the number of VF interfaces equals ```sriov_totalvfs```

sriov/tests/SR_IOV_MultipleVFCreation_withMTU/test_SR_IOV_MultipleVFCreation_withMTU.py

@@ -1,3 +1,4 @@
+# UUID: 67f7961c-1967-41fa-a2f1-dc1f73ef37d1
 from sriov.common.utils import (
     execute_and_assert,
     create_vfs,

sriov/tests/SR_IOV_Permutation/README.md

@@ -1,13 +1,14 @@
 
 ## Test Case Name: SR-IOV.Permutation
+# UUID: dc4d0d06-1544-45f4-87c0-01015b11355a
 
 ### Objective(s): Test VFs (virtual function) configuration with different properties combined
 
 ### Preconfig:  If a switch is used between the DUT and the trafficgen, the DUT facing switchport should be configured as dot1q trunk to allowed vlan tag from the DUT
 
 ### Test procedure
 
-* On DUT, create 1 VF; set VF 0 with a permutation of the following permutations, 
+* On DUT, create 1 VF; set VF 0 with a permutation of the following permutations,
 
 | Spoofchk (on/off) | Trust(on/off) | vlan(with/without) | QoS (with/without) | max_tx_rate (with/without) |
 | --- | --- | --- | --- | --- |
@@ -45,6 +46,3 @@ ping -W 1 -c 1 ${DUT_IP}
 ```
 echo 0 > /sys/class/net/${DUT_PF}/device/sriov_numvfs
 ```
-
-
-

sriov/tests/SR_IOV_Permutation/test_SR_IOV_Permutation.py

@@ -1,3 +1,4 @@
+# UUID: dc4d0d06-1544-45f4-87c0-01015b11355a
 import pytest
 from sriov.common.utils import (
     create_vfs,

sriov/tests/SR_IOV_Permutation_DPDK/README.md

@@ -1,11 +1,12 @@
 
 ## Test Case Name: SR-IOV.Permutation
+# UUID: 52d5e6ca-8a98-4151-9b2e-7a2946073325
 
 ### Objective(s): Test VFs (virtual function) bound to DPDK driver with different properties combined
 
 ### Test procedure
 
-* On DUT, create 1 VF; set VF 0 with a permutation of the following permutations, 
+* On DUT, create 1 VF; set VF 0 with a permutation of the following permutations,
 
 | Spoofchk (on/off) | Trust(on/off) | vlan(with/without) | QoS (with/without) | max_tx_rate (with/without) |
 | --- | --- | --- | --- | --- |
@@ -60,6 +61,3 @@ testpmd> quit
 ```
 echo 0 > /sys/class/net/${DUT_PF}/device/sriov_numvfs
 ```
-
-
-

sriov/tests/SR_IOV_Permutation_DPDK/test_SR_IOV_Permutation_DPDK.py

@@ -1,3 +1,4 @@
+# UUID: 52d5e6ca-8a98-4151-9b2e-7a2946073325
 import pytest
 from time import sleep
 from sriov.common.utils import (

sriov/tests/SR_IOV_QinQ/README.md

@@ -1,5 +1,6 @@
 
 ## Test Case Name: SR-IOV.QinQ
+# UUID: ddcb6e81-8433-4dfb-bea9-0351d20f8ac2
 
 ### Objective(s): Test and ensure that QinQ on VF works with kernel drive.
 
@@ -28,6 +29,3 @@ timeout 3 tcpdump -i ${trafficgen_pf} -c 1 vlan ${OUTSIDE_VLAN} and vlan ${INSID
 ```
 echo 0 > /sys/class/net/${DUT_PF}/device/sriov_numvfs
 ```
-
-
-

sriov/tests/SR_IOV_QinQ/test_SR_IOV_QinQ.py

@@ -1,3 +1,4 @@
+# UUID: ddcb6e81-8433-4dfb-bea9-0351d20f8ac2
 from sriov.common.utils import create_vfs, execute_and_assert, start_tmux, stop_tmux
 
 

sriov/tests/SR_IOV_Spoof_Mac/README.md

@@ -1,5 +1,6 @@
 
 ## Test Case Name: SR-IOV.Spoof.Mac
+# UUID: 66627809-09ff-4c41-a732-35b2f6f9528c
 
 ### Objective(s): Test and ensure that VF (virtual function) spoof check and custom mac can be set at the same time.
 
@@ -38,6 +39,3 @@ timeout 2 tcpdump -i ${trafficgen_PF} -c 1 ether host ${SPOOF_MAC}
 ```
 echo 0 > /sys/class/net/${DUT_PF}/device/sriov_numvfs
 ```
-
-
-

sriov/tests/SR_IOV_Spoof_Mac/test_SR_IOV_Spoof_Mac.py

@@ -1,3 +1,4 @@
+# UUID: 66627809-09ff-4c41-a732-35b2f6f9528c
 import pytest
 from sriov.common.utils import create_vfs, execute_and_assert, start_tmux, stop_tmux
 

sriov/tests/SR_IOV_TrustMode/README.md

@@ -1,5 +1,6 @@
 
 ## Test Case Name: SR-IOV.TrustMode
+# UUID: 28be87c8-0b05-4eac-8ce4-6e5ebe95efc8
 
 ### Objective(s): Test and ensure that VF (virtual function) trust mode functions as intended.
 
@@ -25,7 +26,7 @@ ip link set ${DUT_PF} vf 0 trust off
 ```
 
 * The next step is expected to produce an error,
-  
+
 ```
 ip link set ${DUT_PF}v0 address ${MAC_3}
 RTNETLINK answers: Permission denied
@@ -37,6 +38,3 @@ RTNETLINK answers: Permission denied
 ```
 echo 0 > /sys/class/net/${DUT_PF}/device/sriov_numvfs
 ```
-
-
-

sriov/tests/SR_IOV_TrustMode/test_SR_IOV_TrustMode.py

@@ -1,3 +1,4 @@
+# UUID: 28be87c8-0b05-4eac-8ce4-6e5ebe95efc8
 from sriov.common.utils import create_vfs, execute_and_assert
 
 

sriov/tests/SR_IOV_macAddress/README.md

@@ -1,5 +1,6 @@
 
 ## Test Case Name: SR-IOV.macAddress
+# UUID: 93ccf4b8-21e6-4148-848b-0f9eefdb20d6
 
 ### Objective(s): Test and ensure that VF (virtual function) MAC address functions as intended.
 
@@ -36,6 +37,3 @@ ip addr del ${TGEN_IP}/24 dev ${TGEN_PF}
 ```
 echo 0 > /sys/class/net/${DUT_PF}/device/sriov_numvfs
 ```
-
-
-

sriov/tests/SR_IOV_macAddress/test_SR_IOV_macAddress.py

@@ -1,3 +1,4 @@
+# UUID: 93ccf4b8-21e6-4148-848b-0f9eefdb20d6
 from sriov.common.utils import (
     create_vfs,
     execute_and_assert,

sriov/tests/SR_IOV_macAddressDuplicate/README.md

@@ -1,5 +1,6 @@
 
 ## Test Case Name: SR-IOV.macAddressDuplicate
+# UUID: 67e90840-5f2f-4423-a80d-828e99d43e55
 
 ### Objective(s): Test and ensure that duplicate mac address across VFs on the same PF is permitted.
 
@@ -33,6 +34,3 @@ ip link set ${DUT_PF} vf 0 mac ${MAC_1}
 ```
 echo 0 > /sys/class/net/${DUT_PF}/device/sriov_numvfs
 ```
-
-
-

sriov/tests/SR_IOV_macAddressDuplicate/test_SR_IOV_macAddressDuplicate.py

@@ -1,3 +1,4 @@
+# UUID: 67e90840-5f2f-4423-a80d-828e99d43e55
 from sriov.common.utils import create_vfs, execute_and_assert, verify_vf_address
 
 

sriov/tests/SR_IOV_macAddress_DPDK/README.md

@@ -1,4 +1,5 @@
 ## Test Case Name: SR-IOV.macAddress.DPDK
+# UUID: d786bfca-219c-4b9f-ad9b-59fa1bc8cbba
 
 ### Objective(s): Test and ensure that VF (virtual function) MAC address functions as intended when bound to the DPDK driver.
 

sriov/tests/SR_IOV_macAddress_DPDK/test_SR_IOV_macAddress_DPDK.py

@@ -1,3 +1,4 @@
+# UUID: d786bfca-219c-4b9f-ad9b-59fa1bc8cbba
 from sriov.common.utils import (
     create_vfs,
     get_pci_address,

sriov/tests/config_template.yaml

@@ -1,5 +1,5 @@
 dpdk_img: "docker.io/patrickkutch/dpdk:v21.11"
 tests_name_field: "Test Case Name:"
+tests_id_field: "UUID:"
 tests_doc_file: "README.md"
 github_tests_path: "https://github.com/redhat-partner-solutions/intel-sriov-test/tree/main/sriov/tests"
-

sriov/tests/conftest.py

@@ -156,12 +156,19 @@ def _report_extras(extra, request, settings, monkeypatch) -> None:
             lines = f.readlines()
 
         case_name = ""
+        case_id = ""
         for line in lines:
             case_index = line.find(settings.config["tests_name_field"])
+            id_index = line.find(settings.config["tests_id_field"])
             if case_index != -1:
                 case_name = (
                     line[case_index + len(settings.config["tests_name_field"]):]
                 ).strip()
+            if id_index != -1:
+                case_id = (
+                    line[id_index + len(settings.config["tests_id_field"]):]
+                ).strip()
+            if case_name and case_id:
                 break
 
         if case_name != "":
@@ -175,7 +182,10 @@ def _report_extras(extra, request, settings, monkeypatch) -> None:
             )
             extra.append(
                 extras.html(
-                    '<p>Link to the test specification: <a href="'
+                    '<p>Local Test Case ID: '
+                    + case_id
+                    + '</p>'
+                    + '<p>Link to the test specification: <a href="'
                     + link
                     + '">'
                     + case_name