Skip to content

Commit

Permalink
Updates for tagging/commit hash identification:
Browse files Browse the repository at this point in the history
Removing reliance on local UUID from conftest, defaulting to no link when missing tag and hash (tag is still preferred to commit hash)
Updating CONTRIBUTING and README to reflect tag instead of UUID for versioning
Updating all tests to remove UUID fields from spec and implementation
  • Loading branch information
dkosteck committed Dec 16, 2022
1 parent d539c4c commit 47165b6
Show file tree
Hide file tree
Showing 33 changed files with 22 additions and 62 deletions.
13 changes: 3 additions & 10 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,15 +78,8 @@ this document, but it's a useful skill and there are [many](https://thoughtbot.c

## Identifying Tests

All SR-IOV tests must be identified via a UUID representing a formal, globally unique identifier for said test. Should the test
specification or reference implementation change, a new UUID is required. This provides traceability and an unambiguous way to
All SR-IOV tests must be identified via either a tag or a commit hash, representing a globally unique identifier for said test. This provides traceability and an unambiguous way to
reference specific tests. For this reason, the "common name" a test receives is not considered its formal reference, and may
remain even if a UUID changes. For specifics on use of UUIDs, see the root level `README.md`.
remain even if a test changes. Semantic versioning of tags should be observed to create relevant links to unique tests. For more information, see the root level `README.md`.

The recommended best practice is to use the CLI tool `uuidgen` to generate a UUID, as follows:
```
$ uuidgen
3df99cbc-ec2f-406c-b344-39ff32b440f0
```

As this identifier is used to identify a specific, unique test definition, the UUID may need to be changed during a pull request. A change to the UUID is required whenever the functionality of a test case changes. This is inclusive of bug fixes, which may lead to differing behavior. A UUID may remain if the difference is purely documentation, and does not change the interpretation of the text. An example of this is a simple spelling or formatting change. Anything more consequential should be considered for a UUID change, implying the test case has changed. This should be a main consideration when reviewing and approving pull requests. Changes which either alter implied functionality or change the reference implementation, but do not receive an updated UUID, will not be acceptable.
As this identifier is used to identify a specific, unique test definition, a change to the tag is required whenever the functionality of a test case changes. This is inclusive of bug fixes, which may lead to differing behavior. A tag may remain if the difference is purely documentation, and does not change the interpretation of the text. An example of this is a simple spelling or formatting change. Anything more consequential should be considered for a tag change, implying the test case version has changed. This should be a main consideration when reviewing and approving pull requests. Changes which either alter implied functionality or change the reference implementation, but do not receive an updated tag, will not be acceptable.
8 changes: 2 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -113,12 +113,8 @@ Each test case has its own folder. Under this folder there are two files: `test_

In order for the HTML test report to be generated properly, the test case name line should start with "Test Case Name: ", or what is defined by `tests_name_field` in the config.yaml file. The script will try to match `tests_name_field` to locate the test case name.

To satisfy the requirement of a unique identifier, the `README.md` of any SR-IOV test case must contain a distinct UUID. This will be used to formally reference a specific test specification. In order for the html test report to be generated properly, the UUID line should start with "UUID: ", or what is defined by the `tests_id_field` in the config.yaml file. Additionally, the reference implementation of a test should also be identified with a corresponding UUID, preferably as a comment on the first line. For example:
```
# UUID: defcb0a3-1d73-45f9-9438-f19c6fba8a8c
...
```
This will allow for clear identification of when a test specification and reference implementation may diverge. See `CONTRIBUTING.md` for more information on test case identification, and the generation of new UUIDs.
To satisfy the requirement of a unique identifier, semantic versioning should be observed when tagging releases. This will be used to formally reference a specific test specification. In order for the html test report to be generated properly, the tests should be run from the git repo, which will get either the tag or, if not tagged, the specific commit hash for a link.
This will allow for clear identification of when a test specification and reference implementation may diverge. See `CONTRIBUTING.md` for more information on test case identification.

## Common Code

Expand Down
1 change: 0 additions & 1 deletion sriov/tests/SR_IOV_BondVF/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
## Test Case Name: SR-IOV.BondVF
# a1df2b99-97af-4290-b55a-2aa777ced821

### Objective(s): Test and ensure that VFs kernel bond across PFs works as expected.

Expand Down
1 change: 0 additions & 1 deletion sriov/tests/SR_IOV_BondVF/test_SR_IOV_BondVF.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
# UUID: a1df2b99-97af-4290-b55a-2aa777ced821
import pytest
from sriov.common.utils import (
create_vfs,
Expand Down
1 change: 0 additions & 1 deletion sriov/tests/SR_IOV_BondVF_DPDK/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
## Test Case Name: SR-IOV.BondVF.DPDK
# ada901f7-fc3f-40dd-8c13-6c07f00badfe

### Objective(s): Test and ensure that VFs DPDK bond across PFs works as expected.

Expand Down
1 change: 0 additions & 1 deletion sriov/tests/SR_IOV_BondVF_DPDK/test_SR_IOV_BondVF_DPDK.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
# UUID: ada901f7-fc3f-40dd-8c13-6c07f00badfe
import pytest
from time import sleep
from sriov.common.utils import (
Expand Down
1 change: 0 additions & 1 deletion sriov/tests/SR_IOV_InterVF/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@

## Test Case Name: SR-IOV.InterVF
# 5d23e6c6-7954-48de-ab5a-d8da6b7b45c5

### Objective(s): Test and ensure that VFs (virtual function) on the same PF can communicate

Expand Down
1 change: 0 additions & 1 deletion sriov/tests/SR_IOV_InterVF/test_SR_IOV_InterVF.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
# UUID: 5d23e6c6-7954-48de-ab5a-d8da6b7b45c5
import pytest
from sriov.common.utils import create_vfs, execute_and_assert

Expand Down
1 change: 0 additions & 1 deletion sriov/tests/SR_IOV_InterVF_DPDK/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@

## Test Case Name: SR-IOV.InterVF.DPDK
# da929907-c54a-485d-8cc9-1ece9348d69f

### Objective(s): Test and ensure that VFs (virtual function) bound to DPDK driver can communicate with VF on the same PF.

Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
# UUID: da929907-c54a-485d-8cc9-1ece9348d69f
import pytest
import time
from sriov.common.utils import (
Expand Down
1 change: 0 additions & 1 deletion sriov/tests/SR_IOV_MTU/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@

## Test Case Name: SR-IOV.MTU
# UUID: 98e9389d-fc1b-4ea3-b0be-cfa5529dfdbe

### Objective(s): Test and ensure that VF (virtual function) MTU functions as intended.

Expand Down
1 change: 0 additions & 1 deletion sriov/tests/SR_IOV_MTU/test_SR_IOV_MTU.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
# UUID: 98e9389d-fc1b-4ea3-b0be-cfa5529dfdbe
import re
from sriov.common.utils import (
create_vfs,
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
# Test Case Name: SR-IOV.MultipleVFCreation.withMAC.DPDK
# UUID: 935c6576-ca12-4592-8b5d-f5a76560be81

### Objective(s): A robustness test to ensure that, from a clean start, VFs (virtual functions) provision MAC address functions with DPDK

Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
# UUID: 935c6576-ca12-4592-8b5d-f5a76560be81
from sriov.common.utils import (
execute_and_assert,
create_vfs,
Expand Down
1 change: 0 additions & 1 deletion sriov/tests/SR_IOV_MultipleVFCreation_withMTU/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
# Test Case Name: SR-IOV.MultipleVFCreation.withMTU
# UUID: 67f7961c-1967-41fa-a2f1-dc1f73ef37d1

### Objective(s): A robustness test to ensure that, from a clean start, the maximum number of VFs (virtual functions) can be created and the maximum MTU (maximum transmission unit) configured for each VF

Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
# UUID: 67f7961c-1967-41fa-a2f1-dc1f73ef37d1
from sriov.common.utils import (
execute_and_assert,
create_vfs,
Expand Down
1 change: 0 additions & 1 deletion sriov/tests/SR_IOV_Permutation/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@

## Test Case Name: SR-IOV.Permutation
# UUID: dc4d0d06-1544-45f4-87c0-01015b11355a

### Objective(s): Test VFs (virtual function) configuration with different properties combined

Expand Down
1 change: 0 additions & 1 deletion sriov/tests/SR_IOV_Permutation/test_SR_IOV_Permutation.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
# UUID: dc4d0d06-1544-45f4-87c0-01015b11355a
import pytest
from sriov.common.utils import (
create_vfs,
Expand Down
1 change: 0 additions & 1 deletion sriov/tests/SR_IOV_Permutation_DPDK/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@

## Test Case Name: SR-IOV.Permutation
# UUID: 52d5e6ca-8a98-4151-9b2e-7a2946073325

### Objective(s): Test VFs (virtual function) bound to DPDK driver with different properties combined

Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
# UUID: 52d5e6ca-8a98-4151-9b2e-7a2946073325
import pytest
from time import sleep
from sriov.common.utils import (
Expand Down
1 change: 0 additions & 1 deletion sriov/tests/SR_IOV_QinQ/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@

## Test Case Name: SR-IOV.QinQ
# UUID: ddcb6e81-8433-4dfb-bea9-0351d20f8ac2

### Objective(s): Test and ensure that QinQ on VF works with kernel drive.

Expand Down
1 change: 0 additions & 1 deletion sriov/tests/SR_IOV_QinQ/test_SR_IOV_QinQ.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
# UUID: ddcb6e81-8433-4dfb-bea9-0351d20f8ac2
from sriov.common.utils import create_vfs, execute_and_assert, start_tmux, stop_tmux


Expand Down
1 change: 0 additions & 1 deletion sriov/tests/SR_IOV_Spoof_Mac/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@

## Test Case Name: SR-IOV.Spoof.Mac
# UUID: 66627809-09ff-4c41-a732-35b2f6f9528c

### Objective(s): Test and ensure that VF (virtual function) spoof check and custom mac can be set at the same time.

Expand Down
1 change: 0 additions & 1 deletion sriov/tests/SR_IOV_Spoof_Mac/test_SR_IOV_Spoof_Mac.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
# UUID: 66627809-09ff-4c41-a732-35b2f6f9528c
import pytest
from sriov.common.utils import create_vfs, execute_and_assert, start_tmux, stop_tmux

Expand Down
1 change: 0 additions & 1 deletion sriov/tests/SR_IOV_TrustMode/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@

## Test Case Name: SR-IOV.TrustMode
# UUID: 28be87c8-0b05-4eac-8ce4-6e5ebe95efc8

### Objective(s): Test and ensure that VF (virtual function) trust mode functions as intended.

Expand Down
1 change: 0 additions & 1 deletion sriov/tests/SR_IOV_TrustMode/test_SR_IOV_TrustMode.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
# UUID: 28be87c8-0b05-4eac-8ce4-6e5ebe95efc8
from sriov.common.utils import create_vfs, execute_and_assert


Expand Down
1 change: 0 additions & 1 deletion sriov/tests/SR_IOV_macAddress/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@

## Test Case Name: SR-IOV.macAddress
# UUID: 93ccf4b8-21e6-4148-848b-0f9eefdb20d6

### Objective(s): Test and ensure that VF (virtual function) MAC address functions as intended.

Expand Down
1 change: 0 additions & 1 deletion sriov/tests/SR_IOV_macAddress/test_SR_IOV_macAddress.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
# UUID: 93ccf4b8-21e6-4148-848b-0f9eefdb20d6
from sriov.common.utils import (
create_vfs,
execute_and_assert,
Expand Down
1 change: 0 additions & 1 deletion sriov/tests/SR_IOV_macAddressDuplicate/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@

## Test Case Name: SR-IOV.macAddressDuplicate
# UUID: 67e90840-5f2f-4423-a80d-828e99d43e55

### Objective(s): Test and ensure that duplicate mac address across VFs on the same PF is permitted.

Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
# UUID: 67e90840-5f2f-4423-a80d-828e99d43e55
from sriov.common.utils import create_vfs, execute_and_assert, verify_vf_address


Expand Down
1 change: 0 additions & 1 deletion sriov/tests/SR_IOV_macAddress_DPDK/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
## Test Case Name: SR-IOV.macAddress.DPDK
# UUID: d786bfca-219c-4b9f-ad9b-59fa1bc8cbba

### Objective(s): Test and ensure that VF (virtual function) MAC address functions as intended when bound to the DPDK driver.

Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
# UUID: d786bfca-219c-4b9f-ad9b-59fa1bc8cbba
from sriov.common.utils import (
create_vfs,
get_pci_address,
Expand Down
33 changes: 17 additions & 16 deletions sriov/tests/conftest.py
Original file line number Diff line number Diff line change
Expand Up @@ -170,26 +170,28 @@ def _report_extras(extra, request, settings, monkeypatch) -> None:
with open(settings.config["tests_doc_file"]) as f:
lines = f.readlines()

case_id = parse_file_for_field(
request.module.__file__, settings.config["tests_id_field"]
)
case_name = parse_file_for_field(
settings.config["tests_doc_file"], settings.config["tests_name_field"]
)

if case_name != "":
repo = git.Repo(search_parent_directories=True)
sha = repo.head.commit
sha = ''
git_tag = ''
for tag in repo.tags:
if tag.commit == sha:
git_tag = tag
break

try:
repo = git.Repo(search_parent_directories=True)
sha = repo.head.commit
# This will assume that there is one tag per commit.
for tag in repo.tags:
if tag.commit == sha:
git_tag = tag
break
except Exception:
pass

test_dir = os.path.dirname(request.module.__file__).split(os.sep)[-1]
if git_tag:
link = (
settings.config["github_tests_path"].replace("main", git_tag)
settings.config["github_tests_path"].replace("main", str(git_tag))
+ "/"
+ test_dir
+ "/"
Expand All @@ -204,14 +206,13 @@ def _report_extras(extra, request, settings, monkeypatch) -> None:
+ settings.config["tests_doc_file"]
)
else:
link = "Not Found"
case_name = "No tag or commit hash: No Link to"
link = "#"


extra.append(
extras.html(
'<p>Local Test Case ID: '
+ case_id
+ '</p>'
+ '<p>Link to the test specification: <a href="'
'<p>Link to the test specification: <a href="'
+ link
+ '">'
+ case_name
Expand Down

0 comments on commit 47165b6

Please sign in to comment.