oscap longrun tests parametrized for client rhel versions #15531

pondrejk · 2024-06-27T15:51:00Z

Problem Statement

Adjusting existing oscap longrun tests to run on multiple client rhel versions (to increase coverage, make them available for pit-client testing etc.)

Solution

Few notes:

As BZ#2176122 is considered not a bug, the default content available on satellite is defined by its underlying rhel version. As per docs, one needs to extract the content for other rhel versions from their respective scap-security-guide packages and do a file upload to satellite. This PR uses contenthosts to do it as they are created anyway. But due to this I cannot do all the preparatory steps in fixtures, as policy can be created only after client exists (hope this makes sense)
different content upload strategies (default/file upload) result in different naming for the contents, this PR accounts for that
separate CVs are now created per rhel version content, having it all in one CV caused some issues with the AK association
atm rhel7 client runs won't pass due to a bug (already fixed, but client repo needs a resnap)

Related Issues

pondrejk · 2024-06-27T15:51:44Z

trigger: test-robottelo
pytest: tests/foreman/longrun/test_oscap.py -k test_positive_oscap_run_via_ansible[rhel9]

Satellite-QE · 2024-06-27T16:15:32Z

PRT Result

Build Number: 7564
Build Status: UNSTABLE
PRT Comment: pytest tests/foreman/longrun/test_oscap.py -k  test_positive_oscap_run_via_ansible[rhel9] --external-logging
Test Result : ========== 1 failed, 15 deselected, 28 warnings in 1106.80s (0:18:26) ==========

pondrejk · 2024-06-28T07:57:58Z

trigger: test-robottelo
pytest: tests/foreman/longrun/test_oscap.py -k test_positive_oscap_run_via_ansible[rhel9]

Satellite-QE · 2024-06-28T08:27:26Z

PRT Result

Build Number: 7566
Build Status: UNSTABLE
PRT Comment: pytest tests/foreman/longrun/test_oscap.py -k test_positive_oscap_run_via_ansible[rhel9] --external-logging
Test Result : ========== 1 failed, 15 deselected, 28 warnings in 1221.92s (0:20:21) ==========

pondrejk · 2024-07-03T17:35:48Z

trigger: test-robottelo
pytest: tests/foreman/longrun/test_oscap.py -k test_positive_oscap_run_via_ansible[rhel8]

pondrejk · 2024-07-08T13:44:11Z

trigger: test-robottelo
pytest: tests/foreman/longrun/test_oscap.py -k test_positive_oscap_run_via_ansible[rhel8]

Satellite-QE · 2024-07-08T14:11:18Z

PRT Result

Build Number: 7666
Build Status: SUCCESS
PRT Comment: pytest tests/foreman/longrun/test_oscap.py -k test_positive_oscap_run_via_ansible[rhel8] --external-logging
Test Result : ========== 1 passed, 16 deselected, 29 warnings in 1144.38s (0:19:04) ==========

pondrejk · 2024-07-10T08:06:15Z

trigger: test-robottelo
pytest: --component SCAPPlugin --junit-xml=sat-SCAPPlugin-results.xml -o junit_suite_name=sat-SCAPPlugin --durations=20 --durations-min=600.0 tests/foreman
env:
    ROBOTTELO_server__deploy_arguments__deploy_rhel_version: '9'
theforeman:
    foreman_openscap: 575

pondrejk · 2024-07-10T08:09:02Z

sorry for a bit of a sidetrack, I was asked to review theforeman/foreman_openscap#575 and as I'd like to use tests from this pr, I'm using the above prt construct. The results shouldn't be used primarily to judge this pr, see the prt success above.

Satellite-QE · 2024-07-10T09:59:02Z

PRT Result

Build Number: 7694
Build Status: UNSTABLE
PRT Comment: pytest --component SCAPPlugin --junit-xml=sat-SCAPPlugin-results.xml -o junit_suite_name=sat-SCAPPlugin --durations=20 --durations-min=600.0 tests/foreman --external-logging
Test Result : = 32 failed, 47 passed, 5428 deselected, 6580 warnings, 28 errors in 6408.87s (1:46:48) =

pondrejk · 2024-07-10T11:52:03Z

trigger: test-robottelo
pytest: --component SCAPPlugin --junit-xml=sat-SCAPPlugin-results.xml -o junit_suite_name=sat-SCAPPlugin --durations=20 --durations-min=600.0 tests/foreman
theforeman:
    foreman_openscap: 575

Satellite-QE · 2024-07-10T14:04:56Z

PRT Result

Build Number: 7697
Build Status: UNSTABLE
PRT Comment: pytest --component SCAPPlugin --junit-xml=sat-SCAPPlugin-results.xml -o junit_suite_name=sat-SCAPPlugin --durations=20 --durations-min=600.0 tests/foreman --external-logging
Test Result : = 13 failed, 93 passed, 5428 deselected, 6667 warnings, 1 error in 7356.10s (2:02:36) =

mistakenly approved wrong PR

pondrejk · 2024-07-23T09:23:26Z

back from PTO, I'll be looking into the proposed changes, just FYI, these tests are affected by SAT-26726

pondrejk · 2024-07-23T11:59:35Z

trigger: test-robottelo
pytest: tests/foreman/longrun/test_oscap.py -k test_positive_oscap_run_via_ansible[rhel9_fips]

Satellite-QE · 2024-07-23T12:30:22Z

PRT Result

Build Number: 7817
Build Status: SUCCESS
PRT Comment: pytest tests/foreman/longrun/test_oscap.py -k test_positive_oscap_run_via_ansible[rhel9_fips] --external-logging
Test Result : ========== 1 passed, 16 deselected, 29 warnings in 1372.67s (0:22:52) ==========

ekohl · 2024-07-25T15:29:16Z

tests/foreman/longrun/test_oscap.py

+            'repo': settings.repos.satclient_repo.rhel9,
+            'akname': ak_name['rhel9'],
+            'cvname': cv_name['rhel9'],
+        },
+        {
+            'repo': settings.repos.satclient_repo.rhel8,
+            'akname': ak_name['rhel8'],
+            'cvname': cv_name['rhel8'],
+        },
+        {
+            'repo': settings.repos.satclient_repo.rhel7,
+            'akname': ak_name['rhel7'],
+            'cvname': cv_name['rhel7'],
+        },


Is this being too smart?

Suggested change

'repo': settings.repos.satclient_repo.rhel9,

'akname': ak_name['rhel9'],

'cvname': cv_name['rhel9'],

},

{

'repo': settings.repos.satclient_repo.rhel8,

'akname': ak_name['rhel8'],

'cvname': cv_name['rhel8'],

},

{

'repo': settings.repos.satclient_repo.rhel7,

'akname': ak_name['rhel7'],

'cvname': cv_name['rhel7'],

},

'repo': getattr(settings.repos.satclient_repo, rhel),

'akname': ak_name[rhel],

'cvname': cv_name[rhel],

} for rhel in ('rhel9', 'rhel8', 'rhel7')

ekohl · 2024-07-25T15:33:54Z

tests/foreman/longrun/test_oscap.py

-        profile = OSCAP_PROFILE['ospp8']
+@pytest.fixture
+def scap_prerequisites(module_org, default_proxy, target_sat):
+    # TODO: add support for RHEL9 (it doesn't have scap content in Sat by default) and parametrize distro


IIRC the problem is that we install scap-security-guide on Satellite, but if that's on RHEL 8 then it doesn't have RHEL 9 content. AFAIK this is why we upload the content from the content host (which you do in upload_scap_content_for_host). Though if the Satellite is on RHEL 9 then it should have the content available.

Would it be better to match the distro to the Satellite distro?

eh, further complication, this part got in with #15491 while this PR was open, I just rearranged stuff after I pulled in changes. Clearly we should consolidate the approches form thes PRs, but I don't want to touch it within this PR.

That's fair. I'd just ask you to address 13fef98#r1691678387 since that does feel in line with this PR (you're introducing profiles in this PR).

@ekohl ah that's right, comitted!

tests/foreman/longrun/test_oscap.py

pondrejk · 2024-07-29T07:06:56Z

trigger: test-robottelo
pytest: tests/foreman/longrun/test_oscap.py -k test_positive_oscap_run_via_ansible[rhel9_fips]

Satellite-QE · 2024-07-29T07:32:57Z

PRT Result

Build Number: 7883
Build Status: SUCCESS
PRT Comment: pytest tests/foreman/longrun/test_oscap.py -k test_positive_oscap_run_via_ansible[rhel9_fips] --external-logging
Test Result : ========== 1 passed, 16 deselected, 29 warnings in 1222.99s (0:20:22) ==========

pondrejk · 2024-07-29T13:19:00Z

trigger: test-robottelo
pytest: tests/foreman/longrun/test_oscap.py -k test_positive_oscap_run_via_ansible[rhel7]

Satellite-QE · 2024-07-29T13:30:19Z

PRT Result

Build Number: 7889
Build Status: UNSTABLE
PRT Comment: pytest tests/foreman/longrun/test_oscap.py -k test_positive_oscap_run_via_ansible[rhel7] --external-logging
Test Result : =========== 16 deselected, 22 warnings, 1 error in 190.75s (0:03:10) ===========

pondrejk · 2024-07-29T13:47:57Z

trigger: test-robottelo
pytest: tests/foreman/longrun/test_oscap.py -k test_positive_oscap_run_via_ansible[rhel7]

Satellite-QE · 2024-07-29T14:15:06Z

PRT Result

Build Number: 7890
Build Status: SUCCESS
PRT Comment: pytest tests/foreman/longrun/test_oscap.py -k test_positive_oscap_run_via_ansible[rhel7] --external-logging
Test Result : ========== 1 passed, 16 deselected, 29 warnings in 1085.36s (0:18:05) ==========

Co-authored-by: Ewoud Kohl van Wijngaarden <[email protected]>

…E#15531) * oscap longrun tests parametrized for client rhel versions * feedback from review * further refactoring * Update tests/foreman/longrun/test_oscap.py Co-authored-by: Ewoud Kohl van Wijngaarden <[email protected]> --------- Co-authored-by: Ewoud Kohl van Wijngaarden <[email protected]>

pondrejk added CherryPick PR needs CherryPick to previous branches Stream Introduced in or relating directly to Satellite Stream/Master 6.15.z Introduced in or relating directly to Satellite 6.15 labels Jun 27, 2024

pondrejk self-assigned this Jun 27, 2024

pondrejk requested review from a team as code owners June 27, 2024 15:51

Satellite-QE added the PRT-Failed Indicates that latest PRT run is failed for the PR label Jun 27, 2024

pondrejk force-pushed the oscap branch from 21a5415 to 728555a Compare June 28, 2024 07:57

pondrejk force-pushed the oscap branch 3 times, most recently from a406629 to 7b627ba Compare July 3, 2024 10:38

pondrejk force-pushed the oscap branch from 7b627ba to 2c30c2f Compare July 8, 2024 13:28

pondrejk requested a review from a team as a code owner July 8, 2024 13:28

Satellite-QE added PRT-Passed Indicates that latest PRT run is passed for the PR and removed PRT-Failed Indicates that latest PRT run is failed for the PR labels Jul 8, 2024

Satellite-QE added PRT-Failed Indicates that latest PRT run is failed for the PR and removed PRT-Passed Indicates that latest PRT run is passed for the PR labels Jul 10, 2024

lhellebr approved these changes Jul 10, 2024

View reviewed changes

Satellite-QE removed the PRT-Passed Indicates that latest PRT run is passed for the PR label Jul 23, 2024

pondrejk added No-CherryPick PR doesnt need CherryPick to previous branches and removed CherryPick PR needs CherryPick to previous branches 6.15.z Introduced in or relating directly to Satellite 6.15 labels Jul 23, 2024

Satellite-QE added the PRT-Passed Indicates that latest PRT run is passed for the PR label Jul 23, 2024

ekohl reviewed Jul 25, 2024

View reviewed changes

pondrejk force-pushed the oscap branch from dba93d3 to 13fef98 Compare July 26, 2024 07:42

Satellite-QE removed the PRT-Passed Indicates that latest PRT run is passed for the PR label Jul 26, 2024

Satellite-QE added the PRT-Passed Indicates that latest PRT run is passed for the PR label Jul 29, 2024

further refactoring

feb51ee

pondrejk force-pushed the oscap branch from 13fef98 to feb51ee Compare July 29, 2024 13:18

Satellite-QE removed the PRT-Passed Indicates that latest PRT run is passed for the PR label Jul 29, 2024

Satellite-QE added the PRT-Failed Indicates that latest PRT run is failed for the PR label Jul 29, 2024

Satellite-QE added PRT-Passed Indicates that latest PRT run is passed for the PR and removed PRT-Failed Indicates that latest PRT run is failed for the PR labels Jul 29, 2024

Update tests/foreman/longrun/test_oscap.py

37a8bfa

Co-authored-by: Ewoud Kohl van Wijngaarden <[email protected]>

Satellite-QE removed the PRT-Passed Indicates that latest PRT run is passed for the PR label Jul 29, 2024

ekohl approved these changes Jul 30, 2024

View reviewed changes

pondrejk merged commit 6d22330 into SatelliteQE:master Jul 31, 2024
9 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

oscap longrun tests parametrized for client rhel versions #15531

oscap longrun tests parametrized for client rhel versions #15531

pondrejk commented Jun 27, 2024

pondrejk commented Jun 27, 2024

Satellite-QE commented Jun 27, 2024

pondrejk commented Jun 28, 2024

Satellite-QE commented Jun 28, 2024

pondrejk commented Jul 3, 2024

pondrejk commented Jul 8, 2024

Satellite-QE commented Jul 8, 2024

pondrejk commented Jul 10, 2024

pondrejk commented Jul 10, 2024

Satellite-QE commented Jul 10, 2024

pondrejk commented Jul 10, 2024

Satellite-QE commented Jul 10, 2024

pondrejk commented Jul 23, 2024

pondrejk commented Jul 23, 2024

Satellite-QE commented Jul 23, 2024

ekohl Jul 25, 2024

ekohl Jul 25, 2024

pondrejk Jul 26, 2024

ekohl Jul 29, 2024

pondrejk Jul 29, 2024

pondrejk commented Jul 29, 2024

Satellite-QE commented Jul 29, 2024

pondrejk commented Jul 29, 2024

Satellite-QE commented Jul 29, 2024

pondrejk commented Jul 29, 2024

Satellite-QE commented Jul 29, 2024

oscap longrun tests parametrized for client rhel versions #15531

oscap longrun tests parametrized for client rhel versions #15531

Conversation

pondrejk commented Jun 27, 2024

Problem Statement

Solution

Related Issues

pondrejk commented Jun 27, 2024

Satellite-QE commented Jun 27, 2024

pondrejk commented Jun 28, 2024

Satellite-QE commented Jun 28, 2024

pondrejk commented Jul 3, 2024

pondrejk commented Jul 8, 2024

Satellite-QE commented Jul 8, 2024

pondrejk commented Jul 10, 2024

pondrejk commented Jul 10, 2024

Satellite-QE commented Jul 10, 2024

pondrejk commented Jul 10, 2024

Satellite-QE commented Jul 10, 2024

pondrejk commented Jul 23, 2024

pondrejk commented Jul 23, 2024

Satellite-QE commented Jul 23, 2024

ekohl Jul 25, 2024

Choose a reason for hiding this comment

ekohl Jul 25, 2024

Choose a reason for hiding this comment

pondrejk Jul 26, 2024

Choose a reason for hiding this comment

ekohl Jul 29, 2024

Choose a reason for hiding this comment

pondrejk Jul 29, 2024

Choose a reason for hiding this comment

pondrejk commented Jul 29, 2024

Satellite-QE commented Jul 29, 2024

pondrejk commented Jul 29, 2024

Satellite-QE commented Jul 29, 2024

pondrejk commented Jul 29, 2024

Satellite-QE commented Jul 29, 2024