Test CV incremental update & publish time #14749

damoore044 · 2024-04-11T18:08:03Z

Problem Statement

Implement backlogged desired coverage - PR #9991 , authored by @latran

Solution

Automation for incremental update duration coverage, for BZ 2117760 and BZ 1829266
Steps:
- On an existing CV with decent amount of content, perform incremental update with only one errata.
- Publish the entire content view.
- Expect: The duration of the incremental update should almost always be shorter than the CV publish. To cover the BZs fully, if the update does take longer, check that it was close, that update did not take significantly longer.
Original issue raised in BZs: CV publish would only take a few minutes as expected, but incremental update with one or a couple errata was taking over an hour.
timeout (to catch any future performance degradation) - for incr. update is 20 seconds,
for publish of CV timeout is 30 seconds, based on prt performance.
Includes logging duration for both incr. update and cv publish.

PRT case

trigger: test-robottelo
pytest: tests/foreman/longrun/test_inc_updates.py::test_positive_incremental_update_time

damoore044 · 2024-04-11T18:08:19Z

trigger: test-robottelo
pytest: tests/foreman/longrun/test_inc_updates.py::test_positive_incremental_update_time

Satellite-QE · 2024-04-11T18:33:46Z

PRT Result

Build Number: 6469
Build Status: SUCCESS
PRT Comment: pytest tests/foreman/longrun/test_inc_updates.py::test_positive_incremental_update_time --external-logging
Test Result : ================== 1 passed, 11 warnings in 989.35s (0:16:29) ==================

damoore044 · 2024-04-11T18:40:43Z

trigger: test-robottelo
pytest: tests/foreman/longrun/test_inc_updates.py::test_positive_incremental_update_time

Satellite-QE · 2024-04-11T19:04:37Z

PRT Result

Build Number: 6470
Build Status: SUCCESS
PRT Comment: pytest tests/foreman/longrun/test_inc_updates.py::test_positive_incremental_update_time --external-logging
Test Result : ================= 1 passed, 11 warnings in 1048.82s (0:17:28) ==================

Griffin-Sullivan · 2024-04-12T12:59:10Z

tests/foreman/longrun/test_inc_updates.py

+    # rhel8 base os
+    rhel8_repo_id = module_target_sat.api_factory.enable_rhrepo_and_fetchid(
+        basearch=DEFAULT_ARCHITECTURE,
+        org_id=module_sca_manifest_org.id,
+        product=PRDS['rhel8'],
+        repo=REPOS['rhel8_bos']['name'],
+        reposet=REPOSET['rhel8_bos'],
+        releasever=REPOS['rhel8_bos']['releasever'],
+    )
+    rhel8_repo = module_target_sat.api.Repository(id=rhel8_repo_id).read()
+    assert rhel8_repo.sync(timeout=3000)['result'] == 'success'
+    # rh sat tools
+    rhst8_repo_id = module_target_sat.api_factory.enable_rhrepo_and_fetchid(
+        basearch=DEFAULT_ARCHITECTURE,
+        org_id=module_sca_manifest_org.id,
+        product=PRDS['rhel8'],
+        repo=REPOS['rhst8']['name'],
+        reposet=REPOSET['rhst8'],
+        releasever=None,
+    )
+    rhst8_repo = module_target_sat.api.Repository(id=rhst8_repo_id).read()
+    assert rhst8_repo.sync()['result'] == 'success'
+    # rh sat client
+    rhsc8_repo_id = module_target_sat.api_factory.enable_rhrepo_and_fetchid(
+        basearch=DEFAULT_ARCHITECTURE,
+        org_id=module_sca_manifest_org.id,
+        product=PRDS['rhel8'],
+        repo=REPOS['rhsclient8']['name'],
+        reposet=REPOSET['rhsclient8'],
+        releasever=None,
+    )
+    rhsc8_repo = module_target_sat.api.Repository(id=rhsc8_repo_id).read()
+    assert rhsc8_repo.sync()['result'] == 'success'


It would be better to loop through these for repo in ['rhel8_bos', 'rhst8', 'rhsclient8']. Or if you want to write a helper function that works too.

May be, writing separate fixtures can be good way for this test module, this could reuse in any new additional test case.
btw there is an existing fixture for Satellite client repo for RHEL7 (sat_client_repo) and can be use instead of rhsclient8

We could also .sync(synchronous=False) to make the test faster (since the syncs would run in parallel) and do wait_for_task to ensure they succeeded.

@vsedmik agree with synch(synchronous=False), the lastest push includes the 3 repos in loop being added to CV and then sync started. Then we wait for repo sync tasks, poll succeeds, then perform incremental testing.

@Griffin-Sullivan @vijaysawant for the implementation of a fixture, I didn't plan to use these repositories often,
do we think a fixture would be needed for just this case ?
For Griffin's recommendation, looping just once through the repo_ids, latest push has this refactored.
Let me know how the changes look!

@Griffin-Sullivan Are you fine with the changes? Can we resolve this thread?

Griffin-Sullivan · 2024-04-12T13:01:11Z

tests/foreman/longrun/test_inc_updates.py

+    cv = module_target_sat.cli_factory.make_content_view(
+        {'organization-id': module_sca_manifest_org.id}
+    )
+    rhel_repos = [rhst8_repo_id, rhsc8_repo_id, rhel8_repo_id]
+    for repo_id in rhel_repos:
+        module_target_sat.cli.ContentView.add_repository(
+            {
+                'id': cv['id'],
+                'organization-id': module_sca_manifest_org.id,
+                'repository-id': repo_id,
+            }
+        )


If you add the loop I described above you could actually:

make the CV

Loop through each repo, sync, then add it to the CV

Griffin-Sullivan · 2024-04-12T13:04:06Z

tests/foreman/longrun/test_inc_updates.py

+    timeout = 120  # seconds
+    start_time = datetime.utcnow() - timedelta(seconds=1)
+    result = module_target_sat.cli.ContentView.version_incremental_update(
+        {'content-view-version-id': cvv['id'], 'errata-ids': 'RHSA-2023:5982'}


Is there any way we can fetch this errata-id from somewhere?

Get the applicable errata reference from same file

errata_list = get_applicable_errata(<any repository object>) ... errata_id = errata_list[0].id

We don't need this specific errata_id, I think we could also use existing REAL_RHEL8_1_ERRATA_ID -
RHSA-2022:4867

Griffin-Sullivan · 2024-04-12T13:05:52Z

tests/foreman/longrun/test_inc_updates.py

+    )
+    assert 'version-1.1' in str(result[0].keys())
+    module_target_sat.cli.ContentView.publish({'id': cv['id']})
+    assert (datetime.utcnow() - timedelta(seconds=1)) - start_time <= timedelta(seconds=timeout)


Is it just the incremental update we want to take less than 120 seconds? Right now this is coded for the incremental update and the publish afterwards.

Right the original BZs mentioned a difference in the time to publish, and the time to promote.
Do you think it's worth checking time for both publish and promote of incremental version, separately asserting both durations?

I'd stick to the BZ (inc-update + publish).

vijaysawant

Please look over pending comments and resolve conversation.
Suggested changes can be implemented.
Otherwise everything looks good.

vijaysawant · 2024-04-12T15:16:16Z

tests/foreman/longrun/test_inc_updates.py

+    # rhel8 base os
+    rhel8_repo_id = module_target_sat.api_factory.enable_rhrepo_and_fetchid(
+        basearch=DEFAULT_ARCHITECTURE,
+        org_id=module_sca_manifest_org.id,
+        product=PRDS['rhel8'],
+        repo=REPOS['rhel8_bos']['name'],
+        reposet=REPOSET['rhel8_bos'],
+        releasever=REPOS['rhel8_bos']['releasever'],
+    )
+    rhel8_repo = module_target_sat.api.Repository(id=rhel8_repo_id).read()
+    assert rhel8_repo.sync(timeout=3000)['result'] == 'success'
+    # rh sat tools
+    rhst8_repo_id = module_target_sat.api_factory.enable_rhrepo_and_fetchid(
+        basearch=DEFAULT_ARCHITECTURE,
+        org_id=module_sca_manifest_org.id,
+        product=PRDS['rhel8'],
+        repo=REPOS['rhst8']['name'],
+        reposet=REPOSET['rhst8'],
+        releasever=None,
+    )
+    rhst8_repo = module_target_sat.api.Repository(id=rhst8_repo_id).read()
+    assert rhst8_repo.sync()['result'] == 'success'
+    # rh sat client
+    rhsc8_repo_id = module_target_sat.api_factory.enable_rhrepo_and_fetchid(
+        basearch=DEFAULT_ARCHITECTURE,
+        org_id=module_sca_manifest_org.id,
+        product=PRDS['rhel8'],
+        repo=REPOS['rhsclient8']['name'],
+        reposet=REPOSET['rhsclient8'],
+        releasever=None,
+    )
+    rhsc8_repo = module_target_sat.api.Repository(id=rhsc8_repo_id).read()
+    assert rhsc8_repo.sync()['result'] == 'success'


May be, writing separate fixtures can be good way for this test module, this could reuse in any new additional test case.
btw there is an existing fixture for Satellite client repo for RHEL7 (sat_client_repo) and can be use instead of rhsclient8

vijaysawant · 2024-04-12T15:23:10Z

tests/foreman/longrun/test_inc_updates.py

+    timeout = 120  # seconds
+    start_time = datetime.utcnow() - timedelta(seconds=1)
+    result = module_target_sat.cli.ContentView.version_incremental_update(
+        {'content-view-version-id': cvv['id'], 'errata-ids': 'RHSA-2023:5982'}


Get the applicable errata reference from same file

errata_list = get_applicable_errata(<any repository object>) ... errata_id = errata_list[0].id

vsedmik

Pending @Griffin-Sullivan's good suggestion, otherwise looks good.

vsedmik · 2024-04-12T15:20:34Z

tests/foreman/longrun/test_inc_updates.py

+    # rhel8 base os
+    rhel8_repo_id = module_target_sat.api_factory.enable_rhrepo_and_fetchid(
+        basearch=DEFAULT_ARCHITECTURE,
+        org_id=module_sca_manifest_org.id,
+        product=PRDS['rhel8'],
+        repo=REPOS['rhel8_bos']['name'],
+        reposet=REPOSET['rhel8_bos'],
+        releasever=REPOS['rhel8_bos']['releasever'],
+    )
+    rhel8_repo = module_target_sat.api.Repository(id=rhel8_repo_id).read()
+    assert rhel8_repo.sync(timeout=3000)['result'] == 'success'
+    # rh sat tools
+    rhst8_repo_id = module_target_sat.api_factory.enable_rhrepo_and_fetchid(
+        basearch=DEFAULT_ARCHITECTURE,
+        org_id=module_sca_manifest_org.id,
+        product=PRDS['rhel8'],
+        repo=REPOS['rhst8']['name'],
+        reposet=REPOSET['rhst8'],
+        releasever=None,
+    )
+    rhst8_repo = module_target_sat.api.Repository(id=rhst8_repo_id).read()
+    assert rhst8_repo.sync()['result'] == 'success'
+    # rh sat client
+    rhsc8_repo_id = module_target_sat.api_factory.enable_rhrepo_and_fetchid(
+        basearch=DEFAULT_ARCHITECTURE,
+        org_id=module_sca_manifest_org.id,
+        product=PRDS['rhel8'],
+        repo=REPOS['rhsclient8']['name'],
+        reposet=REPOSET['rhsclient8'],
+        releasever=None,
+    )
+    rhsc8_repo = module_target_sat.api.Repository(id=rhsc8_repo_id).read()
+    assert rhsc8_repo.sync()['result'] == 'success'


We could also .sync(synchronous=False) to make the test faster (since the syncs would run in parallel) and do wait_for_task to ensure they succeeded.

vsedmik · 2024-04-12T15:23:55Z

tests/foreman/longrun/test_inc_updates.py

+    timeout = 120  # seconds
+    start_time = datetime.utcnow() - timedelta(seconds=1)


Btw, what is the current inc-update + publish time? Does the 120s have a safe margin?
Also wanted to ask about the timedelta(seconds=1) purpose. Some rounding?

Locally I get about 10 seconds to update/create incremental version, and maybe 15-20 seconds to publish.
*edit- avg times reflect most recent testing/logging

~~I will add some logging output of the time, and per @Griffin-Sullivan suggestion, think we should track the duration of each, incremental update & publish, individually.~~
^ Done

As for timedelta rounding, I find this works best for asserting some satellite task time started on or after.
It's not looking for any tasks yet so wasn't quite needed, but the next push will find and wait_for_tasks.

vsedmik · 2024-04-12T15:25:37Z

tests/foreman/longrun/test_inc_updates.py

+    )
+    assert 'version-1.1' in str(result[0].keys())
+    module_target_sat.cli.ContentView.publish({'id': cv['id']})
+    assert (datetime.utcnow() - timedelta(seconds=1)) - start_time <= timedelta(seconds=timeout)


I'd stick to the BZ (inc-update + publish).

damoore044 · 2024-04-12T19:41:37Z

trigger: test-robottelo
pytest: tests/foreman/longrun/test_inc_updates.py::test_positive_incremental_update_time

Satellite-QE · 2024-04-12T20:05:26Z

PRT Result

Build Number: 6492
Build Status: SUCCESS
PRT Comment: pytest tests/foreman/longrun/test_inc_updates.py::test_positive_incremental_update_time --external-logging
Test Result : ================= 1 passed, 14 warnings in 1123.09s (0:18:43) ==================

Griffin-Sullivan

If PRT passes I'm good with this. Nice work!

tests/foreman/longrun/test_inc_updates.py

Satellite-QE · 2024-04-16T14:53:04Z

PRT Result

Build Number: 6523
Build Status: SUCCESS
PRT Comment: pytest tests/foreman/longrun/test_inc_updates.py::test_positive_incremental_update_time --external-logging
Test Result : ================== 1 passed, 14 warnings in 964.66s (0:16:04) ==================

damoore044 · 2024-04-16T17:43:04Z

trigger: test-robottelo
pytest: tests/foreman/longrun/test_inc_updates.py::test_positive_incremental_update_time

damoore044 · 2024-04-16T18:03:16Z

trigger: test-robottelo
pytest: tests/foreman/longrun/test_inc_updates.py::test_positive_incremental_update_time

Satellite-QE · 2024-04-16T18:09:15Z

PRT Result

Build Number: 6532
Build Status: SUCCESS
PRT Comment: pytest tests/foreman/longrun/test_inc_updates.py::test_positive_incremental_update_time --external-logging
Test Result : ================= 1 passed, 14 warnings in 1034.16s (0:17:14) ==================

Satellite-QE · 2024-04-16T18:29:10Z

PRT Result

Build Number: 6534
Build Status: SUCCESS
PRT Comment: pytest tests/foreman/longrun/test_inc_updates.py::test_positive_incremental_update_time --external-logging
Test Result : ================== 1 passed, 14 warnings in 998.03s (0:16:38) ==================

tests/foreman/longrun/test_inc_updates.py

damoore044 · 2024-04-17T00:41:14Z

trigger: test-robottelo
pytest: tests/foreman/longrun/test_inc_updates.py::test_positive_incremental_update_time

Satellite-QE · 2024-04-17T01:03:41Z

PRT Result

Build Number: 6537
Build Status: SUCCESS
PRT Comment: pytest tests/foreman/longrun/test_inc_updates.py::test_positive_incremental_update_time --external-logging
Test Result : ================= 1 passed, 14 warnings in 1014.44s (0:16:54) ==================

jyejare

ACK pending question.

tests/foreman/longrun/test_inc_updates.py

Test incremental update time (cherry picked from commit 11858fa)

Test incremental update time

damoore044 added the Stream Introduced in or relating directly to Satellite Stream/Master label Apr 11, 2024

Satellite-QE added the PRT-Passed Indicates that latest PRT run is passed for the PR label Apr 11, 2024

damoore044 changed the title ~~Test incremental update time~~ Test CV incremental update & publish time Apr 11, 2024

damoore044 force-pushed the test_inc_update branch from c458337 to a11c399 Compare April 11, 2024 18:40

Satellite-QE removed the PRT-Passed Indicates that latest PRT run is passed for the PR label Apr 11, 2024

Satellite-QE added the PRT-Passed Indicates that latest PRT run is passed for the PR label Apr 11, 2024

damoore044 requested a review from omkarkhatavkar April 12, 2024 11:45

damoore044 added CherryPick PR needs CherryPick to previous branches 6.14.z Introduced in or relating directly to Satellite 6.14 6.15.z Introduced in or relating directly to Satellite 6.15 labels Apr 12, 2024

damoore044 marked this pull request as ready for review April 12, 2024 11:49

damoore044 requested a review from a team as a code owner April 12, 2024 11:49

github-actions bot assigned damoore044 Apr 12, 2024

Griffin-Sullivan suggested changes Apr 12, 2024

View reviewed changes

vijaysawant reviewed Apr 12, 2024

View reviewed changes

vsedmik reviewed Apr 12, 2024

View reviewed changes

damoore044 force-pushed the test_inc_update branch from a11c399 to 6a60b92 Compare April 12, 2024 19:31

Satellite-QE removed the PRT-Passed Indicates that latest PRT run is passed for the PR label Apr 12, 2024

damoore044 requested review from vijaysawant, Griffin-Sullivan and vsedmik April 12, 2024 19:38

damoore044 force-pushed the test_inc_update branch from 6a60b92 to 87edf52 Compare April 12, 2024 19:46

Satellite-QE added the PRT-Passed Indicates that latest PRT run is passed for the PR label Apr 12, 2024

vijaysawant approved these changes Apr 15, 2024

View reviewed changes

Griffin-Sullivan approved these changes Apr 16, 2024

View reviewed changes

tests/foreman/longrun/test_inc_updates.py Show resolved Hide resolved

Satellite-QE added the PRT-Passed Indicates that latest PRT run is passed for the PR label Apr 16, 2024

sambible approved these changes Apr 16, 2024

View reviewed changes

vsedmik approved these changes Apr 16, 2024

View reviewed changes

damoore044 force-pushed the test_inc_update branch from 1985aad to df22692 Compare April 16, 2024 17:42

Satellite-QE removed the PRT-Passed Indicates that latest PRT run is passed for the PR label Apr 16, 2024

damoore044 force-pushed the test_inc_update branch from df22692 to 06486b8 Compare April 16, 2024 18:00

Satellite-QE added the PRT-Passed Indicates that latest PRT run is passed for the PR label Apr 16, 2024

jameerpathan111 requested changes Apr 16, 2024

View reviewed changes

tests/foreman/longrun/test_inc_updates.py Show resolved Hide resolved

Test incremental update time

c57b26a

damoore044 force-pushed the test_inc_update branch from 06486b8 to c57b26a Compare April 17, 2024 00:39

Satellite-QE removed the PRT-Passed Indicates that latest PRT run is passed for the PR label Apr 17, 2024

damoore044 requested a review from jameerpathan111 April 17, 2024 00:42

Satellite-QE added the PRT-Passed Indicates that latest PRT run is passed for the PR label Apr 17, 2024

jameerpathan111 approved these changes Apr 17, 2024

View reviewed changes

jyejare reviewed Apr 17, 2024

View reviewed changes

tests/foreman/longrun/test_inc_updates.py Show resolved Hide resolved

lpramuk approved these changes Apr 17, 2024

View reviewed changes

lpramuk merged commit 11858fa into SatelliteQE:master Apr 17, 2024
11 checks passed

github-actions bot pushed a commit that referenced this pull request Apr 17, 2024

Test CV incremental update & publish time (#14749)

4a31ac0

Test incremental update time (cherry picked from commit 11858fa)

This was referenced Apr 17, 2024

[6.14.z] Test CV incremental update & publish time #14809

Merged

[Failed-AutoCherryPick] - Test CV incremental update & publish time #14810

Closed

damoore044 mentioned this pull request Apr 17, 2024

[6.15.z] Test CV incremental update & publish time #14815

Merged

jyejare pushed a commit to jyejare/robottelo that referenced this pull request Oct 19, 2024

Test CV incremental update & publish time (SatelliteQE#14749)

20862af

Test incremental update time

		timeout = 120 # seconds
		start_time = datetime.utcnow() - timedelta(seconds=1)

Test CV incremental update & publish time #14749

Test CV incremental update & publish time #14749

Conversation

damoore044 commented Apr 11, 2024 • edited Loading

Problem Statement

Solution

PRT case

damoore044 commented Apr 11, 2024

Satellite-QE commented Apr 11, 2024

damoore044 commented Apr 11, 2024

Satellite-QE commented Apr 11, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

damoore044 Apr 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vijaysawant left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vsedmik left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

damoore044 Apr 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

damoore044 commented Apr 12, 2024

Satellite-QE commented Apr 12, 2024

Griffin-Sullivan left a comment

Choose a reason for hiding this comment

Satellite-QE commented Apr 16, 2024

damoore044 commented Apr 16, 2024

damoore044 commented Apr 16, 2024

Satellite-QE commented Apr 16, 2024

Satellite-QE commented Apr 16, 2024

damoore044 commented Apr 17, 2024

Satellite-QE commented Apr 17, 2024

jyejare left a comment

Choose a reason for hiding this comment

damoore044 commented Apr 11, 2024 •

edited

Loading

damoore044 Apr 12, 2024 •

edited

Loading

damoore044 Apr 12, 2024 •

edited

Loading