diff --git a/ods_ci/tests/Tests/0100__platform/0101__deploy/0104__operators/0104__rhods_operator/0113__dsc_components.robot b/ods_ci/tests/Tests/0100__platform/0101__deploy/0104__operators/0104__rhods_operator/0113__dsc_components.robot index 93abff6ae..071a74dfe 100644 --- a/ods_ci/tests/Tests/0100__platform/0101__deploy/0104__operators/0104__rhods_operator/0113__dsc_components.robot +++ b/ods_ci/tests/Tests/0100__platform/0101__deploy/0104__operators/0104__rhods_operator/0113__dsc_components.robot @@ -63,7 +63,7 @@ ${IS_NOT_PRESENT} 1 Validate Kueue Managed State [Documentation] Validate that the DSC Kueue component Managed state creates the expected resources, ... check that kueue deployment is created and pod is in Ready state - [Tags] Operator Tier1 RHOAIENG-5435 kueue-managed + [Tags] Operator Tier1 RHOAIENG-5435 kueue-managed Integration Set DSC Component Managed State And Wait For Completion kueue ${KUEUE_DEPLOYMENT_NAME} ${KUEUE_LABEL_SELECTOR} @@ -71,7 +71,7 @@ Validate Kueue Managed State Validate Kueue Removed State [Documentation] Validate that Kueue management state Removed does remove relevant resources. - [Tags] Operator Tier1 RHOAIENG-5435 kueue-removed + [Tags] Operator Tier1 RHOAIENG-5435 kueue-removed Integration Set DSC Component Removed State And Wait For Completion kueue ${KUEUE_DEPLOYMENT_NAME} ${KUEUE_LABEL_SELECTOR} @@ -80,7 +80,7 @@ Validate Kueue Removed State Validate Codeflare Managed State [Documentation] Validate that the DSC Codeflare component Managed state creates the expected resources, ... check that Codeflare deployment is created and pod is in Ready state - [Tags] Operator Tier1 RHOAIENG-5435 codeflare-managed + [Tags] Operator Tier1 RHOAIENG-5435 codeflare-managed Integration Set DSC Component Managed State And Wait For Completion codeflare ${CODEFLARE_DEPLOYMENT_NAME} ${CODEFLARE_LABEL_SELECTOR} @@ -88,7 +88,7 @@ Validate Codeflare Managed State Validate Codeflare Removed State [Documentation] Validate that Codeflare management state Removed does remove relevant resources. - [Tags] Operator Tier1 RHOAIENG-5435 codeflare-removed + [Tags] Operator Tier1 RHOAIENG-5435 codeflare-removed Integration Set DSC Component Removed State And Wait For Completion codeflare ${CODEFLARE_DEPLOYMENT_NAME} ${CODEFLARE_LABEL_SELECTOR} @@ -97,7 +97,7 @@ Validate Codeflare Removed State Validate Ray Managed State [Documentation] Validate that the DSC Ray component Managed state creates the expected resources, ... check that Ray deployment is created and pod is in Ready state - [Tags] Operator Tier1 RHOAIENG-5435 ray-managed + [Tags] Operator Tier1 RHOAIENG-5435 ray-managed Integration Set DSC Component Managed State And Wait For Completion ray ${RAY_DEPLOYMENT_NAME} ${RAY_LABEL_SELECTOR} @@ -105,7 +105,7 @@ Validate Ray Managed State Validate Ray Removed State [Documentation] Validate that Ray management state Removed does remove relevant resources. - [Tags] Operator Tier1 RHOAIENG-5435 ray-removed + [Tags] Operator Tier1 RHOAIENG-5435 ray-removed Integration Set DSC Component Removed State And Wait For Completion ray ${RAY_DEPLOYMENT_NAME} ${RAY_LABEL_SELECTOR} @@ -114,7 +114,7 @@ Validate Ray Removed State Validate Training Operator Managed State [Documentation] Validate that the DSC Training Operator component Managed state creates the expected resources, ... check that Training deployment is created and pod is in Ready state - [Tags] Operator Tier1 RHOAIENG-6627 training-managed + [Tags] Operator Tier1 RHOAIENG-6627 training-managed Integration Set DSC Component Managed State And Wait For Completion trainingoperator ${TRAINING_DEPLOYMENT_NAME} ${TRAINING_LABEL_SELECTOR} @@ -122,7 +122,7 @@ Validate Training Operator Managed State Validate Training Operator Removed State [Documentation] Validate that Training Operator management state Removed does remove relevant resources. - [Tags] Operator Tier1 RHOAIENG-6627 training-removed + [Tags] Operator Tier1 RHOAIENG-6627 training-removed Integration Set DSC Component Removed State And Wait For Completion trainingoperator ${TRAINING_DEPLOYMENT_NAME} ${TRAINING_LABEL_SELECTOR} @@ -131,7 +131,7 @@ Validate Training Operator Removed State Validate Dashboard Managed State [Documentation] Validate that the DSC Dashboard component Managed state creates the expected resources, ... check that Dashboard deployment is created and all pods are in Ready state - [Tags] Operator Tier1 RHOAIENG-7298 dashboard-managed + [Tags] Operator Tier1 RHOAIENG-7298 dashboard-managed Integration Set DSC Component Managed State And Wait For Completion dashboard ${DASHBOARD_DEPLOYMENT_NAME} ${DASHBOARD_LABEL_SELECTOR} @@ -139,7 +139,7 @@ Validate Dashboard Managed State Validate Dashboard Removed State [Documentation] Validate that Dashboard management state Removed does remove relevant resources. - [Tags] Operator Tier1 RHOAIENG-7298 dashboard-removed + [Tags] Operator Tier1 RHOAIENG-7298 dashboard-removed Integration Set DSC Component Removed State And Wait For Completion dashboard ${DASHBOARD_DEPLOYMENT_NAME} ${DASHBOARD_LABEL_SELECTOR} @@ -148,7 +148,7 @@ Validate Dashboard Removed State Validate Datasciencepipelines Managed State [Documentation] Validate that the DSC Datasciencepipelines component Managed state creates the expected resources, ... check that Datasciencepipelines deployment is created and pod is in Ready state - [Tags] Operator Tier1 RHOAIENG-7298 operator-datasciencepipelines-managed + [Tags] Operator Tier1 RHOAIENG-7298 operator-datasciencepipelines-managed Integration Set DSC Component Managed State And Wait For Completion datasciencepipelines ${DATASCIENCEPIPELINES_DEPLOYMENT_NAME} ${DATASCIENCEPIPELINES_LABEL_SELECTOR} @@ -156,7 +156,7 @@ Validate Datasciencepipelines Managed State Validate Datasciencepipelines Removed State [Documentation] Validate that Datasciencepipelines management state Removed does remove relevant resources. - [Tags] Operator Tier1 RHOAIENG-7298 operator-datasciencepipelines-removed + [Tags] Operator Tier1 RHOAIENG-7298 operator-datasciencepipelines-removed Integration Set DSC Component Removed State And Wait For Completion datasciencepipelines ${DATASCIENCEPIPELINES_DEPLOYMENT_NAME} ${DATASCIENCEPIPELINES_LABEL_SELECTOR} @@ -165,7 +165,7 @@ Validate Datasciencepipelines Removed State Validate TrustyAi Managed State [Documentation] Validate that the DSC TrustyAi component Managed state creates the expected resources, ... check that TrustyAi deployment is created and pod is in Ready state - [Tags] Operator Tier1 RHOAIENG-14018 trustyai-managed + [Tags] Operator Tier1 RHOAIENG-14018 trustyai-managed Integration Set DSC Component Managed State And Wait For Completion trustyai ${TRUSTYAI_CONTROLLER_MANAGER_DEPLOYMENT_NAME} ${TRUSTYAI_CONTROLLER_MANAGER_LABEL_SELECTOR} @@ -173,7 +173,7 @@ Validate TrustyAi Managed State Validate TrustyAi Removed State [Documentation] Validate that TrustyAi management state Removed does remove relevant resources. - [Tags] Operator Tier1 RHOAIENG-14018 trustyai-removed + [Tags] Operator Tier1 RHOAIENG-14018 trustyai-removed Integration Set DSC Component Removed State And Wait For Completion trustyai ${TRUSTYAI_CONTROLLER_MANAGER_DEPLOYMENT_NAME} ${TRUSTYAI_CONTROLLER_MANAGER_LABEL_SELECTOR} @@ -182,7 +182,7 @@ Validate TrustyAi Removed State Validate Modelmeshserving Managed State [Documentation] Validate that the DSC Modelmeshserving component Managed state creates the expected resources, ... check that Modelmeshserving deployment is created and pods are in Ready state - [Tags] Operator Tier1 RHOAIENG-8546 modelmeshserving-managed + [Tags] Operator Tier1 RHOAIENG-8546 modelmeshserving-managed Integration Set DSC Component Managed State And Wait For Completion modelmeshserving ${MODELMESH_CONTROLLER_DEPLOYMENT_NAME} ${MODELMESH_CONTROLLER_LABEL_SELECTOR} @@ -196,7 +196,7 @@ Validate Modelmeshserving Managed State Validate Modelmeshserving Removed State [Documentation] Validate that Modelmeshserving management state Removed does remove relevant resources. - [Tags] Operator Tier1 RHOAIENG-8546 modelmeshserving-removed + [Tags] Operator Tier1 RHOAIENG-8546 modelmeshserving-removed Integration Set DSC Component Removed State And Wait For Completion modelmeshserving ${MODELMESH_CONTROLLER_DEPLOYMENT_NAME} ${MODELMESH_CONTROLLER_LABEL_SELECTOR} @@ -214,7 +214,7 @@ Validate Modelmeshserving Removed State Validate ModelRegistry Managed State [Documentation] Validate that the DSC ModelRegistry component Managed state creates the expected resources, ... check that ModelRegistry deployment is created and pod is in Ready state - [Tags] Operator Tier1 RHOAIENG-10404 modelregistry-managed + [Tags] Operator Tier1 RHOAIENG-10404 modelregistry-managed Integration Set DSC Component Managed State And Wait For Completion modelregistry ${MODELREGISTRY_CONTROLLER__DEPLOYMENT_NAME} ${MODELREGISTRY_CONTROLLER__LABEL_SELECTOR} @@ -224,7 +224,7 @@ Validate ModelRegistry Managed State Validate ModelRegistry Removed State [Documentation] Validate that ModelRegistry management state Removed does remove relevant resources. - [Tags] Operator Tier1 RHOAIENG-10404 modelregistry-removed + [Tags] Operator Tier1 RHOAIENG-10404 modelregistry-removed Integration # Properly validate Removed state by first setting to Manged, which will ensure that namspace # was created as needed for later validating that namespace persisted when component is Removed @@ -240,7 +240,7 @@ Validate ModelRegistry Removed State Validate KServe Controller Manager Managed State [Documentation] Validate that the DSC KServe Controller Manager component Managed state creates the expected resources, ... check that KServe Controller Manager deployment is created and pod is in Ready state - [Tags] Operator Tier1 RHOAIENG-7217 kserve-controller-manager-managed + [Tags] Operator Tier1 RHOAIENG-7217 kserve-controller-manager-managed Integration Set DSC Component Managed State And Wait For Completion kserve ${KSERVE_CONTROLLER_MANAGER_DEPLOYMENT_NAME} ${KSERVE_CONTROLLER_MANAGER_LABEL_SELECTOR} @@ -248,7 +248,7 @@ Validate KServe Controller Manager Managed State Validate KServe Controller Manager Removed State [Documentation] Validate that KServe Controller Manager management state Removed does remove relevant resources. - [Tags] Operator Tier1 RHOAIENG-7217 kserve-controller-manager-removed + [Tags] Operator Tier1 RHOAIENG-7217 kserve-controller-manager-removed Integration Set DSC Component Removed State And Wait For Completion kserve ${KSERVE_CONTROLLER_MANAGER_DEPLOYMENT_NAME} ${KSERVE_CONTROLLER_MANAGER_LABEL_SELECTOR} @@ -261,7 +261,7 @@ Validate KServe Controller Manager Removed State Validate Workbenches Managed State [Documentation] Validate that the DSC Workbenches component Managed state creates the expected resources, ... check that Workbenches deployment is created and pods are in Ready state - [Tags] Operator Tier1 workbenches-managed + [Tags] Operator Tier1 workbenches-managed Integration Set DSC Component Managed State And Wait For Completion workbenches ${NOTEBOOK_DEPLOYMENT_NAME} ${NOTEBOOK_CONTROLLER_MANAGER_LABEL_SELECTOR} @@ -271,7 +271,7 @@ Validate Workbenches Managed State Validate Workbenches Removed State [Documentation] Validate that Workbenches component management state Removed does remove relevant resources. - [Tags] Operator Tier1 workbenches-removed + [Tags] Operator Tier1 workbenches-removed Integration Set DSC Component Removed State And Wait For Completion workbenches ${NOTEBOOK_DEPLOYMENT_NAME} ${NOTEBOOK_CONTROLLER_MANAGER_LABEL_SELECTOR} @@ -281,7 +281,7 @@ Validate Workbenches Removed State Validate Support For Configuration Of Controller Resources [Documentation] Validate support for configuration of controller resources in component deployments - [Tags] Operator Tier1 ODS-2664 + [Tags] Operator Tier1 ODS-2664 Integration FOR ${controller} IN @{CONTROLLERS_LIST} ${rc}= Run And Return Rc ... oc patch Deployment ${controller} -n ${APPLICATIONS_NAMESPACE} --type=json -p="[{'op': 'replace', 'path': '/spec/template/spec/containers/0/resources/limits/cpu', 'value': '600m'}]" # robocop: disable diff --git a/ods_ci/tests/Tests/0100__platform/0102__monitor_and_manage/0101__metrics/0101__metrics.robot b/ods_ci/tests/Tests/0100__platform/0102__monitor_and_manage/0101__metrics/0101__metrics.robot index 00defde73..f110c2571 100644 --- a/ods_ci/tests/Tests/0100__platform/0102__monitor_and_manage/0101__metrics/0101__metrics.robot +++ b/ods_ci/tests/Tests/0100__platform/0102__monitor_and_manage/0101__metrics/0101__metrics.robot @@ -28,6 +28,7 @@ Test Existence of Prometheus Alerting Rules [Tags] Smoke ... Tier1 ... ODS-509 + ... Monitoring Skip If RHODS Is Self-Managed Check Prometheus Alerting Rules @@ -36,6 +37,7 @@ Test Existence of Prometheus Recording Rules [Tags] Smoke ... Tier1 ... ODS-510 + ... Monitoring Skip If RHODS Is Self-Managed Check Prometheus Recording Rules @@ -44,6 +46,7 @@ Test Metric "Notebook CPU Usage" On ODS Prometheus [Tags] Sanity ... Tier1 ... ODS-178 + ... Monitoring Skip If RHODS Is Self-Managed ${cpu_usage_before} = Read Current CPU Usage Run Jupyter Notebook For 5 Minutes @@ -56,6 +59,7 @@ Test Metric "Rhods_Total_Users" On ODS Prometheus [Tags] Sanity ... Tier1 ... ODS-628 + ... Monitoring Skip If RHODS Is Self-Managed # Note: the expression ends with "step=1" to obtain the value for current second ${expression} = Set Variable rhods_total_users&step=1 @@ -81,6 +85,7 @@ Test Metric Existence For "Rhods_Aggregate_Availability" On ODS Prometheus [Tags] Sanity ... Tier1 ... ODS-636 + ... Monitoring Skip If RHODS Is Self-Managed ${expression} = Set Variable rhods_aggregate_availability&step=1 ${resp} = Prometheus.Run Query ${RHODS_PROMETHEUS_URL} ${RHODS_PROMETHEUS_TOKEN} ${expression} diff --git a/ods_ci/tests/Tests/0100__platform/0102__monitor_and_manage/0101__metrics/0102__billing_metrics.robot b/ods_ci/tests/Tests/0100__platform/0102__monitor_and_manage/0101__metrics/0102__billing_metrics.robot index 452e3dd09..4fd2bff15 100644 --- a/ods_ci/tests/Tests/0100__platform/0102__monitor_and_manage/0101__metrics/0102__billing_metrics.robot +++ b/ods_ci/tests/Tests/0100__platform/0102__monitor_and_manage/0101__metrics/0102__billing_metrics.robot @@ -27,6 +27,7 @@ Verify OpenShift Monitoring Results Are Correct When Running Undefined Queries [Tags] Sanity ... Tier1 ... ODS-173 + ... Monitoring Run OpenShift Metrics Query ${METRIC_RHODS_UNDEFINED} username=${OCP_ADMIN_USER.USERNAME} password=${OCP_ADMIN_USER.PASSWORD} ... auth_type=${OCP_ADMIN_USER.AUTH_TYPE} retry_attempts=1 Metrics.Verify Query Results Dont Contain Data @@ -37,6 +38,7 @@ Test Billing Metric (Notebook Cpu Usage) On OpenShift Monitoring [Tags] Sanity ... Tier1 ... ODS-175 + ... Monitoring Run Jupyter Notebook For 5 Minutes Verify Previus CPU Usage Is Greater Than Zero [Teardown] CleanUp JupyterHub And Close All Browsers @@ -46,6 +48,7 @@ Test Metric "Rhods_Total_Users" On Cluster Monitoring Prometheus [Tags] Sanity ... ODS-634 ... Tier1 + ... Monitoring Skip If RHODS Is Self-Managed ${value} = Run OpenShift Metrics Query query=rhods_total_users username=${OCP_ADMIN_USER.USERNAME} password=${OCP_ADMIN_USER.PASSWORD} ... auth_type=${OCP_ADMIN_USER.AUTH_TYPE} @@ -59,7 +62,7 @@ Test Metric "Rhods_Aggregate_Availability" On Cluster Monitoring Prometheus [Tags] Smoke ... ODS-637 ... Tier1 - + ... Monitoring Skip If RHODS Is Self-Managed ${value_openshift_observe} = Run OpenShift Metrics Query @@ -80,7 +83,7 @@ Test Metric "Active_Users" On OpenShift Monitoring On Cluster Monitoring Prometh [Tags] Sanity ... ODS-1053 ... Tier1 - + ... Monitoring ${active_users_before} = Run OpenShift Metrics Query ... username=${OCP_ADMIN_USER.USERNAME} password=${OCP_ADMIN_USER.PASSWORD} ... auth_type=${OCP_ADMIN_USER.AUTH_TYPE} query=cluster:usage:consumption:rhods:active_users @@ -110,6 +113,7 @@ Test Metric "Active Notebook Pod Time" On OpenShift Monitoring - Cluster Monitor [Tags] Sanity ... ODS-1055 ... Tier1 + ... Monitoring @{list_of_usernames} = Create List ${TEST_USER_3.USERNAME} ${TEST_USER_4.USERNAME} Log In N Users To JupyterLab And Launch A Notebook For Each Of Them ... list_of_usernames=${list_of_usernames} diff --git a/ods_ci/tests/Tests/0100__platform/0102__monitor_and_manage/0101__metrics/0103__check_pager_duty.robot b/ods_ci/tests/Tests/0100__platform/0102__monitor_and_manage/0101__metrics/0103__check_pager_duty.robot index f0e284cf4..d517b72b8 100644 --- a/ods_ci/tests/Tests/0100__platform/0102__monitor_and_manage/0101__metrics/0103__check_pager_duty.robot +++ b/ods_ci/tests/Tests/0100__platform/0102__monitor_and_manage/0101__metrics/0103__check_pager_duty.robot @@ -33,6 +33,7 @@ PagerDuty Dummy Secret Verification ... Tier1 ... ODS-737 ... Deployment-Cli + ... Monitoring Skip If RHODS Is Self-Managed ${service_key} Get PagerDuty Key From Alertmanager ConfigMap ${secret_key} Get PagerDuty Key From Secrets diff --git a/ods_ci/tests/Tests/0100__platform/0102__monitor_and_manage/0102__alerts/0102__alerts.robot b/ods_ci/tests/Tests/0100__platform/0102__monitor_and_manage/0102__alerts/0102__alerts.robot index 874038d2d..3629d240f 100644 --- a/ods_ci/tests/Tests/0100__platform/0102__monitor_and_manage/0102__alerts/0102__alerts.robot +++ b/ods_ci/tests/Tests/0100__platform/0102__monitor_and_manage/0102__alerts/0102__alerts.robot @@ -32,7 +32,7 @@ Verify All Alerts Severity [Tags] Smoke ... Tier1 ... ODS-1227 - + ... Monitoring Verify "DeadManSnitch" Alerts Severity And Continue On Failure Verify "Kubeflow Notebook Controller Pod Is Not Running" Alerts Severity And Continue On Failure Verify "ODH Notebook Controller Pod Is Not Running" Alerts Severity And Continue On Failure @@ -46,7 +46,7 @@ Verify No Alerts Are Firing Except For DeadManSnitch # robocop: disable:too-l [Tags] Smoke ... Tier1 ... ODS-540 - + ... Monitoring Verify Alert Is Firing And Continue On Failure ... DeadManSnitch DeadManSnitch @@ -67,7 +67,7 @@ Verify Alert RHODS-PVC-Usage-Above-90 Is Fired When User PVC Is Above 90 Percent ... verifies that alert "User notebook pvc usage above 90%" is fired [Tags] Tier2 ... ODS-516 - + ... Monitoring Fill Up User PVC ${NOTEBOOK_REPO_URL} ${TEST_ALERT_PVC90_NOTEBOOK_PATH} Prometheus.Wait Until Alert Is Firing ${RHODS_PROMETHEUS_URL} @@ -83,7 +83,7 @@ Verify Alert RHODS-PVC-Usage-At-100 Is Fired When User PVC Is At 100 Percent ... verifies that alert "User notebook pvc usage at 100%" is fired [Tags] Tier2 ... ODS-517 - + ... Monitoring Fill Up User PVC ${NOTEBOOK_REPO_URL} ${TEST_ALERT_PVC100_NOTEBOOK_PATH} Prometheus.Wait Until Alert Is Firing ${RHODS_PROMETHEUS_URL} @@ -99,7 +99,7 @@ Verify Alerts Are Fired When RHODS Dashboard Is Down # robocop: disable:too-l ... are fired when rhods-dashboard is not working [Tags] Tier3 ... ODS-739 - + ... Monitoring Skip If Alert Is Already Firing ${RHODS_PROMETHEUS_URL} ... ${RHODS_PROMETHEUS_TOKEN} ... SLOs-haproxy_backend_http_responses_dashboard @@ -145,7 +145,7 @@ Verify Alert "Kubeflow notebook controller pod is not running" Is Fired When Kub ... when notebook-controller-deployment-xxx pod is not running [Tags] Tier3 ... ODS-1700 - + ... Monitoring Skip If Alert Is Already Firing ${RHODS_PROMETHEUS_URL} ... ${RHODS_PROMETHEUS_TOKEN} ... RHODS Notebook controllers @@ -177,7 +177,7 @@ Verify Alert "ODH notebook controller pod is not running" Is Fired When ODH Cont ... when odh-notebook-controller-manager-xxx pod is not running [Tags] Tier3 ... ODS-1701 - + ... Monitoring Skip If Alert Is Already Firing ${RHODS_PROMETHEUS_URL} ... ${RHODS_PROMETHEUS_TOKEN} ... RHODS Notebook controllers @@ -209,7 +209,7 @@ Verify That MT-SRE Are Not Paged For Alerts In Clusters Used For Development Or [Tags] Sanity ... ODS-1058 ... Tier1 - + ... Monitoring ${res} = Check Cluster Name Contain "Aisrhods" Or Not IF ${res} ${receiver} = Set Variable alerts-sink