Test script for updateRecommendation API #836

msvinaykumar · 2023-05-29T13:07:31Z

added updateREcomendation API to scale test and enabled simultaneous uploading of results and update recommendations. Additionally, we have introduced extra flags to selectively post results or update recommendations only. Moreover, it is now possible to specify a start date as well.

syntax ex:

python3 -u quickTestScalability.py --ip master-0.kruizevin.lab.psi.pnq2.redhat.com --port 30792 --count 1,1440 --minutesjump=15 --generaterecommendation --parallelresultcount 1 --startdate=2023-01-01T00:00:00.000Z --name temp24Hrs3 --postresults

above example creates experiment name temp24Hrs3_1
with 15 days of result data also triggers updateRecommendations api

Signed-off-by: msvinaykumar [email protected]

chandrams · 2023-06-15T10:14:30Z

@msvinaykumar - Please include functional test for updateRecommendations, I have listed a few scenarios below add if anything is missing to it.

Test scenarios for updateRecommendations, invoke updateRecommendations as below:

Post a valid experiment name & valid end time and validate recommendations are as expected (What is the valid range for end time?)
Post a valid experiment name, valid end time and valid start time (valid range), validate recommendations are as expected
Post the same experiment name & end time twice, validate recommendations - Should be same or within acceptable threshold values?

Validate status code & expected error messages for the below:
Post without any parameters
Post an invalid experiment name
Post only valid interval end time
Post a valid experiment name but an invalid end time
Post a valid experiment name and an invalid end time format
Post a valid experiment name and an valid end time format but an invalid start time format
Post a valid experiment name, valid end time and invalid start time (future date or past date beyond accepted valid range)
Test for boundary values for the timestamps if there are any ranges

chandrams · 2023-07-13T06:01:43Z

tests/scripts/remote_monitoring_tests/rest_apis/test_update_recommendations.py

+                        assert notification["type"] != "error"
+
+        response = update_recommendations(experiment_name, interval_start_time, end_time)
+        data = response.json()


@msvinaykumar - Please include validation of the json response of update_recommendations for all the success cases

I think we should add a jsonschema validation for the output of updateRecommendations json too. Please include this in all tests were updateRecommendations API is invoked.

chandrams · 2023-07-13T06:04:12Z

tests/scripts/remote_monitoring_tests/rest_apis/test_update_recommendations.py

+
+@pytest.mark.sanity
+def test_update_valid_recommendations_after_results_after_create_exp(cluster_type):
+    input_json_file = "../json_files/create_exp.json"


Include description for all the tests included with expected behavior

chandrams · 2023-07-13T06:06:07Z

tests/scripts/remote_monitoring_tests/rest_apis/test_e2e_workflow.py

-                # Sleep for 1 sec to get recommendations
-                time.sleep(1)
-
+                response = update_recommendations(experiment_name, interval_start_time, end_time)


Please update all the tests in test_list_recommendations.py to invoke update_recommendations

chandrams · 2023-07-13T06:27:03Z

tests/scripts/remote_monitoring_tests/rest_apis/test_update_recommendations.py

+    response = update_recommendations(None, None, None)
+    data = response.json()
+    assert response.status_code == ERROR_STATUS_CODE
+    assert data['message'] == UPDATE_RECOMMENDATIONS_MANDATORY_DEFAULT_MESSAGE


Shouldn't the error messages returned for negative tests match what is mentioned in the monitoringAPI.md ?#854

chandrams · 2023-07-13T06:28:35Z

tests/scripts/remote_monitoring_tests/rest_apis/test_update_recommendations.py

+    response = update_recommendations(experiment_name, None, None)
+    data = response.json()
+    assert response.status_code == ERROR_STATUS_CODE
+    assert data['message'] == UPDATE_RECOMMENDATIONS_MANDATORY_INTERVAL_END_DATE


Here the experiment_name doesn't exist I think, will it still return error message related to interval end date? Would it return experiment_name is invalid ?

Update accordingly. In general it will display 'data not found'.

chandrams · 2023-07-13T06:31:50Z

tests/scripts/remote_monitoring_tests/rest_apis/test_update_recommendations.py

+    response = update_recommendations(experiment_name, start_time, end_time)
+    data = response.json()
+    assert response.status_code == ERROR_STATUS_CODE
+    assert data['message'] == UPDATE_RECOMMENDATIONS_START_TIME_PRECEDE_END_TIME


Please update the API doc with this error message

Can you please share the results for all these tests.

Include these tests too

Test with boundary values for the timestamps, I see 15 days as the acceptable range in the md file

Include tests with invalid time format for start and end time

Can you please share the results for all these tests.

https://drive.google.com/drive/folders/17e1KE3eZKi8Waf3o7uvlMQ1SqJC8cpfW?usp=sharing

msvinaykumar · 2023-08-02T14:24:58Z

https://drive.google.com/drive/folders/17e1KE3eZKi8Waf3o7uvlMQ1SqJC8cpfW?usp=sharing

chandrams · 2023-08-02T12:02:51Z

tests/scripts/remote_monitoring_tests/rest_apis/test_list_recommendations.py

@@ -165,12 +185,13 @@ def test_list_recommendations_invalid_exp(cluster_type):
    response = delete_experiment(input_json_file)
    print("delete exp = ", response.status_code)

+
 @pytest.mark.sanity
 def test_list_recommendations_without_results(cluster_type):


Invoke updateRecommendations API in this test too to check behavior without results

chandrams · 2023-08-02T13:07:52Z

tests/scripts/remote_monitoring_tests/rest_apis/test_list_recommendations.py

@@ -112,7 +131,7 @@ def test_list_recommendations_without_parameters(cluster_type):
    # Validate the json values
    create_exp_json = read_json_data_from_file(input_json_file)
    update_results_json = []
-    update_results_json.append(result_json_arr[len(result_json_arr)-1])
+    update_results_json.append(result_json_arr[len(result_json_arr) - 1])


Does validate_reco_json work with this update_results_json?

chandrams · 2023-08-02T13:14:15Z

tests/scripts/remote_monitoring_tests/rest_apis/test_list_recommendations.py

-    assert data['message'] == "Bulk entries are currently unsupported!"
+    assert response.status_code == SUCCESS_STATUS_CODE
+    assert data['status'] == SUCCESS_STATUS
+    assert data['message'] == UPDATE_RESULTS_SUCCESS_MSG


Also update the below code in this test to invoke updateRecommendations API & validate recommendations using update results json instead of None

# Uncomment the below lines when bulk entries are allowed # update_results_json = read_json_data_from_file(result_json_file) # Since bulk entries are not supported passing None for update results json update_results_json = None validate_reco_json(create_exp_json[0], update_results_json, list_reco_json[0])

chandrams · 2023-08-02T13:33:06Z

tests/scripts/remote_monitoring_tests/rest_apis/test_update_recommendations.py

+                        assert notification["type"] != "error"
+
+        response = update_recommendations(experiment_name, interval_start_time, end_time)
+        data = response.json()


I think we should add a jsonschema validation for the output of updateRecommendations json too. Please include this in all tests were updateRecommendations API is invoked.

chandrams · 2023-08-03T06:31:07Z

tests/scripts/remote_monitoring_tests/rest_apis/test_update_results.py

@@ -453,17 +471,23 @@ def test_update_results_with_same_result(cluster_type):
    response = update_results(result_json_file)

    data = response.json()
-    assert response.status_code == ERROR_409_STATUS_CODE
+    assert response.status_code == ERROR_STATUS_CODE


Have we changed the response code to 400 now if the result with an existing timestamp is posted again? We will need to discuss this with Suraj too as they might have some logic based on 409 error code. Also if there is a change in error code we will need to update the UpdateResults.md

chandrams · 2023-08-03T07:03:51Z

tests/scripts/remote_monitoring_tests/rest_apis/test_update_results.py

+        assert response.status_code == ERROR_STATUS_CODE
+        assert data['status'] == ERROR_STATUS
+        assert data['message'] == 'Out of a total of 1 records, 1 failed to save'
+        assert data['data'][0]['errorReasons'][0] == UPDATE_RESULTS_DATE_PRECEDE_ERROR_MSG


UpdateResults.md needs to be updated with the errorReasons and error messages

chandrams · 2023-08-09T11:17:32Z

tests/scripts/quickTestScalability.py

@@ -18,359 +18,511 @@
 import datetime
 import requests
 import argparse
+import multiprocessing


@msvinaykumar - As discussed, please raise a separate PR for this on top of PR 832

chandrams · 2023-08-10T11:07:30Z

@msvinaykumar - Tests did not run completely, extended and negative were not run due to the below error, did it work for you? Looking into this

Cmd used:

./test_autotune.sh -c minikube -i quay.io/vinakuma/autotune_operator:ros4 --testsuite=remote_monitoring_tests --resultsdir=/home/csubrama/kruize-scalability-res

Error:

FAILED test_create_experiment.py::test_create_exp_with_both_deployment_name_selector
FAILED test_create_experiment.py::test_create_exp_with_invalid_header - asser...
=========== 2 failed, 37 passed, 216 deselected in 82.78s (0:01:22) ============

/home/csubrama/github/vinay/autotune/tests/scripts/remote_monitoring_tests/remote_monitoring_tests.sh: line 127: 0 + 2
3
3
3
3
3
3
1
1
1
1: syntax error in expression (error token is "3
3
3
3
3
3
1
1
1
1")

chandrams · 2023-08-10T12:39:45Z

@msvinaykumar - Tests did not run completely, extended and negative were not run due to the below error, did it work for you? Looking into this

Cmd used:

./test_autotune.sh -c minikube -i quay.io/vinakuma/autotune_operator:ros4 --testsuite=remote_monitoring_tests --resultsdir=/home/csubrama/kruize-scalability-res

Error:

FAILED test_create_experiment.py::test_create_exp_with_both_deployment_name_selector
FAILED test_create_experiment.py::test_create_exp_with_invalid_header - asser...
=========== 2 failed, 37 passed, 216 deselected in 82.78s (0:01:22) ============

/home/csubrama/github/vinay/autotune/tests/scripts/remote_monitoring_tests/remote_monitoring_tests.sh: line 127: 0 + 2
3
3
3
3
3
3
1
1
1
1: syntax error in expression (error token is "3
3
3
3
3
3
1
1
1
1")

grep at line 124 seems to be matching these prints in the test output - Out of a total of 1 records, 1 failed to save
Please modify this to match the failed results count

chandrams · 2023-08-10T12:58:12Z

tests/scripts/remote_monitoring_tests/rest_apis/test_list_recommendations.py

@@ -651,6 +708,7 @@ def test_list_recommendations_multiple_exps_with_missing_metrics(cluster_type):
        response = delete_experiment(create_exp_json_file)
        print("delete exp = ", response.status_code)

+
 @pytest.mark.extended
 @pytest.mark.parametrize("latest", ["true", "false"])
 def test_list_recommendations_with_only_latest(latest, cluster_type):


@msvinaykumar - rest_apis/test_list_recommendations.py::test_list_recommendations_with_only_latest[false] this test failed with the below error, can you please check

# Validate the json against the json schema errorMsg = validate_list_reco_json(list_reco_json, list_reco_json_schema) > assert errorMsg == "" E assert [<ValidationError: "'monitoring_start_time' is a required property">, <ValidationError: "'monitoring_end_time' is a re...Error: "'variation' is a required property">, <ValidationError: "'monitoring_start_time' is a required property">, ...] == '' test_list_recommendations.py:805: AssertionError

dinogun · 2023-08-12T04:31:35Z

@msvinaykumar does this need a rebase as well?

…ous uploading of results and recommendations. Additionally, we have introduced extra flags to selectively post results or update recommendations only. Moreover, it is now possible to specify a start date as well. Signed-off-by: msvinaykumar <[email protected]>

Signed-off-by: msvinaykumar <[email protected]>

chandrams · 2023-08-17T12:22:53Z

@msvinaykumar - Please check the timestamp format in the error message

{"message":"Out of a total of 1 records, 1 failed to save","httpcode":400,"documentationLink":"","status":"ERROR","data":[{"interval_start_time":"Aug 17, 2023, 4:35:46 PM","interval_end_time":"Aug 17, 2023, 4:50:46 PM",

Signed-off-by: msvinaykumar <[email protected]>

dinogun

LGTM

msvinaykumar self-assigned this May 29, 2023

msvinaykumar added this to the Kruize 0.0.16_rm Release milestone May 29, 2023

msvinaykumar linked an issue May 29, 2023 that may be closed by this pull request

New API 'UpdateRecommendations' to support non-chronological order of data #816

Closed

msvinaykumar changed the title ~~Included updateRecommendation API~~ Test script for updateRecommendation API Jun 5, 2023

rbadagandi1 modified the milestones: Kruize 0.0.16_rm Release, Kruize 0.0.17_rm Release Jun 21, 2023

msvinaykumar force-pushed the quicktest2 branch 2 times, most recently from 9545363 to c4c8710 Compare June 27, 2023 15:24

msvinaykumar requested a review from chandrams June 27, 2023 15:25

dinogun modified the milestones: Kruize 0.0.17_rm Release, Kruize 0.0.19_rm Release, Kruize 0.0.18_rm Release Jul 7, 2023

chandrams reviewed Jul 13, 2023

View reviewed changes

msvinaykumar added analyzer Analyzer module related issues API Requires API Changes labels Jul 25, 2023

chandrams reviewed Aug 3, 2023

View reviewed changes

chandrams reviewed Aug 9, 2023

View reviewed changes

chandrams reviewed Aug 10, 2023

View reviewed changes

dinogun mentioned this pull request Aug 12, 2023

updateRecommendation API. #825

Merged

msvinaykumar added 5 commits August 16, 2023 20:53

Added progress prints

0d8824a

Signed-off-by: msvinaykumar <[email protected]>

minor fix

c1bed78

Signed-off-by: msvinaykumar <[email protected]>

Included updateRecommendation API call into e2e module'

c2e4ab6

Signed-off-by: msvinaykumar <[email protected]>

included sanity and negeative testcases

dbaa5ac

Signed-off-by: msvinaykumar <[email protected]>

msvinaykumar added 5 commits August 16, 2023 20:53

incoporated review comments

8f2c905

Signed-off-by: msvinaykumar <[email protected]>

incroporated review comments

97cdbde

Signed-off-by: msvinaykumar <[email protected]>

invalid date tc added

3d498ca

Signed-off-by: msvinaykumar <[email protected]>

fixed failed tc

199dbe7

Signed-off-by: msvinaykumar <[email protected]>

Bulk update results test script

d85e09d

Signed-off-by: msvinaykumar <[email protected]>

syntax error and bulk script wrapper

3184fae

Signed-off-by: msvinaykumar <[email protected]>

msvinaykumar force-pushed the quicktest2 branch from a9f085c to 3184fae Compare August 22, 2023 02:37

renamed files

8b7f02c

Signed-off-by: msvinaykumar <[email protected]>

dinogun approved these changes Aug 22, 2023

View reviewed changes

dinogun merged commit cc988f5 into kruize:mvp_demo Aug 22, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Test script for updateRecommendation API #836

Test script for updateRecommendation API #836

msvinaykumar commented May 29, 2023 •

edited

Loading

chandrams commented Jun 15, 2023

chandrams Jul 13, 2023

chandrams Aug 2, 2023

msvinaykumar Aug 2, 2023

chandrams Jul 13, 2023

msvinaykumar Aug 2, 2023

chandrams Jul 13, 2023

msvinaykumar Aug 2, 2023

chandrams Jul 13, 2023

msvinaykumar Aug 2, 2023

chandrams Jul 13, 2023

msvinaykumar Aug 2, 2023

chandrams Jul 13, 2023

chandrams Jul 13, 2023

chandrams Jul 18, 2023

msvinaykumar Aug 2, 2023

msvinaykumar Aug 2, 2023

msvinaykumar commented Aug 2, 2023

chandrams Aug 2, 2023

chandrams Aug 2, 2023

chandrams Aug 2, 2023

chandrams Aug 2, 2023

chandrams Aug 3, 2023

chandrams Aug 3, 2023

chandrams Aug 9, 2023

chandrams commented Aug 10, 2023

chandrams commented Aug 10, 2023

chandrams Aug 10, 2023

dinogun commented Aug 12, 2023

chandrams commented Aug 17, 2023

dinogun left a comment

Test script for updateRecommendation API #836

Test script for updateRecommendation API #836

Conversation

msvinaykumar commented May 29, 2023 • edited Loading

chandrams commented Jun 15, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

msvinaykumar commented Aug 2, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chandrams commented Aug 10, 2023

chandrams commented Aug 10, 2023

Choose a reason for hiding this comment

dinogun commented Aug 12, 2023

chandrams commented Aug 17, 2023

dinogun left a comment

Choose a reason for hiding this comment

msvinaykumar commented May 29, 2023 •

edited

Loading