Added new ID tests and README file for verify_test_type #1555

yugvajani · 2024-12-02T16:45:03Z

The tests are ID tests detected by NonDex.
Java Version - 17.0.13
Maven Version - 3.9.6

Ran the nondex command and the test command without nondex on the repository to find the new flaky tests.

The logs for the tests can be found at /home/yvajani2/openhab-core/openhab.txt

Have also added a README.md file which states the steps to run the verify_test_type.sh script locally.

darko-marinov · 2024-12-02T17:08:49Z

Did you run tests individually to confirm them? Can you check how https://github.com/TestingResearchIllinois/idoft/tree/main/verify_test_type can help you confirm the type?

yugvajani · 2024-12-02T17:13:22Z

I ran the mvn test -Dtest and the mvn edu.illinois:nondex-maven-plugin:2.1.7:nondex -Dtest for each test individually. The logs for it can be found here /home/yvajani2/openhab-core/openhab.txt

darko-marinov · 2024-12-02T17:14:39Z

Okay, great. Now try that verify_test_type and see if you need to improve its code or documentation.

darko-marinov · 2024-12-02T17:16:14Z

You can run the checker locally before pushing some changes. You need not repeatedly find these failures on the CI.

yugvajani · 2024-12-02T17:17:08Z

Yes now i will do it locally only thanks.

darko-marinov · 2024-12-03T03:38:58Z

The checker passes now, thanks. What does verify_test_type report?

yugvajani · 2024-12-03T05:02:49Z

I ran the verify_test_type command individually for all the tests. Below are the logs for each set of tests:

Tests 1-5 - /home/yvajani2/openhab-core/verify_test.txt
Tests 6-10 - /home/yvajani2/openhab-core/verify_test_1.txt
Tests 11-15 - /home/yvajani2/openhab-core/verify_test_2.txt
Tests 16-20 - /home/yvajani2/openhab-core/verify_test_3.txt
Tests 21-26 - /home/yvajani2/openhab-core/verify_test_4.txt

darko-marinov

Please address the comment on README

darko-marinov · 2024-12-04T05:51:09Z

verify_test_type/README.md

+
+Go to the root directory of the repository which has the newly found flaky test.
+
+#### 2. Download the script in your repository


Why not just git clone ...idoft, especially if you're already changing pr-data.csv or other files in the repo?

We only need the script file and not the entire repository so its better to just add the verify_test script to the repo.

But installing wget ends up more challenging than just git clone. Also, why include here instructions for downloading wget when people can download that file/script in any other way they want.

I will remove the wget and chmod commands, and instead mentioning that users can download that file/script in any way they want.

darko-marinov · 2024-12-04T05:51:28Z

verify_test_type/README.md

+Run the script for a particular test using this command:
+
+```
+./verify_test_type.sh <module-name> <commit-hashcode> <test-name> <class-name>


Can you add some example invocation?

Added an example of invocation

darko-marinov

Please see more reviews on the README now that I read it in detail.

darko-marinov · 2024-12-04T17:31:00Z

verify_test_type/README.md

@@ -0,0 +1,43 @@
+<h1 align="center">Verify Test Type</h1>


Why use HTML tags in a Markdown?

Since this was used for certain scripts in idoft for example in https://github.com/TestingResearchIllinois/idoft/blob/main/format_checker/README.md. Tried to make README consistent.

Change those READMEs (likely in a different PR).

So should i remove the html tags or not from the markdown in this README file?

Yes, remove (rather replace with # and such) in this file in this PR. Then open another PR to remove from the other README files.

darko-marinov · 2024-12-04T17:31:13Z

verify_test_type/README.md

@@ -0,0 +1,43 @@
+<h1 align="center">Verify Test Type</h1>
+
+<p align="center">Script Purpose: Ensure the identification of the test type (e.g., ID, OD, NOD, etc.) for new tests discovered in a repository. </p>


Does Markdown support centering text?

darko-marinov · 2024-12-04T17:32:39Z

verify_test_type/README.md

+
+Go to the root directory of the repository which has the newly found flaky test.
+
+#### 2. Download the script in your repository


But installing wget ends up more challenging than just git clone. Also, why include here instructions for downloading wget when people can download that file/script in any other way they want.

darko-marinov · 2024-12-04T17:33:22Z

verify_test_type/README.md

+
+#### 3. Make the script executable
+
+Use this command to make the script executable:


Wouldn't need this for git clone (in fact, shouldn't need it if the file has the right permission in the repo, and if the file doesn't have, just open a PR for that).

darko-marinov · 2024-12-04T17:34:46Z

verify_test_type/README.md

+
+Example of an invocation:
+```
+./verify_test_type.sh bundles/org.openhab.core.thing 660102e3f93f928473b66f56f2955090dfa3c30e testCreateChannelDescriptionChangedEventOnlyNewValue ThingEventFactoryTest


Since you're provided repo and commit SHA here, what ensures that the exact commit is checked? Does the script clone the repo? Or does it assume that you've already cloned it? If the latter, does it check that the commit SHA matches git rev-parse HEAD?

The repository is forcefully reset to the exact commit SHA specified by $commit_hash. This guarantees that all operations performed by the script will use this specific state of the repository. It assumes the repository is already cloned.

Good description. Please add it in README (on in the script), because the users won't be reading this PR discussion.

yugvajani · 2024-12-04T18:41:13Z

I have updated the README with the description of script and have removed the HTML tags from it

darko-marinov · 2024-12-04T18:44:17Z

Okay, let's declare victory for this PR, so you can review #1554

yugvajani force-pushed the newtests branch from ce24cad to ffc6a22 Compare December 2, 2024 17:08

yugvajani force-pushed the newtests branch from ffc6a22 to eb91afa Compare December 2, 2024 17:14

yugvajani force-pushed the newtests branch from eb91afa to 0d71658 Compare December 2, 2024 21:14

yugvajani force-pushed the newtests branch from 0d71658 to 6e3fa80 Compare December 4, 2024 01:01

yugvajani changed the title ~~Added new ID tests~~ Added new ID tests and README file for verify_test_type Dec 4, 2024

darko-marinov requested changes Dec 4, 2024

View reviewed changes

darko-marinov mentioned this pull request Dec 4, 2024

Add ID Tests #1554

Merged

yugvajani force-pushed the newtests branch from 6e3fa80 to ff4023e Compare December 4, 2024 09:22

yugvajani requested a review from darko-marinov December 4, 2024 16:41

darko-marinov requested changes Dec 4, 2024

View reviewed changes

yugvajani force-pushed the newtests branch from ff4023e to 4fecd69 Compare December 4, 2024 18:05

yugvajani requested a review from darko-marinov December 4, 2024 18:05

Added new ID tests and a readme file for running the verify type script

ddb575e

yugvajani force-pushed the newtests branch from 4fecd69 to ddb575e Compare December 4, 2024 18:40

darko-marinov merged commit f7aa809 into TestingResearchIllinois:main Dec 4, 2024
1 check passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added new ID tests and README file for verify_test_type #1555

Added new ID tests and README file for verify_test_type #1555

yugvajani commented Dec 2, 2024 •

edited

Loading

darko-marinov commented Dec 2, 2024

yugvajani commented Dec 2, 2024

darko-marinov commented Dec 2, 2024

darko-marinov commented Dec 2, 2024

yugvajani commented Dec 2, 2024

darko-marinov commented Dec 3, 2024

yugvajani commented Dec 3, 2024

darko-marinov left a comment

darko-marinov Dec 4, 2024

yugvajani Dec 4, 2024

darko-marinov Dec 4, 2024

yugvajani Dec 4, 2024

darko-marinov Dec 4, 2024

yugvajani Dec 4, 2024

darko-marinov left a comment

darko-marinov Dec 4, 2024

yugvajani Dec 4, 2024

darko-marinov Dec 4, 2024

yugvajani Dec 4, 2024

darko-marinov Dec 4, 2024

darko-marinov Dec 4, 2024

yugvajani Dec 4, 2024

darko-marinov Dec 4, 2024

darko-marinov Dec 4, 2024

darko-marinov Dec 4, 2024

yugvajani Dec 4, 2024

darko-marinov Dec 4, 2024

yugvajani commented Dec 4, 2024

darko-marinov commented Dec 4, 2024


		Go to the root directory of the repository which has the newly found flaky test.

		#### 2. Download the script in your repository

		@@ -0,0 +1,43 @@
		<h1 align="center">Verify Test Type</h1>

		<p align="center">Script Purpose: Ensure the identification of the test type (e.g., ID, OD, NOD, etc.) for new tests discovered in a repository. </p>


		#### 3. Make the script executable

		Use this command to make the script executable:

Added new ID tests and README file for verify_test_type #1555

Added new ID tests and README file for verify_test_type #1555

Conversation

yugvajani commented Dec 2, 2024 • edited Loading

darko-marinov commented Dec 2, 2024

yugvajani commented Dec 2, 2024

darko-marinov commented Dec 2, 2024

darko-marinov commented Dec 2, 2024

yugvajani commented Dec 2, 2024

darko-marinov commented Dec 3, 2024

yugvajani commented Dec 3, 2024

darko-marinov left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

darko-marinov left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yugvajani commented Dec 4, 2024

darko-marinov commented Dec 4, 2024

yugvajani commented Dec 2, 2024 •

edited

Loading