Manual testing of Release 2022.11

[alisevych]

Initial set-up

Check that the IntelliJ Idea UTBot plugin can be successfully installed

• Build from the unit-test-bot/rc112022 branch
• Download plugin
• Check downloaded zip-file size < 100 MB
• Open IntelliJ IDEA 2022.2.3 (last released)
• Remove previously installed UTBot plugin
• Clone or reuse UTBot project (https://github.com/UnitTestBot/UTBotJava.git)
• Open the project in the IDE
• Install the downloaded plugin

Go through manual scenarios

Manual scenario #1

• Use default plugin settings
• Open the utbot-sample/src/main/java/org/utbot/examples/algorithms/ArraysQuickSort.java file
• Generate tests for the class
• Remove results
• Generate and Run test for a method

Manual scenario #2

• Use default plugin settings
• Open the utbot-sample/src/main/java/org/utbot/examples/mock/CommonMocksExample.java file
• Generate tests with different Mocking options combinations

Regression checks

Collect too complex reflection in tests

Add a class / package - generate / generate and run tests for class / method / class - check tests

• Installed on 2022.1.4 - Gradle project
• Installed on 2022.2.3 - Gradle project
• Installed on 2022.1.4 - Maven project
• Installed on 2022.2.3 - Maven project
• Installed on 2022.1.4 - IntelliJ project
• Installed on 2022.2.3 - IntelliJ project

Add a class / package - generate tests for class / method / class

@alisevych

• Gradle - existing test sources root - No mocks
• Maven - existing test sources root - Mocks outside package
• Maven - new test sources root - Mocks outside class
• IntelliJ - new test sources root - Mocks outside class - Mock static methods

Add a class / package - generate tests for class / method / class

• JDK 8 - JUnit 5 - Parametrized
• JDK 8 - JUnit 4 - Parametrized off
• JDK 8 - TestNG - Parametrized off
• JDK 11 - JUnit 5 - Parametrized
• JDK 11 - JUnit 4 - Parametrized off @tyuldashev
• JDK 11 - TestNG - Parametrized
• JDK 17 - JUnit 5 - Parametrized
• JDK 17 - JUnit 4 - Parametrized off
• JDK 17 - TestNG - Parametrized off

UI checks - in a project with JDK 11

• timeout changes - 1/10/60 sec - check with stopwatch @tyuldashev
• cancellation - check on different steps @tyuldashev
• regions: Symbolic first, Fuzzing,
• regions: error suite (NPE), explicitly thrown exceptions, caught exceptions (successful)
• exceptions - check Problems tab @alisevych
• context menu / shortcut
• call from method / class / package - check Members list
• call for several classes - Generate and Run - check all test classes are run

Check Settings - in a project with JDK 17

• Symbolic 100%
• Symbolic 95%
• Symbolic 5%
• Fuzzing 100%
• Summaries with custom tags - check all tags
• Summaries with plain text - check for Fuzzing
• Summaries turned off
• Tests with exceptions: Fail
• Tests with exceptions: Pass
• Overflow : Ignore overflows
• Overflow: Treat overflows as errors

Other features testing

• UtUtils class generation - Manual testing of UtUtils class generation - build#2022.11.502. #1233
• Sandboxing - check default behavior
• Sandboxing - add some permission
• Sandboxing - add incorrect permission
• Strings
• Collections

[alisevych]

02/11/2022
Started testing on build from main

Verified fixes

• Generated test output size should be customizable #1235
• StringExamples generation exceeds timeout in 2-3 times #1141
• IDEA hangs during test generation #1207
• @utbot.triggersRecursion tag is missing #1014

Found new bugs:

• IllegalStateException is thrown in IDEA together with notification that file exceeds configured limit #1295
• Setting maxTestsPerMethod is ignored #1297

03/11/2022
Continue with build from release candidate branch - see the first check-list

[alisevych]

New bugs:

• !!! OutOfMemoryError in Concrete Executor logs and error test #1292
• !!! Socket.connect and Files.readAllBytes produce errors in Concrete execution #1314
• Regions names for parameterized tests #1301
• Test sources root for IntelliJ or Maven project can be created automatically #1302
• TestNG Parametrization doesn't work for Symbolic execution #1304
• Fuzzing mode with JDK 13/17 CommonMocksExample.nextValue. 343 errors in Concrete Executor #1305
• Test generation is not canceled when user tries to stop it at building phase #1306
• Don't show error message when user explicitly cancel test generation #1307
• JvmCrashExamples Symbolic + Fuzzing mode produces errors with JDK 17 #1308
• Logs of Concrete Executor should contain Exception with stack trace #1287 (Testability)
• Summaries disabled in config file is not applied #1321

[alisevych]

Reproducing:

• Empty region names and missing javadoc for Sort.arrayCopy #1204
• !!! Server.EngineProcessModel.generateTestReport is not bound to a protocol #1183
• Incorrect links in Java docs for classes without packages #1174
• Test with comment 'This test fails because ... java.lang.ArithmeticException' does not fail #1272
• Test generation results are eliminated after cancel #1234
• HashCode of a mocked object is used in path constraints #286

[alisevych]

Cherry-picked this commit:
https://github.com/UnitTestBot/UTBotJava/pull/1298/files

Additional checklist:

@tyuldashev

IntelliJ IDEA 2022.1.4

• Summaries turned OFF for Kotlin user code
• + check tests generated
• Summaries turned OFF for Java user code - Kotlin generation language selected
• + check tests generated
• check in existing project
• check in new project

@alisevych

IntelliJ IDEA 2022.2.3

• Summaries turned ON for Java - Java
• + check tests generated with Java docs tags
• + check tests generated with Java doc plain text
• + check tests generated with Summaries turned off
• check in existing project
• check in new project

Additional checks

• Sandboxing - check annotation Disabled is correct for Summaries on / off
• Check enableSummariesGeneration = true / false with settings.properties