Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Manual testing of Release 2022.10.1 #1125

Closed
36 tasks done
alisevych opened this issue Oct 10, 2022 · 15 comments
Closed
36 tasks done

Manual testing of Release 2022.10.1 #1125

alisevych opened this issue Oct 10, 2022 · 15 comments
Assignees
@alisevych
Labels
ctg-qa Issue is related to QA process
Milestone
2022.10.1 Release

Comments

@alisevych
Copy link
Member

alisevych commented Oct 10, 2022 •
edited
Loading

Initial set-up

Check that the IntelliJ Idea UTBot plugin can be successfully installed

Go through manual scenarios

Manual scenario #1

  • Use default plugin settings
  • Open the utbot-sample/src/main/java/org/utbot/examples/algorithms/ArraysQuickSort.java file
  • Generate tests for the class
  • Remove results
  • Generate and Run test for a method

Manual scenario #2

  • Use default plugin settings
  • Open the utbot-sample/src/main/java/org/utbot/examples/mock/CommonMocksExample.java file
  • Generate tests with different Mocking options combinations

Manual scenario #3

  • Create a new Gradle project with JDK 17
  • Add a simple java file to test
  • Generate a test in the existing test root

Manual scenario #4

  • Create a new Maven project with JDK 8
  • Add a simple java file to test
  • Generate a test with a new test root

Manual scenario #5

  • Create a new Idea project with JDK 11
  • Add a simple java file to test
  • Generate tests for several classes

Regression testing

  • IDEA 2022.1.4 (JRE 11)
  • IDEA 2022.2.2 (JRE 17)
  • Sandbox
  • Nested calls
  • Recursion
  • Cycles
  • Mocks
  • UtUtils class generation
  • Summaries: tags, plain text
  • Summaries: regions
  • Exceptions: thrown, caught, check regions
@alisevych alisevych added the ctg-qa Issue is related to QA process label Oct 10, 2022
@alisevych alisevych self-assigned this Oct 10, 2022
@alisevych
Copy link
Member Author

alisevych commented Oct 10, 2022 •
edited
Loading

reproducing #1028
reproducing #286
reproducing #747

@alisevych alisevych added this to the 2022.10.1 Release milestone Oct 10, 2022
@korifey korifey moved this to Todo in UTBot Java Oct 10, 2022
@alisevych
Copy link
Member Author

alisevych commented Oct 10, 2022 •
edited
Loading

new bug with Sandboxing and Fuzzing - many tests with same coverage are generated and no minimized
#1129

@alisevych alisevych mentioned this issue Oct 10, 2022
Closed
@alisevych
Copy link
Member Author

alisevych commented Oct 10, 2022 •
edited
Loading

new bug with Recursion - empty test class and
java.lang.Exception
at com.jetbrains.rd.util.LoggerKt.defaultLogFormat(Logger.kt:80)
at org.utbot.rd.loggers.UtRdConsoleLogger.log(UtRdConsoleLogger.kt:24)
at com.jetbrains.rd.util.SwitchLogger.log(Logger.kt:47)
at org.utbot.instrumentation.instrumentation.et.RuntimeTraceStorage.visit(TraceHandler.kt:162)
at org.utbot.examples.recursion.Recursion.fib(Recursion.java:21)
at org.utbot.examples.recursion.Recursion.fib(Recursion.java:22)
at org.utbot.examples.recursion.Recursion.fib(Recursion.java:22)
...
at org.utbot.instrumentation.instrumentation.InvokeInstrumentation$invoke$2$result$1.invoke-IoAF18A(InvokeInstrumentation.kt:60)
#1130 : reproduced once with Recursion, not reproducing with Cycles

@alisevych alisevych mentioned this issue Oct 10, 2022
Closed
@alisevych
Copy link
Member Author

alisevych commented Oct 10, 2022 •
edited
Loading

Reopened #273 - please see the comment with stack trace

@alisevych
Copy link
Member Author

reproducing #711 - static field and static method

@alisevych
Copy link
Member Author

alisevych commented Oct 11, 2022 •
edited
Loading

new bug : sandboxed test is not compiling because commented stack trace contains "illegal unicode escape" in "jkmqh\uhauh"
#1134

@alisevych alisevych mentioned this issue Oct 11, 2022
Closed
@alisevych
Copy link
Member Author

reproducing #1135

@alisevych
Copy link
Member Author

alisevych commented Oct 11, 2022 •
edited
Loading

clarification task for Summaries:
regions with Explicitly thrown exception region #1133
to be discussed and corrected

@alisevych
Copy link
Member Author

reproducing #1141

@alisevych
Copy link
Member Author

reproducing #1098 - on different code, when no tests are generated (timeout is minimal for example)

@alisevych
Copy link
Member Author

alisevych commented Oct 12, 2022 •
edited
Loading

proceeding with RC testing of build#430

Check with different settings

Check other parts

  • CLI, JDK 11
  • CLI, JDK 17

@alisevych alisevych changed the title Manual testing of RC build#417 Manual testing of RC build#417 - RC build#434 Oct 12, 2022
@alisevych
Copy link
Member Author

alisevych commented Oct 13, 2022 •
edited
Loading

proceeding with RC testing of build#434

Check with different settings - in IDEA 2022.2.3 (new version 05-Oct-2022)

Check other parts

  • CLI, JDK 11
  • CLI, JDK 17
  • CLI, JDK 8

Environmental checks

  • macOs, last IDEA, JDK 17,
  • IDEA 2022.2.3 (new version 05-Oct-2022)

@alisevych alisevych changed the title Manual testing of RC build#417 - RC build#434 Manual testing of RC build#417 - RC build#430 - RC build#434 Oct 13, 2022
@alisevych alisevych changed the title Manual testing of RC build#417 - RC build#430 - RC build#434 Manual testing of RC Oct 13, 2022
@alisevych
Copy link
Member Author

alisevych commented Oct 13, 2022 •
edited
Loading

new bug #1149 - reproducing on latest main:
Generate and Run for several classes: tests are generated correctly, but run is executed only for one test class.

@alisevych
Copy link
Member Author

alisevych commented Oct 13, 2022 •
edited
Loading

new bug: #1160
IDEA dead lock -trying to save SARIF report - when resources folder is removed before generations

@alisevych
Copy link
Member Author

alisevych commented Oct 14, 2022 •
edited
Loading

new bug: test generated by Symbolic engine for StringExamples.isNotBlank has long string with 6000+ "\u0000"
#1157

This was referenced Oct 14, 2022
Closed
Closed
@alisevych alisevych changed the title Manual testing of RC Manual testing of RC 2022.10.1 Oct 14, 2022
Repository owner moved this from Todo to Done in UTBot Java Oct 14, 2022
@alisevych alisevych changed the title Manual testing of RC 2022.10.1 Manual testing of Release 2022.10.1 Oct 19, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@alisevych alisevych

Labels
ctg-qa Issue is related to QA process
Projects
UTBot Java
Archived in project
Milestone
2022.10.1 Release
Development

No branches or pull requests
1 participant
@alisevych