Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Performance tests #121

Merged
merged 21 commits into from
Nov 1, 2023
Merged

Performance tests #121

Changes from 1 commit
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
4a11293
add config case class
ailrst Oct 23, 2023
805e596
add IL to expected
ailrst Oct 23, 2023
4fc09f7
fix add IL updateExpected
ailrst Oct 23, 2023
7477bbd
update gitignore
ailrst Oct 23, 2023
81fa6f0
add IL expected files
ailrst Oct 23, 2023
c953a7e
add csv test summary and timing
ailrst Oct 23, 2023
b2e6406
add test summary
ailrst Oct 23, 2023
37c0661
and condition on store generator
ailrst Oct 23, 2023
d1b558c
Change memory updates > 8 bits to use lambdas
ncough Oct 24, 2023
13a5186
add lambda mem flag
ailrst Oct 24, 2023
a1b4d37
distinguish timeout from failure, add summary test
ailrst Oct 24, 2023
277f167
update gitignore
ailrst Oct 24, 2023
ff0049a
revert defaults
ailrst Oct 24, 2023
55b1911
add bigendian support for lambda stores
ailrst Oct 24, 2023
1a358bc
always consider timeout a failure
ailrst Oct 25, 2023
7370c6a
use list for results
ailrst Nov 1, 2023
71624fb
do not output IL.expected
ailrst Nov 1, 2023
51d94f7
remove il.expected files
ailrst Nov 1, 2023
23a7b80
Merge branch 'main' into performance-tests
Nov 1, 2023
cbf547d
make summary give more useful information
Nov 1, 2023
e9b5b0a
add num verify/failed to summary
ailrst Nov 1, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions src/test/scala/SystemTests.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,20 +53,20 @@ class SystemTests extends AnyFunSuite {
val csv: String = "testCase," + TestResult.csvHeader + System.lineSeparator() + testResults.map(r => s"${r._1},${r._2.toCsv}").mkString(System.lineSeparator())
log(csv, testPath + "testResults.csv")

val numVerified = testResults.count(_._2.verified)
val numNotVerified = testResults.count(!_._2.verified)
val numSuccess = testResults.count(_._2.passed)
val numFail = testResults.count(!_._2.passed)
val numTimeout = testResults.count(_._2.timedOut)
val verifying = testResults.filter(x => !x._2.timedOut && x._2.verified).map(_._2.verifyTime)
val counterExamples = testResults.filter(x => !x._2.timedOut && !x._2.verified).map(_._2.verifyTime)

info(s"Test summary: $numVerified verified, $numNotVerified failed to verify including $numTimeout timeouts.")
info(s"Test summary: $numSuccess succeeded, $numFail failed (including $numTimeout timeouts).")
if (verifying.nonEmpty)
info(s"Average time to verify: ${verifying.sum / verifying.size}")
if (counterExamples.nonEmpty)
info(s"Average time to counterexample: ${counterExamples.sum/ counterExamples.size}")

val summaryHeader = "verifiedCount,counterexampleCount,timeoutCount,verifyTotalTime,counterexampleTotalTime"
val summaryRow = s"$numVerified,${counterExamples.size},$numTimeout,${verifying.sum},${counterExamples.sum}"
val summaryRow = s"$numSuccess,${counterExamples.size},$numTimeout,${verifying.sum},${counterExamples.sum}"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wanted verifyTotalTime/verifyCount = mean verify time. scalaTest already tells you how many tests passed.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll just make the summary include both things.

log(summaryHeader + System.lineSeparator() + summaryRow, testPath + "summary.csv")
}

Expand Down