Metrics #23
Merged
Metrics #23
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 and MIT licenses. --------- Co-authored-by: Cole Vick <[email protected]> Co-authored-by: Michael Tautschnig <[email protected]> Co-authored-by: Carolyn Zech <[email protected]>
Shard Kani verification workflow across multiple runners by:
1. Running `kani list --format json`, which outputs a JSON file like
this:
```json
{
"kani-version": "0.56.0",
"file-version": "0.1",
"standard-harnesses": {
"src/lib.rs": [
"proof3",
"verify::proof2"
],
"src/test.rs": [
"test::proof4",
"test::proof5"
]
},
"contract-harnesses": {
"src/lib.rs": [
"proof"
]
},
"contracts": [
{
"function": "bar",
"file": "src/lib.rs",
"harnesses": [
"proof"
]
}
],
"totals": {
"standard-harnesses": 4,
"contract-harnesses": 1,
"functions-under-contract": 1
}
}
```
2. Extracting the harnesses inside `"standard-harnesses"` and
`"contract-harnesses"` into an array called `ALL_HARNESSES` and the
length of that array into `HARNESS_COUNT`. (So in this example,
`ALL_HARNESSES = [proof3, verify::proof2, test::proof4, test::proof5,
proof]` and `HARNESS_COUNT=5`).
3. Dividing the harnesses evenly between four workers. For example, if
worker 1's harnesses are `proof3` and `verify::proof2`, then we call
`kani verify-std --harness proof3 --harness verify::proof2 --exact`. The
`--exact` makes Kani look for the exact harness name. This is important
so that we don't match on partial patterns, e.g., if there is a harness
called "foo" and a harness called "foo_bar", passing `--harness foo`
without `--exact` would match against both harnesses, and then `foo_bar`
would run twice.
4. Also parallelize verification within a single runner by passing `-j`
to Kani.
I chose four workers somewhat arbitrarily--it makes each worker take
about 45 minutes to an hour to finish. I thought it was good to have a
balance between too few workers (which still makes us wait a while) and
too many workers (which makes you look through more logs to find where a
given harness is being verified). But happy to play with this number if
people have opinions.
By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 and MIT licenses.
This harness takes between 22-25 minutes in CI. Change the harnesses's macro to use kissat solver instead, which brings verification time down to ~7 seconds. As an aside, I noticed this problem because the partition 1 runner in model-checking#229 is taking ~55 minutes, where the other partitions are taking ~30 minutes. This harness accounts for almost the entire difference. One nice consequence of partitioning verification is that it will make performance issues like this more noticeable. By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 and MIT licenses.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Merge metrics into main so that I can test the pull request workflow
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 and MIT licenses.