Seed management to improve reproducibility #1380
Conversation
| ) | ||
| ( | ||
| len in Just(len), | ||
| tv_bits in Just(tv_bits), | ||
| inputs in prop::collection::vec(prop::array::uniform(0u32..1 << tv_bits), len), | ||
| ) | ||
| -> AggregatePropTestInputs { | ||
| let mut rng = StdRng::seed_from_u64(seed); | ||
| let mut expected = vec![0; PROP_BUCKETS]; | ||
| let inputs = repeat_with(|| { | ||
| let row: [u32; PROP_BUCKETS] = array::from_fn(|_| rng.gen_range(0..1 << tv_bits)); | ||
| for row in &inputs { | ||
| for (exp, val) in expected.iter_mut().zip(row) { | ||
| *exp = min(*exp + val, (1 << PropHistogramValue::BITS) - 1); | ||
| } | ||
| row | ||
| }) | ||
| .take(len) | ||
| .collect(); | ||
| } |
There was a problem hiding this comment.
These changes are not essential to this PR, but it seemed clearer not to have two different seeds, and using proptest for input generation enables using its other capabilities like shrinking.
ipa-core/src/lib.rs
Outdated
| // iterations (or to make the API use `Fn` to match the shuttle version), the | ||
| // simplest strategy might be to seed per-iteration RNGs from a primary RNG, like | ||
| // `TestWorld::rng`. | ||
|
|
ipa-core/src/test_fixture/world.rs
Outdated
| @@ -190,6 +200,13 @@ impl Default for TestWorld { | |||
| } | |||
| } | |||
|
|
|||
| impl TestWorld { | |||
| #[must_use] | |||
| pub fn with_seed(seed: u64) -> Self { | |||
There was a problem hiding this comment.
Default values for generic parameters is a blessing and a curse.
impl TestWorld<NotSharded> that exists on the line 266 is probably a better place for this fn.
There was a problem hiding this comment.
Is there a reason to implement Default only for TestWorld<NotSharded>?
There was a problem hiding this comment.
backward compatibility was the only one - I didn't want to change a bunch of non-sharded tests when introducing the in-memory sharding infrastructure
|
Thank you for making this change @andyleiserson, we are getting closer to fully reproducible tests with it |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #1380 +/- ##
==========================================
+ Coverage 93.54% 93.57% +0.03%
==========================================
Files 222 223 +1
Lines 36986 37117 +131
==========================================
+ Hits 34600 34734 +134
+ Misses 2386 2383 -3 ☔ View full report in Codecov by Sentry. |
Changes to tests and test infrastructure to be able to reproduce test runs from a seed.
This just updates one example of each kind of test I could think of to demonstrate how to achieve reproducibility. There are still many other tests that would need to be updated to match.
Fixes #1321