feat(bloom): store running aggregate in memory #2367
Conversation
Yeah, it seems to slow down our startup time by a few seconds on average. If that's too much, a good middle ground might be to store the running aggregate filter, but do it less often than the filter's entire range. In other words: aggregate filters cover 32k blocks each, and we can store the running aggregate filter for example after every 8k or 4k blocks. This is a middle ground between the approach implemented by this PR and the approach which was in place previously (and actually thinking about it, this might be really easy to implement). I'll let you know exactly how long the reconstruction takes in the worst possible case (when 32k - 1 blocks need to be reconstructed). |
I think a few seconds should be totally fine. |
| let running_aggregate = match self.running_aggregate.lock() { | ||
| Ok(guard) => guard, | ||
| Err(poisoned) => { | ||
| tracing::error!("Poisoned lock in load_aggregate_bloom_range"); | ||
| poisoned.into_inner() | ||
| } | ||
| }; |
There was a problem hiding this comment.
We don't have a uniform way of locking mutexes in the code and it's high time we had one. 🤔
sistemd
force-pushed
the
sistemd/running-aggregate-in-memory
branch
from
1a3205d
to
6f74d53
Compare
Closes #2354.
Closes #2355.