Skip to content

Commit

Permalink
chore: added benchmarks documentation
Browse files Browse the repository at this point in the history
  • Loading branch information
Eagle941 committed Sep 18, 2024
1 parent e39ae0b commit 18a5e50
Show file tree
Hide file tree
Showing 2 changed files with 79 additions and 0 deletions.
79 changes: 79 additions & 0 deletions docs/BENCHMARK.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
# Addition of `VisitedPcs` trait

Current blockifier doesn't store the complete vector of visited program counters
for each entry point call in an invoke transaction. Instead, visited program
counters are pushed in a `HashSet`. This is a limiting factor to perform
profiling operations which require record of the full trace returned by
`cairo-vm`. More flexibility is added to the blockifier with the introduction of
trait `VisitedPcs` which allows the user to process visited program counters in
the most suitable way for the task.

## Existing code

Visited program counters are kept in the `CachedState` structure as shown below:

```rust
#[derive(Debug)]
pub struct CachedState<S: StateReader> {
pub state: S,
// Invariant: read/write access is managed by CachedState.
// Using interior mutability to update caches during `State`'s immutable getters.
pub(crate) cache: RefCell<StateCache>,
pub(crate) class_hash_to_class: RefCell<ContractClassMapping>,
/// A map from class hash to the set of PC values that were visited in the class.
pub visited_pcs: HashMap<ClassHash, HashSet<usize>>,
}
```

## New code

`VisitedPcs` is an additional generic parameter of `CachedState`.

```rust
#[derive(Debug)]
pub struct CachedState<S: StateReader, V: VisitedPcs> {
pub state: S,
// Invariant: read/write access is managed by CachedState.
// Using interior mutability to update caches during `State`'s immutable getters.
pub(crate) cache: RefCell<StateCache>,
pub(crate) class_hash_to_class: RefCell<ContractClassMapping>,
/// A map from class hash to the set of PC values that were visited in the class.
pub visited_pcs: V,
}
```

An implementation of the trait `VisitedPcs` is included in the blockifier with
the name `VisitedPcsSet` and it mimics the existing `HashSet<usize>`. Also, for
test purposes, `CachedState` is instantiated using `VisitedPcsSet`.

## Performance considerations

Given the importance of the blockifier in the Starknet ecosystem, we want to
measure the performance impact of adding the trait `VisitedPcs`. The existing
bechmark `transfers_benchmark` doesn't cover operations with `CachedState`
therefore we need to design new ones. We have created two new benchmarks:

- `cached_state_benchmark`: this benchmark tests the performance impact of
populating `visited_pcs` (implemented using `VisitedPcsSet`) with a realistic
amount of visited program counters. The size of the sets is taken from
transaction
`0x0177C9365875CAA840EA8F03F97B0E3A8EE8851A8B952BF157B5DBD4FECCB060` in the
mainnet. This transaction has been chosen randomly, but there is no assurance
that it's representative of the most common Starknet invoke transaction. This
benchmark tests the write performance of visited program counters in the state
struct.
- `execution_benchmark`: this benchmark simulates a whole invoke transaction
using a dummy contract.

## Performance impact

A script `bench.sh` has been added to benchmark the performance impact of these
changes: it is called as
`bash scripts/bench.sh 14e6a87722c1d0c757b1aa2756ffabe3f248fd7d e39ae0be4cec31938399199e0a1070279b4a78ed`.
The computer running the benchmark is: Debian VM over Windows 10 with VMWare
Workstation 17, i9-9900K, 64GB RAM, Samsung 990 Pro NVME SSD.

![Benchmark results](./criterion.png)

The analysis of criterion shows that there isn't statistically significant
performance decrese.
Binary file added docs/criterion.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 18a5e50

Please sign in to comment.