Skip to content
bench

Add SDK #284

Sign in to view logs

Add SDK

Add SDK #284

Triggered via pull request September 29, 2024 23:28
@hinthornwhinthornw
synchronize #1892
wfh/store_sdk
Status Success
Total duration 45m 6s
Artifacts –

bench.yml

on: pull_request
benchmark
44m 56s
benchmark
Fit to window
Zoom out
Zoom in

Annotations

1 warning and 2 notices
benchmark
The following actions use a deprecated Node.js version and will be forced to run on node20: actions/cache@v3. For more info: https://github.blog/changelog/2024-03-07-github-actions-all-actions-will-run-on-node20-instead-of-node16-by-default/
Benchmark results: libs/checkpoint/langgraph/store/base.py#L1
......................................... fanout_to_subgraph_10x: Mean +- std dev: 58.8 ms +- 1.3 ms ......................................... WARNING: the benchmark result may be unstable * the standard deviation (7.13 ms) is 12% of the mean (57.1 ms) Try to rerun the benchmark with more runs, values and/or loops. Run 'python -m pyperf system tune' command to reduce the system jitter. Use pyperf stats, pyperf dump and pyperf hist to analyze results. Use --quiet option to hide these warnings. fanout_to_subgraph_10x_sync: Mean +- std dev: 57.1 ms +- 7.1 ms ......................................... fanout_to_subgraph_10x_checkpoint: Mean +- std dev: 77.4 ms +- 1.8 ms ......................................... fanout_to_subgraph_10x_checkpoint_sync: Mean +- std dev: 82.4 ms +- 2.0 ms ......................................... fanout_to_subgraph_100x: Mean +- std dev: 567 ms +- 18 ms ......................................... fanout_to_subgraph_100x_sync: Mean +- std dev: 501 ms +- 6 ms ......................................... fanout_to_subgraph_100x_checkpoint: Mean +- std dev: 757 ms +- 20 ms ......................................... fanout_to_subgraph_100x_checkpoint_sync: Mean +- std dev: 787 ms +- 9 ms ......................................... react_agent_10x: Mean +- std dev: 41.7 ms +- 3.6 ms ......................................... react_agent_10x_sync: Mean +- std dev: 30.0 ms +- 0.5 ms ......................................... react_agent_10x_checkpoint: Mean +- std dev: 52.9 ms +- 1.4 ms ......................................... react_agent_10x_checkpoint_sync: Mean +- std dev: 42.2 ms +- 3.1 ms ......................................... react_agent_100x: Mean +- std dev: 411 ms +- 7 ms ......................................... react_agent_100x_sync: Mean +- std dev: 329 ms +- 2 ms ......................................... react_agent_100x_checkpoint: Mean +- std dev: 914 ms +- 10 ms ......................................... react_agent_100x_checkpoint_sync: Mean +- std dev: 817 ms +- 10 ms ......................................... wide_state_25x300: Mean +- std dev: 20.4 ms +- 0.3 ms ......................................... wide_state_25x300_sync: Mean +- std dev: 12.8 ms +- 0.3 ms ......................................... wide_state_25x300_checkpoint: Mean +- std dev: 239 ms +- 8 ms ......................................... wide_state_25x300_checkpoint_sync: Mean +- std dev: 237 ms +- 15 ms ......................................... wide_state_15x600: Mean +- std dev: 23.7 ms +- 0.4 ms ......................................... wide_state_15x600_sync: Mean +- std dev: 14.7 ms +- 0.1 ms ......................................... wide_state_15x600_checkpoint: Mean +- std dev: 415 ms +- 11 ms ......................................... wide_state_15x600_checkpoint_sync: Mean +- std dev: 417 ms +- 19 ms ......................................... wide_state_9x1200: Mean +- std dev: 23.6 ms +- 0.4 ms ......................................... wide_state_9x1200_sync: Mean +- std dev: 14.7 ms +- 0.2 ms ......................................... wide_state_9x1200_checkpoint: Mean +- std dev: 266 ms +- 7 ms ......................................... wide_state_9x1200_checkpoint_sync: Mean +- std dev: 266 ms +- 16 ms
Comparison against main: libs/checkpoint/langgraph/store/base.py#L1
+-----------------------------------------+---------+-----------------------+ | Benchmark | main | changes | +=========================================+=========+=======================+ | fanout_to_subgraph_100x | 625 ms | 567 ms: 1.10x faster | +-----------------------------------------+---------+-----------------------+ | fanout_to_subgraph_10x_sync | 60.7 ms | 57.1 ms: 1.06x faster | +-----------------------------------------+---------+-----------------------+ | fanout_to_subgraph_100x_checkpoint | 791 ms | 757 ms: 1.04x faster | +-----------------------------------------+---------+-----------------------+ | fanout_to_subgraph_100x_sync | 522 ms | 501 ms: 1.04x faster | +-----------------------------------------+---------+-----------------------+ | fanout_to_subgraph_10x | 60.9 ms | 58.8 ms: 1.04x faster | +-----------------------------------------+---------+-----------------------+ | fanout_to_subgraph_10x_checkpoint | 80.0 ms | 77.4 ms: 1.03x faster | +-----------------------------------------+---------+-----------------------+ | react_agent_10x_checkpoint_sync | 43.3 ms | 42.2 ms: 1.02x faster | +-----------------------------------------+---------+-----------------------+ | fanout_to_subgraph_10x_checkpoint_sync | 84.3 ms | 82.4 ms: 1.02x faster | +-----------------------------------------+---------+-----------------------+ | wide_state_9x1200_checkpoint | 270 ms | 266 ms: 1.01x faster | +-----------------------------------------+---------+-----------------------+ | react_agent_10x_checkpoint | 53.4 ms | 52.9 ms: 1.01x faster | +-----------------------------------------+---------+-----------------------+ | react_agent_10x_sync | 30.2 ms | 30.0 ms: 1.01x faster | +-----------------------------------------+---------+-----------------------+ | react_agent_100x_checkpoint | 922 ms | 914 ms: 1.01x faster | +-----------------------------------------+---------+-----------------------+ | react_agent_100x_checkpoint_sync | 823 ms | 817 ms: 1.01x faster | +-----------------------------------------+---------+-----------------------+ | react_agent_100x | 414 ms | 411 ms: 1.01x faster | +-----------------------------------------+---------+-----------------------+ | react_agent_100x_sync | 330 ms | 329 ms: 1.00x faster | +-----------------------------------------+---------+-----------------------+ | wide_state_25x300_sync | 12.8 ms | 12.8 ms: 1.00x slower | +-----------------------------------------+---------+-----------------------+ | fanout_to_subgraph_100x_checkpoint_sync | 782 ms | 787 ms: 1.01x slower | +-----------------------------------------+---------+-----------------------+ | Geometric mean | (ref) | 1.01x faster | +-----------------------------------------+---------+-----------------------+ Benchmark hidden because not significant (11): wide_state_25x300_checkpoint, wide_state_25x300, react_agent_10x, wide_state_15x600_checkpoint, wide_state_9x1200_sync, wide_state_25x300_checkpoint_sync, wide_state_15x600_checkpoint_sync, wide_state_9x1200, wide_state_15x600, wide_state_9x1200_checkpoint_sync, wide_state_15x600_sync