Streaming events #1808

ch1bo · 2025-01-28T14:27:20Z

Changes I made originally for the hydra-doom project to load all events in state (12GB+ in that use case) with constant memory using conduit streams.

There is a big TODO on this: IOSim s does not have a MonadUnliftIO instance and its not impossible to have one. We need to change the interface further such that we can compose createHydraNode and hydrate functions with in-memory EventSource variants.

A bit of a wart: The mkProjection does run the conduit for each projection, instead we should run the conduit once and build the projected in-memory read/query model once.

Also, the change here is not constant-memory as the ServerOutput history is still kept fully in memory. But this is a different story and should be covered by #1618

CHANGELOG updated or not needed
Documentation updated or not needed
Haddocks updated or not needed
No new TODOs introduced or explained herafter

…dles

This requires MonadUnliftIO on hydrate and createNode functions to run the conduits. However, the io-sim tests' IOSim s monad does not have an instance!

This re-loading the same server outputs 3 times though.

github-actions · 2025-01-28T16:17:39Z

Transaction cost differences

Script summary

Name	Size (Bytes)
νInitial	-
νCommit	-
νHead	-
μHead	-
νDeposit	-

`Init` transaction costs

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	-	-	-	-
2	-	-	-	-
3	-	-	-	-
5	-	-	-	-
10	-	-	-	-
44	-	-	-	-

`Commit` transaction costs

UTxO	Tx size	% max Mem	% max CPU	Min fee ₳
1	-	-	-	-
2	-	-	-	-
3	-	-	-	-
5	-	-	-	-
10	-	-	-	-
54	-	-	-	-

`CollectCom` transaction costs

Parties	UTxO (bytes)	Tx size	% max Mem	% max CPU	Min fee ₳
1	-	-	-	-	-
2	-	-	-	-	-
3	-	-	-	-	-
4	-	-	-	-	-
5	-	-	-	-	-
6	-	-	-	-	-
7	-	-	-	-	-
8	-	-	-	-	-
9	-	-	-	-	-
10	-	-	-	-	-

Cost of Increment Transaction

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	-	$${\color{green}-0.39}$$	$${\color{green}-0.09}$$	-
2	-	$${\color{green}-0.39}$$	$${\color{green}-0.09}$$	$${\color{green}-0.01}$$
3	-	-	-	-
5	-	+0.39	+0.09	-
10	-	-	-	-
43	-	-	-	-

Cost of Decrement Transaction

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	-	-	-	-
2	-	-	-	-
3	-	-	-	-
5	-	-	-	-
10	-	-	-	-
45	-	-	-	-

`Close` transaction costs

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	-	-	-	-
2	-	-	-	-
3	-	-	-	-
5	-	-	-	-
10	-	-	-	-
41	-	-	-	-

`Contest` transaction costs

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	-	-	-	-
2	-	-	-	-
3	-	-	-	-
5	-	-	-	-
10	-	-	-	-
33	-	-	-	-

`FanOut` transaction costs

UTxO, Parties	UTxO (bytes)	Tx size	% max Mem	% max CPU	Min fee ₳
(0, 10)	-	-	-	-	-
(1, 10)	-	-	-	-	-
(5, 10)	-	-	-	-	-
(10, 10)	-	-	-	-	-
(20, 10)	-	-	-	-	-
(40, 10)	-	-	-	-	-
(41, 10)	-	-	-	-	-

github-actions · 2025-01-28T16:17:48Z

Transaction costs

Sizes and execution budgets for Hydra protocol transactions. Note that unlisted parameters are currently using arbitrary values and results are not fully deterministic and comparable to previous runs.

Metadata
Generated at	2025-01-29 21:41:27.705580313 UTC
Max. memory units	14000000
Max. CPU units	10000000000
Max. tx size (kB)	16384

Script summary

Name	Hash	Size (Bytes)
νInitial	c8a101a5c8ac4816b0dceb59ce31fc2258e387de828f02961d2f2045	2652
νCommit	61458bc2f297fff3cc5df6ac7ab57cefd87763b0b7bd722146a1035c	685
νHead	5350e9d521552ebfd9e846fd70c3b801f716fc14296134ec0fb71e97	14492
μHead	6b49dc4e571207d615dba01996548cb610b0caa85c30423d169d0091*	5612
νDeposit	ae01dade3a9c346d5c93ae3ce339412b90a0b8f83f94ec6baa24e30c	1102

The minting policy hash is only usable for comparison. As the script is parameterized, the actual script is unique per head.

`Init` transaction costs

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	6169	10.33	3.30	0.53
2	6365	12.04	3.82	0.56
3	6570	14.40	4.58	0.59
5	6966	18.26	5.77	0.65
10	7974	28.59	9.03	0.80
44	14810	98.28	30.94	1.82

`Commit` transaction costs

This uses ada-only outputs for better comparability.

UTxO	Tx size	% max Mem	% max CPU	Min fee ₳
1	561	2.44	1.16	0.20
2	740	3.38	1.73	0.22
3	917	4.36	2.33	0.24
5	1280	6.41	3.60	0.28
10	2173	12.13	7.25	0.40
54	10059	98.61	68.52	1.88

`CollectCom` transaction costs

Parties	UTxO (bytes)	Tx size	% max Mem	% max CPU	Min fee ₳
1	57	525	24.20	7.07	0.42
2	114	636	34.03	9.82	0.52
3	170	747	42.25	12.19	0.61
4	227	858	50.61	14.55	0.70
5	282	969	63.89	18.18	0.84
6	338	1081	63.63	18.48	0.85
7	393	1192	86.78	24.46	1.08
8	450	1303	92.19	26.25	1.14

Cost of Increment Transaction

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	1794	24.13	8.02	0.48
2	2004	26.10	9.43	0.52
3	2077	26.74	10.22	0.53
5	2401	30.52	12.83	0.60
10	3119	38.75	18.86	0.74
43	8028	97.81	60.41	1.73

Cost of Decrement Transaction

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	616	22.72	7.33	0.41
2	760	23.47	8.19	0.43
3	828	23.84	8.96	0.44
5	1085	26.91	11.14	0.50
10	2058	40.02	18.17	0.70
42	6881	99.60	56.21	1.66

`Close` transaction costs

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	654	28.61	9.16	0.48
2	812	28.64	9.90	0.49
3	964	30.22	11.07	0.52
5	1277	34.25	13.77	0.58
10	2097	44.25	20.61	0.75
39	6636	99.36	58.73	1.67

`Contest` transaction costs

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	678	33.35	10.42	0.52
2	810	35.33	11.69	0.56
3	946	37.33	12.97	0.59
5	1380	42.90	16.34	0.68
10	2064	53.42	22.99	0.84
32	5275	98.74	52.13	1.56

`Abort` transaction costs

There is some variation due to the random mixture of initial and already committed outputs.

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	6047	26.04	8.82	0.69
2	6211	35.68	12.11	0.80
3	6325	44.55	15.10	0.90
4	6517	53.44	18.18	1.00
5	6530	58.70	19.83	1.05
6	6687	67.90	22.98	1.16
7	6804	75.70	25.60	1.25
8	6905	88.24	29.90	1.38
9	7094	95.47	32.30	1.47

`FanOut` transaction costs

Involves spending head output and burning head tokens. Uses ada-only UTXO for better comparability.

Parties	UTxO	UTxO (bytes)	Tx size	% max Mem	% max CPU	Min fee ₳
10	0	0	6162	18.39	6.28	0.62
10	1	57	6197	19.99	6.94	0.63
10	5	284	6331	27.61	10.01	0.72
10	10	569	6501	37.77	14.09	0.84
10	30	1705	7179	76.06	29.51	1.29
10	40	2276	7521	95.81	37.45	1.53
10	41	2328	7550	97.77	38.24	1.55

End-to-end benchmark results

This page is intended to collect the latest end-to-end benchmark results produced by Hydra's continuous integration (CI) system from the latest master code.

Please note that these results are approximate as they are currently produced from limited cloud VMs and not controlled hardware. Rather than focusing on the absolute results, the emphasis should be on relative results, such as how the timings for a scenario evolve as the code changes.

Generated at 2025-01-29 21:44:19.656617337 UTC

Baseline Scenario

Number of nodes	1
Number of txs	300
Avg. Confirmation Time (ms)	5.131451220
P99	13.88525022999995ms
P95	6.564831050000002ms
P50	4.623981499999999ms
Number of Invalid txs	0

Three local nodes

Number of nodes	3
Number of txs	900
Avg. Confirmation Time (ms)	22.036612918
P99	38.232529879999994ms
P95	30.5940183ms
P50	20.895989ms
Number of Invalid txs	0

Instead of running the conduits (which would require runResourceT), we just construct the 'DraftHydraNode' directly.

Mistakenly used fuseBoth instead of zipSinks

This avoids redundant definitions at the cost of not having all functions bound in the handle.

ch1bo added 3 commits January 28, 2025 15:22

Define source conduits for PersistenceIncremental and EventSource han…

faf7535

…dles

Update hydrate to stream events

ed75eda

Implement sourceEvents

62bcef4

This requires MonadUnliftIO on hydrate and createNode functions to run the conduits. However, the io-sim tests' IOSim s monad does not have an instance!

ch1bo requested a review from a team January 28, 2025 14:31

ch1bo force-pushed the streaming-events branch 3 times, most recently from 06c5c9f to f3762c6 Compare January 28, 2025 16:08

Stream server events into API projection

53a0ff3

This re-loading the same server outputs 3 times though.

ch1bo force-pushed the streaming-events branch from f3762c6 to 53a0ff3 Compare January 28, 2025 16:10

ch1bo added 3 commits January 29, 2025 16:41

Fix behavior and model tests

33d5153

Instead of running the conduits (which would require runResourceT), we just construct the 'DraftHydraNode' directly.

Fix conduits to actually zip results

019fbed

Mistakenly used fuseBoth instead of zipSinks

Define loadAll and getEvents in terms of conduits

c4691d0

This avoids redundant definitions at the cost of not having all functions bound in the handle.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Streaming events #1808

Streaming events #1808

ch1bo commented Jan 28, 2025 •

edited

Loading

github-actions bot commented Jan 28, 2025 •

edited

Loading

github-actions bot commented Jan 28, 2025 •

edited

Loading

Streaming events #1808

Are you sure you want to change the base?

Streaming events #1808

Conversation

ch1bo commented Jan 28, 2025 • edited Loading

github-actions bot commented Jan 28, 2025 • edited Loading

Transaction cost differences

Script summary

Init transaction costs

Commit transaction costs

CollectCom transaction costs

Cost of Increment Transaction

Cost of Decrement Transaction

Close transaction costs

Contest transaction costs

FanOut transaction costs

github-actions bot commented Jan 28, 2025 • edited Loading

Transaction costs

Script summary

Init transaction costs

Commit transaction costs

CollectCom transaction costs

Cost of Increment Transaction

Cost of Decrement Transaction

Close transaction costs

Contest transaction costs

Abort transaction costs

FanOut transaction costs

End-to-end benchmark results

Baseline Scenario

Three local nodes

ch1bo commented Jan 28, 2025 •

edited

Loading

github-actions bot commented Jan 28, 2025 •

edited

Loading

`Init` transaction costs

`Commit` transaction costs

`CollectCom` transaction costs

`Close` transaction costs

`Contest` transaction costs

`FanOut` transaction costs

github-actions bot commented Jan 28, 2025 •

edited

Loading

`Init` transaction costs

`Commit` transaction costs

`CollectCom` transaction costs

`Close` transaction costs

`Contest` transaction costs

`Abort` transaction costs

`FanOut` transaction costs