Productionize KZG EIP-4844

[mratsim]

This addresses #300

• C API for Ethereum KZG EIP-4844
  • Parallel C API
  • Tests for C
• Go wrapper for fuzzing in https://github.com/jtraglia/kzg-fuzz (cc @jtraglia)
• Rust wrapper for benchmarking in https://github.com/sifraitech/rust-kzg (cc @sauliusgrigaitis)
• Update trusted setup to official output of KZG ceremony / mainnet: https://github.com/CarlBeek/kzg-ceremony-verifier/blob/master/output_setups/trusted_setup_4096.json
  • Optional, ability to ingest the official json trusted setup (ideally using a formally verified or correct-by-construction json parser like https://project-everest.github.io/assets/everparse.pdf
  • Or use the c-kzg-4844 format to limit switching friction

[sauliusgrigaitis]

@mratsim let me know if you have any questions regarding adding constantine to rust-kzg. The latest work now is in the integration branch. After recent refactorings in that branch it's not that hard to add a new ECC library. Let me know when you are planning to work on it. If you are overloaded I can also check with my students.

[mratsim]

3-way benchmarks vs c-kzg-4844 (cc @jtraglia) and go-kzg-4844 (cc @kevaundray)

Bench	c-kzg-4844 (serial)	go-kzg-4844 (serial)	go-kzg-4844 (parallel)	constantine (serial)	constantine (parallel)
blob_to_kzg_commitment	37.773 ms	-	5.823 ms	1.987 ms	0.532 ms
compute_kzg_proof	39.945 ms	-	7.146 ms	23.984 ms	4.654 ms
compute_blob_kzg_proof	40.212 ms	-	7.205 ms	24.092 ms	4.712 ms
verify_kzg_proof	0.915 ms	0.923 ms	-	0.780 ms	-
verify_blob_kzg_proof	1.531 ms	-	1.390 ms	1.260 ms	1.101 ms
verify_blob_kzg_proof_batch 1	1.528 ms	1.392 ms	1.405 ms	1.287 ms	1.176 ms
verify_blob_kzg_proof_batch 2	2.589 ms	3.233 ms	1.591 ms	2.015 ms	1.490 ms
verify_blob_kzg_proof_batch 4	4.553 ms	4.671 ms	1.914 ms	3.413 ms	1.843 ms
verify_blob_kzg_proof_batch 8	8.446 ms	7.410 ms	2.738 ms	6.136 ms	3.134 ms
verify_blob_kzg_proof_batch 16	16.228 ms	12.734 ms	3.542 ms	11.354 ms	5.205 ms
verify_blob_kzg_proof_batch 32	32.016 ms	23.048 ms	7.215 ms	21.666 ms	9.790 ms
verify_blob_kzg_proof_batch 64	63.415 ms	43.224 ms	14.438 ms	42.903 ms	17.738 ms

[sauliusgrigaitis]

Impressive numbers! Howevere, blob_to_kzg_commitment looks very suspicious. Did you fuzz your implementation against go-kzg-4844 and c-kzg-4844

[mratsim]

Not too sure why blob_to_kzg_commitment is so fast, AFAIK I generate random values similar to the other benchmarks.

Anyway to give an upper bound, because blob_to_kzg_commitment is just a 4096-sized MSM, here is the pure MSM bench

26ms serial, 4.5 ms parallel

[mratsim]

Impressive numbers! Howevere, blob_to_kzg_commitment looks very suspicious. Did you fuzz your implementation against go-kzg-4844 and c-kzg-4844

Not fuzzed yet but I do pass the tests, I assume it's my random values that are weird because the code is really just forwarding to the MSM.

[jtraglia]

Yes, very impressive if accurate! I agree with Saulius that the blob_to_kzg_commitment benchmark looks fishy. Are you sure it's being done serially? Can you artificially limit the CPU to only use one core? Also, what are the random values exactly? Is the blob composed of valid field elements?

Regarding the go-kzg-4844 benchmarks, even the serial functions still use multithreading so it's slightly misleading. It's due to how gnark-crypto is implemented. Here's a better comparison, ignoring the parallel benchmarks:

[jtraglia]

To my eyes, it appears that blobs are only initialized with a single field element. This would explain things.

constantine/benchmarks/bench_ethereum_eip4844_kzg.nim

Lines 46 to 52 in c784928

	proc new(T: type BenchSet, ctx: ptr EthereumKZGContext): T =
	new(result)
	for i in 0 ..< result.N:
	let t {.noInit.} = rng.random_unsafe(Fr[BLS12_381])
	result.blobs[i].marshal(t, bigEndian)
	discard ctx.blob_to_kzg_commitment(result.commitments[i], result.blobs[i].addr)
	discard ctx.compute_blob_kzg_proof(result.proofs[i], result.blobs[i].addr, result.commitments[i])

[mratsim]

To my eyes, it appears that blobs are only initialized with a single field element. This would explain things.

Good catch, this was it, new bench (didn't rebench c-kzg-4844 and go-kzg-4844)

Bench	c-kzg-4844 (serial)	go-kzg-4844 (serial)	go-kzg-4844 (parallel)	constantine (serial)	constantine (parallel)
blob_to_kzg_commitment	37.773 ms	-	5.823 ms	24.987 ms	4.465 ms
compute_kzg_proof	39.945 ms	-	7.146 ms	24.332 ms	4.720 ms
compute_blob_kzg_proof	40.212 ms	-	7.205 ms	24.291 ms	4.747 ms
verify_kzg_proof	0.915 ms	0.923 ms	-	0.789 ms	-
verify_blob_kzg_proof	1.531 ms	-	1.390 ms	1.265 ms	1.115 ms
verify_blob_kzg_proof_batch 1	1.528 ms	1.392 ms	1.405 ms	1.337 ms	1.180 ms
verify_blob_kzg_proof_batch 2	2.589 ms	3.233 ms	1.591 ms	2.008 ms	1.466 ms
verify_blob_kzg_proof_batch 4	4.553 ms	4.671 ms	1.914 ms	3.492 ms	1.830 ms
verify_blob_kzg_proof_batch 8	8.446 ms	7.410 ms	2.738 ms	6.207 ms	3.137 ms
verify_blob_kzg_proof_batch 16	16.228 ms	12.734 ms	3.542 ms	11.483 ms	5.204 ms
verify_blob_kzg_proof_batch 32	32.016 ms	23.048 ms	7.215 ms	23.165 ms	9.590 ms
verify_blob_kzg_proof_batch 64	63.415 ms	43.224 ms	14.438 ms	45.152 ms	17.493 ms

[jtraglia]

Nice, that's still really impressive. Good job!

[mratsim]

Turned out I wasn't parallelizing kzg batch verification but only preprocessing. See a1ced2d

There is a whopping 52% speedup on parallel batch verification.

Bench	c-kzg-4844 (serial)	go-kzg-4844 (serial)	go-kzg-4844 (parallel)	constantine (serial)	constantine (parallel)
blob_to_kzg_commitment	37.773 ms	-	5.823 ms	23.765 ms	4.425 ms
compute_kzg_proof	39.945 ms	-	7.146 ms	24.255 ms	4.710 ms
compute_blob_kzg_proof	40.212 ms	-	7.205 ms	24.288 ms	4.794 ms
verify_kzg_proof	0.915 ms	0.923 ms	-	0.782 ms	-
verify_blob_kzg_proof	1.531 ms	-	1.390 ms	1.266 ms	1.113 ms
verify_blob_kzg_proof_batch 1	1.528 ms	1.392 ms	1.405 ms	1.286 ms	1.130 ms
verify_blob_kzg_proof_batch 2	2.589 ms	3.233 ms	1.591 ms	2.006 ms	1.152 ms
verify_blob_kzg_proof_batch 4	4.553 ms	4.671 ms	1.914 ms	3.437 ms	1.250 ms
verify_blob_kzg_proof_batch 8	8.446 ms	7.410 ms	2.738 ms	6.115 ms	1.891 ms
verify_blob_kzg_proof_batch 16	16.228 ms	12.734 ms	3.542 ms	11.567 ms	3.091 ms
verify_blob_kzg_proof_batch 32	32.016 ms	23.048 ms	7.215 ms	21.779 ms	6.764 ms
verify_blob_kzg_proof_batch 64	63.415 ms	43.224 ms	14.438 ms	43.099 ms	11.538 ms

[mratsim]

PR too long due to trusted setup updates and .yaml test vectors.

The parallel C API, Go and Rust bindings will be provided in a separate.