-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: FFLONK support for compressor #3359
base: main
Are you sure you want to change the base?
Changes from all commits
976e550
75bb3a4
175d987
01b5bf8
61915d5
e16afbb
ab463f7
1eff67d
ed64ac9
b931948
d6eafce
3ac09ce
b2d5000
264e69f
9920051
ebd5c7c
295990c
ba6791d
169a6b5
d1178f7
78fedef
31525f5
fb9bcb1
852c749
001ae61
2e5043c
c30d79b
bc070c4
c35745f
df58203
25a8d2e
22fff07
59faf9b
7dd3ba3
d3dbaf5
b45a6da
05286b4
b3995d4
2d7572c
924bdeb
f2b43e7
3c2fa71
5d7ae04
b2594e7
837ea8b
8544d16
0171eed
90be08b
f02c6d2
e76ed7f
efed1ac
34f542d
f27a2a4
6dc05d8
dee361e
e45f0e8
019e33d
3de4496
4acdd70
3cdcae8
dc78371
0a37276
a38be1e
0df8945
08fc7ec
bcecc16
f44c2bd
cf780a9
4ad7509
549a6a9
f8f59bf
2e372a1
69e88c3
f511d1d
1b882ca
76c6bfb
a7e004d
c303551
01762e1
dccc9d7
4d3a733
fe574c6
e688b6e
c2a34ed
723dbaf
8baf3f3
92c9025
1ebb279
fbdbce7
2e7967d
0ba982b
9252602
175986c
1f6c673
b5e9e27
7de9135
dacd488
ee28a19
b58bfc5
7f15f0f
959490c
c753c4a
32255b0
e054b90
0a9c239
b78b2e4
56c5066
e197bc5
d39c585
45ff601
96078b7
0107be5
fe898af
afdf441
d3e398a
190ae28
8694a0a
c3b2460
3275cf1
837ea56
f8683cd
e10cc0a
6a6c6c0
88eacad
e24d512
07dfdf9
bef7828
3003016
1df00ec
e3c0eb7
76262ff
3b1354f
49ac31c
bc36651
4def64c
4cbb544
12b6743
0b8d354
14cb430
9590bc1
e069d6c
dbb140c
cd52eec
fed8c17
750673a
cad280a
8805dd2
ed3c3b9
9f37ad7
2ac76f1
1c00984
f993307
9d4f0b2
da7f78c
9ad683e
7f55be9
47a5525
0c494df
0eb7676
8f48976
5ee8a36
477e3b4
2760b08
040253b
9903f87
a99caf4
6fe62d3
bbce735
9073736
04bdb83
1913a3a
98d1b51
27a39c3
221af86
d789bbb
e84ea57
10de1f7
e37b45b
25bec51
6ec1d16
c954f47
f5a4143
2aa3ff9
66078c5
20a698e
1cbd575
6afcdab
e6438ee
a129450
b62e30a
90c9020
1513b7a
5a4f53f
b5c69cb
f2558a6
eba5d5e
2b0fe46
38baa92
5bcab6c
1632cc9
16e8c43
f0f0b2f
36c7bd1
39821dc
6c671a9
5c53770
c3866f0
ef1f29e
dceb6be
765fcb1
715fe80
c45744b
9342a69
187a64d
03b204c
9dbda25
dc3bba7
9c97918
efa4eaa
0766cca
6e0fdec
a099b19
c5953ec
5b4163c
26ac57a
3ab2534
07857d5
ae4a391
620676e
2ec536d
8b10af6
32a0951
a961e69
c5aad48
ca1c1cc
6475d4e
22de0c9
54a61c7
e4ab244
2db5063
68a1f14
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -5,6 +5,10 @@ on: | |
jobs: | ||
e2e-test: | ||
runs-on: [ matterlabs-ci-gpu-l4-runner-prover-tests ] | ||
strategy: | ||
fail-fast: false | ||
matrix: | ||
compressor-mode: ["fflonk", "plonk"] | ||
env: | ||
RUNNER_COMPOSE_FILE: "docker-compose-gpu-runner-cuda-12-0.yml" | ||
|
||
|
@@ -24,17 +28,26 @@ jobs: | |
echo "SCCACHE_GCS_RW_MODE=READ_WRITE" >> .env | ||
echo "RUSTC_WRAPPER=sccache" >> .env | ||
|
||
mkdir -p prover_logs | ||
mkdir -p prover_logs_${{matrix.compressor-mode}} | ||
|
||
- name: Start services | ||
run: | | ||
run_retried docker-compose -f ${RUNNER_COMPOSE_FILE} pull | ||
docker-compose -f ${RUNNER_COMPOSE_FILE} --profile runner up -d --wait | ||
ci_run sccache --start-server | ||
ci_run git config --global --add safe.directory "*" | ||
|
||
- name: Checkout DualVerifier commit | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Uh-oh, can't wait to get rid of this, but decent workaround for the time being. |
||
working-directory: ./contracts | ||
if: matrix.compressor-mode == 'fflonk' | ||
run: | | ||
git fetch | ||
git checkout b4d5b984 | ||
git submodule update --init --recursive | ||
git rev-parse HEAD | ||
|
||
- name: Init | ||
run: | | ||
ci_run git config --global --add safe.directory "*" | ||
ci_run chmod -R +x ./bin | ||
|
||
ci_run ./zkstack_cli/zkstackup/install -g --path ./zkstack_cli/zkstackup/zkstackup || true | ||
|
@@ -51,28 +64,37 @@ jobs: | |
--base-token-price-denominator 1 \ | ||
--set-as-default true \ | ||
--ignore-prerequisites \ | ||
--evm-emulator false | ||
--evm-emulator false \ | ||
--update-submodules=${{matrix.compressor-mode == 'plonk'}} | ||
|
||
ci_run zkstack ecosystem init --dev --update-submodules=${{matrix.compressor-mode == 'plonk'}} --verbose | ||
|
||
ci_run zkstack ecosystem init --dev --verbose | ||
ci_run zkstack prover init --dev --verbose | ||
|
||
echo "URL=$(grep "http_url" ./chains/proving_chain/configs/general.yaml | awk '{ print $2 }')" >> $GITHUB_ENV | ||
- name: Build prover binaries | ||
Artemka374 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
- name: Switch verifier to FFLONK interface | ||
working-directory: ./chains/proving_chain/configs | ||
if: matrix.compressor-mode == 'fflonk' | ||
run: | | ||
ci_run cargo build --release --workspace --manifest-path=prover/Cargo.toml | ||
sudo sed -i 's/is_verifier_pre_fflonk: true/is_verifier_pre_fflonk: false/' general.yaml | ||
grep "is_verifier_pre_fflonk" general.yaml | awk '{ print $2 }' | ||
|
||
- name: Prepare prover subsystem | ||
run: | | ||
ci_run zkstack prover init-bellman-cuda --clone --verbose | ||
ci_run zkstack prover setup-keys --mode=download --region=us --verbose | ||
|
||
- name: Run server | ||
run: | | ||
ci_run zkstack server --uring --chain=proving_chain --components=api,tree,eth,state_keeper,commitment_generator,proof_data_handler,vm_runner_protective_reads,vm_runner_bwip &>prover_logs/server.log & | ||
ci_run zkstack server --uring --chain=proving_chain --components=api,tree,eth,state_keeper,commitment_generator,proof_data_handler,vm_runner_protective_reads,vm_runner_bwip &>prover_logs_${{matrix.compressor-mode}}/server.log & | ||
- name: Run Gateway | ||
run: | | ||
ci_run zkstack prover run --component=gateway --docker=false &>prover_logs/gateway.log & | ||
ci_run zkstack prover run --component=gateway --docker=false &>prover_logs_${{matrix.compressor-mode}}/gateway.log & | ||
- name: Run Prover Job Monitor | ||
run: | | ||
ci_run zkstack prover run --component=prover-job-monitor --docker=false &>prover_logs/prover-job-monitor.log & | ||
ci_run zkstack prover run --component=prover-job-monitor --docker=false &>prover_logs_${{matrix.compressor-mode}}/prover-job-monitor.log & | ||
# Sleeping until the server, gateway and prover job monitor are up | ||
ci_run sleep 150 | ||
Artemka374 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
- name: Wait for batch to be passed through gateway | ||
env: | ||
DATABASE_URL: postgres://postgres:notsecurepassword@localhost:5432/zksync_prover_localhost_proving_chain | ||
|
@@ -84,31 +106,31 @@ jobs: | |
ci_run ./bin/prover_checkers/batch_availability_checker | ||
- name: Run Witness Generator | ||
run: | | ||
ci_run zkstack prover run --component=witness-generator --round=all-rounds --docker=false &>prover_logs/witness-generator.log & | ||
ci_run zkstack prover run --component=witness-generator --round=all-rounds --docker=false &>prover_logs_${{matrix.compressor-mode}}/witness-generator.log & | ||
- name: Run Circuit Prover | ||
run: | | ||
ci_run zkstack prover run --component=circuit-prover -l=23 -h=3 --docker=false &>prover_logs/circuit_prover.log & | ||
ci_run zkstack prover run --component=circuit-prover -l=23 -h=3 --docker=false &>prover_logs_${{matrix.compressor-mode}}/circuit_prover.log & | ||
- name: Wait for prover jobs to finish | ||
env: | ||
DATABASE_URL: postgres://postgres:notsecurepassword@localhost:5432/zksync_prover_localhost_proving_chain | ||
BATCH_NUMBER: 1 | ||
INTERVAL: 30 | ||
TIMEOUT: 1200 | ||
TIMEOUT: 1800 | ||
run: | | ||
PASSED_ENV_VARS="DATABASE_URL,BATCH_NUMBER,INTERVAL,TIMEOUT" \ | ||
ci_run ./bin/prover_checkers/prover_jobs_status_checker | ||
|
||
- name: Kill prover & start compressor | ||
run: | | ||
sudo ./bin/prover_checkers/kill_prover | ||
ci_run zkstack prover run --component=compressor --docker=false --mode=${{matrix.compressor-mode}} &>prover_logs_${{matrix.compressor-mode}}/compressor.log & | ||
|
||
ci_run zkstack prover run --component=compressor --docker=false &>prover_logs/compressor.log & | ||
- name: Wait for batch to be executed on L1 | ||
env: | ||
DATABASE_URL: postgres://postgres:notsecurepassword@localhost:5432/zksync_prover_localhost_proving_chain | ||
BATCH_NUMBER: 1 | ||
INTERVAL: 30 | ||
TIMEOUT: 600 | ||
TIMEOUT: 1200 | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I see a bunch of timeouts have been increased. Is this caused by a slowdown in the process, or why is it necessary? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We need it since build time is increased |
||
run: | | ||
PASSED_ENV_VARS="BATCH_NUMBER,DATABASE_URL,URL,INTERVAL,TIMEOUT" \ | ||
ci_run ./bin/prover_checkers/batch_l1_status_checker | ||
|
@@ -117,8 +139,8 @@ jobs: | |
uses: actions/upload-artifact@50769540e7f4bd5e21e526ee35c689e35e0d6874 # v4.4.0 | ||
if: always() | ||
with: | ||
name: prover_logs | ||
path: prover_logs | ||
name: prover_logs_${{matrix.compressor-mode}} | ||
path: prover_logs_${{matrix.compressor-mode}} | ||
|
||
- name: Show sccache logs | ||
if: always() | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Whilst not a problem, any reason for moving this change from Init to start services?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
just because for checkouting/updating submodules I needed this to be before actual checkout