Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: Initial draft of testnet-runbook #10085

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

stevenplatt
Copy link
Collaborator

@stevenplatt stevenplatt commented Nov 20, 2024

Change log

This PR includes an initial draft of the Aztec testnet runbook.
fixes: #9839

@stevenplatt stevenplatt self-assigned this Nov 20, 2024
@stevenplatt stevenplatt enabled auto-merge (squash) November 20, 2024 22:47
Copy link
Contributor

Changes to circuit sizes

Generated at commit: 95c93eba1055ec0e92199d8bf5d70f75c0db2694, compared to commit: 4129e27e5ed202786ea79da801d5e308d14a5f7d

🧾 Summary (100% most significant diffs)

Program ACIR opcodes (+/-) % Circuit size (+/-) %
rollup_base_private +137,564 ❌ +69.59% +750,997 ❌ +28.25%
rollup_base_public +25,379 ❌ +5.68% -251,515 ✅ -6.29%

Full diff report 👇
Program ACIR opcodes (+/-) % Circuit size (+/-) %
rollup_base_private 335,237 (+137,564) +69.59% 3,409,576 (+750,997) +28.25%
rollup_base_public 472,504 (+25,379) +5.68% 3,745,928 (-251,515) -6.29%


## Releases

The engineering team's testnet responsibilities begin after code-freeze. Here are the primary tasks:

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Worth mentioning maybe that the code freeze is for the branch that passes #9631 the previous week


## Deployment

After cutting a release, deploy a testnet (typically with 48 validators) using the new Docker containers. Verbose logging on Aztec nodes should be enabled by default using the following `ENV VARS`:

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the testnets we deploy, we should begin with a small number (i.e. 4 validators) to start with.

Then under Sanity Check, we can add 1-2 more validators (run by us) to test that nodes can sync the chain. After that we invite external validators to join

- Block proving (Prover: `Processed 1 new L2 blocks`)
- Epoch proving (Prover: `Submitted proof for epoch`)

3. Test external node connection and sync

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about we standardize when we do it? i.e. spotcheck for 3 epochs then add one external validator (that we run) to test it can sync.

3. BOOT_NODE_URL (Kubernetes: `kubectl get services -n <namespace> | (head -3; grep boot)`)
- boot-node-lb-tcp: `<EXTERNAL-IP>:40400`
- boot-node-lb-udp: `<EXTERNAL-IP>:40400`

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At a certain epoch (i.e. 5) once the sanity check passes, Product / DevRel will coordinate with node operators who have already connected to the network using the information above.

Product / DevRel verify that node operators are seeing correct logs, then pass on validator addresses of those ready to engineering so that engineering can add them to the validator set. We do this until we add all 48 validators.


### Network Connection Info

After a successful sanity check, share the following network connection information in the `#team-alpha` slack channel and with the wider Aztec community:

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Product / DevRel to share with the sequencer & prover discord channel.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

chore: draft spec for live network run book
2 participants