feat: Poll the main node for batch to vote on (BFT-496)

[aakoshh]

What ❔

Poll the main node for which batch height to vote on.

• Return an Option rather than a Vec of BatchQC from BatchVotes::find_quorum: we decided not to implement voting on multiple heights for now, which makes it just confusing that a vector is returned.
• Create an AttestationStatusClient trait with a method to poll the next BatchNumber to vote on, which can be injected as a dyn dependency
• Create an AttestationStatusWatch that contains the next BatchNumber to vote on; this is updated by a single background task and listened to by multiple subscribers
• Create an AttestationStatusRunner which is supposed to be started along with the BatchStore in zksync-era and polls the client; the point of this is to push out the poll interval configuration to the edge and make the AttestationStatusWatch the integration point
• Change AttesterRunner to wait for the AttestationStatusWatch to change, then the payload to appear, then cast the vote; it doesn't need to worry about reverts because the node will restart if that happens
• Change run_batch_qc_finder to wait for the AttestationStatusWatch to change, then wait for the next available QC, and save it; this might produce gaps which is normal on the external nodes, but undersireable on the main node - on the main node it only produces gaps if the attesters vote on higher heights despite the main node telling them not to, which if they are majority honest should not happen, and if they are not then what can we do anyway
• Rename PersistentBatchStore::earliest_batch_number_to_sign to next_batch_to_attest in accordance with the new method on consensus_dal.
• Initialise BatchNumber::min_batch_number to by asking the main node where to resume from This initialisation isn't necessary because 1) we decided that we'll only allow 1 vote per attester, so there is no attack vector of casting votes from batch 0 and 2) run_batch_qc_finder first waits for the API status to appear and then prunes all preceding data, so an older QC will not be saved. It was also undesirable to have to initialise from an API that might return nothing (or errors) for an unknown amount of time.

Poll interval

The AttesterStatusRunner polls the API at fixed intervals without any exponential backoff. I thought 5s would be a reasonable default value. With 60s batches this seems fine because the next batch to sign will be signalled as soon as the current batch QC is inserted into the database, which is well ahead of time of when the next batch and its commitment will be available. If we think we'll need to catch up with historical batch signatures it might be a bit slow. We generally still have to account for gossiping around the votes though, which in a large enough network could be on the order of several seconds as well.

Potential deadlock

There is a pathological sequence of events that could prevent the feature from starting on a fresh database with no prior batch QCs:

1. Say we start from batch 0, and a QC is gathered, but saving it to the database (which is an async process) takes a long time on the main node.
2. Say it takes so long that batch 1 and batch 2 are created, but none of them have a QC yet.
3. Say we have two attesters A and B, and A polled the API when it showed batch 2, and the main node set its minimum batch number in the vote registry to batch 2 as well.
4. Now the QC for batch 0 is saved to the database and the API indicates the next one to vote on is batch 1.
5. Attester B casts its vote on batch 1 but it goes ignored by the main node vote registry because its looking for a quorum with at least batch number 2.
6. Having missed an attestation the main node will never save a QC and never move on until it's restarted.

Note that even if the registry minimum batch number was adjusted down that might happen after the vote for batch 1 has already been discarded, and because new votes are only pushed once through gossip there is no guarantee that it will get it again.

The solution is coming in the form of fixing the starting point of gossip to be where the genesis starts, even if that means filling a potentially long history of batches in the beginning. See matter-labs/zksync-era#2539

Why ❔

We want to collect batch QCs without gaps, and for now put the main node in charge of what to vote on. The API to serve this information is in matter-labs/zksync-era#2480 and this PR is the follow up to make polling that information part of the signing process.

[pompon0]

LGTM! thx