Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(tee): add error handling for unstable_getTeeProofs API endpoint #3321

Merged
merged 1 commit into from
Nov 26, 2024

Conversation

pbeza
Copy link
Collaborator

@pbeza pbeza commented Nov 22, 2024

What ❔

This PR adds more information to the response of the unstable_getTeeProofs API endpoint, enabling the client that sent the request to determine whether it makes sense to retry fetching the TEE proof for a particular batch number.

Why ❔

Currently, the TEE verifier – the tool for continuous SGX attestation and batch signature verification – is stuck on batches that failed to be proven and are marked as permanently_ignored. The tool should be able to distinguish between batches that are permanently ignored (and should be skipped) and batches that have failed but will be retried. This PR enables that distinction.

Example use cases:

  • requesting TEE proof for a batch with the permanently_ignored status
    $ curl -i -X POST -H "Content-Type: application/json" --data '{"jsonrpc": "2.0", "id": 1, "method": "unstable_getTeeProofs", "params": [14, "sgx"] }' 'http://localhost:3152'
    HTTP/1.1 200 OK
    content-type: application/json; charset=utf-8
    vary: origin, access-control-request-method, access-control-request-headers
    access-control-allow-origin: *
    content-length: 207
    date: Tue, 26 Nov 2024 12:42:48 GMT
    
    {"jsonrpc":"2.0","result":[{"l1BatchNumber":14,"teeType":"sgx","pubkey":null,"signature":null,"proof":null,"provedAt":"2024-11-20T15:43:46.112146Z","status":"permanently_ignored","attestation":null}],"id":1}
    
  • requesting TEE proof for a batch with the failed status
    $ curl -i -X POST -H "Content-Type: application/json" --data '{"jsonrpc": "2.0", "id": 1, "method": "unstable_getTeeProofs", "params": [15, "sgx"] }' 'http://localhost:3152'
    HTTP/1.1 200 OK
    content-type: application/json; charset=utf-8
    vary: origin, access-control-request-method, access-control-request-headers
    access-control-allow-origin: *
    content-length: 194
    date: Tue, 26 Nov 2024 12:44:19 GMT
    
    {"jsonrpc":"2.0","result":[{"l1BatchNumber":15,"teeType":"sgx","pubkey":null,"signature":null,"proof":null,"provedAt":"2024-11-20T15:43:46.121432Z","status":"failed","attestation":null}],"id":1}
    
  • requesting TEE proof for a batch with the generated status
    $ curl -i  -X POST -H "Content-Type: application/json"     --data '{"jsonrpc": "2.0", "id": 1, "method": "unstable_getTeeProofs", "params": [28, "sgx"] }' 'http://localhost:3152'
    HTTP/1.1 200 OK
    content-type: application/json; charset=utf-8
    vary: origin, access-control-request-method, access-control-request-headers
    access-control-allow-origin: *
    content-length: 229
    date: Tue, 26 Nov 2024 12:45:27 GMT
    
    {"jsonrpc":"2.0","result":[{"l1BatchNumber":28,"teeType":"sgx","pubkey":"0506070809","signature":"0001020304","proof":"1011121314","provedAt":"2024-11-20T15:21:16.129128Z","status":"generated","attestation":"0403020100"}],"id":1}
    
  • requesting TEE proof for a non-existent batch
    $ curl -i -X POST -H "Content-Type: application/json" --data '{"jsonrpc": "2.0", "id": 1, "method": "unstable_getTeeProofs", "params": [1337, "sgx"] }' 'http://localhost:3152'
    HTTP/1.1 200 OK
    content-type: application/json; charset=utf-8
    vary: origin, access-control-request-method, access-control-request-headers
    access-control-allow-origin: *
    content-length: 36
    date: Tue, 26 Nov 2024 12:46:08 GMT
    
    {"jsonrpc":"2.0","result":[],"id":1}
    

Relevant database entries for the use cases mentioned above:

zksync_server_localhost_legacy=# SELECT * FROM tee_proof_generation_details WHERE l1_batch_number IN (14, 15, 28, 1337);
 l1_batch_number |       status        |  signature   |    pubkey    |    proof     | tee_type |         created_at         |         updated_at         |      prover_taken_at       
-----------------+---------------------+--------------+--------------+--------------+----------+----------------------------+----------------------------+----------------------------
              14 | permanently_ignored |              |              |              | sgx      | 2023-11-20 15:27:47.281293 | 2024-11-20 15:43:46.112146 | 2024-11-20 15:43:46.106042
              15 | failed              |              |              |              | sgx      | 2024-11-20 15:27:47.287777 | 2024-11-20 15:43:46.121432 | 2024-11-20 15:43:46.115853
              28 | generated           | \x0001020304 | \x0506070809 | \x1011121314 | sgx      | 2024-11-20 12:56:33.055642 | 2024-11-20 15:21:16.129128 | 2024-11-20 14:53:14.25949
(3 rows)

Checklist

  • PR title corresponds to the body of PR (we generate changelog entries from PRs).
  • Tests for the changes have been added / updated.
  • Documentation comments have been added / updated.
  • Code has been formatted via zkstack dev fmt and zkstack dev lint.

@pbeza pbeza force-pushed the tee/pab/error-handling-get-tee-proofs-api branch from 6793a52 to c6c92f2 Compare November 22, 2024 11:49
@pbeza pbeza requested review from haraldh and slowli November 22, 2024 11:54
@pbeza pbeza force-pushed the tee/pab/error-handling-get-tee-proofs-api branch 3 times, most recently from b8a9b51 to 41ce9d0 Compare November 22, 2024 12:20
@pbeza
Copy link
Collaborator Author

pbeza commented Nov 22, 2024

I get that exposing the status directly from the database (line 58 below) might not be the best idea, since any changes to the status types would instantly show up on the client side, making the API unstable. That said, I’m not sure how to handle this in a cleaner way. Open to any suggestions you've got, @slowli @haraldh.

pub async fn get_tee_proofs_impl(
&self,
l1_batch_number: L1BatchNumber,
tee_type: Option<TeeType>,
) -> Result<Vec<TeeProof>, Web3Error> {
let mut storage = self.state.acquire_connection().await?;
let proofs = storage
.tee_proof_generation_dal()
.get_tee_proofs(l1_batch_number, tee_type)
.await
.map_err(DalError::generalize)?
.into_iter()
.map(|proof| TeeProof {
l1_batch_number,
tee_type,
pubkey: proof.pubkey,
signature: proof.signature,
proof: proof.proof,
proved_at: DateTime::<Utc>::from_naive_utc_and_offset(proof.updated_at, Utc),
status: proof.status,
attestation: proof.attestation,
})
.collect::<Vec<_>>();
if proofs.is_empty() {
return Err(Web3Error::NoBlock);
}
Ok(proofs)
}

@pbeza pbeza force-pushed the tee/pab/error-handling-get-tee-proofs-api branch 2 times, most recently from ece164e to 7a534c8 Compare November 22, 2024 14:26
slowli
slowli previously approved these changes Nov 25, 2024
Copy link
Contributor

@slowli slowli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regarding the status field: I don't think this is a large concern as long as the corresponding RPC method is unstable.

core/lib/dal/src/tee_proof_generation_dal.rs Outdated Show resolved Hide resolved
core/node/api_server/src/web3/namespaces/unstable.rs Outdated Show resolved Hide resolved
This PR adds more information to the response of the
`unstable_getTeeProofs` API endpoint, enabling the [client][1] that sent
the [request][2] to determine whether it makes sense to retry fetching
the TEE proof for a particular batch number.

Currently, the [TEE verifier][1] – the tool for continuous SGX
attestation and batch signature verification – is [stuck][3] on batches
that failed to be proven and are marked as `permanently_ignored`. The
tool should be able to distinguish between batches that are permanently
ignored (and should be skipped) and batches that have failed but will be
retried. This PR enables that distinction.

Example use cases:
- requesting TEE proof for a batch with the `permanently_ignored` status
  ```
  $ curl -i -X POST -H "Content-Type: application/json" --data '{"jsonrpc": "2.0", "id": 1, "method": "unstable_getTeeProofs", "params": [14, "sgx"] }' 'http://localhost:3152'
  HTTP/1.1 200 OK
  content-type: application/json; charset=utf-8
  vary: origin, access-control-request-method, access-control-request-headers
  access-control-allow-origin: *
  content-length: 207
  date: Tue, 26 Nov 2024 12:42:48 GMT

  {"jsonrpc":"2.0","result":[{"l1BatchNumber":14,"teeType":"sgx","pubkey":null,"signature":null,"proof":null,"provedAt":"2024-11-20T15:43:46.112146Z","status":"permanently_ignored","attestation":null}],"id":1}
  ```
- requesting TEE proof for a batch with the `failed` status
  ```
  $ curl -i -X POST -H "Content-Type: application/json" --data '{"jsonrpc": "2.0", "id": 1, "method": "unstable_getTeeProofs", "params": [15, "sgx"] }' 'http://localhost:3152'
  HTTP/1.1 200 OK
  content-type: application/json; charset=utf-8
  vary: origin, access-control-request-method, access-control-request-headers
  access-control-allow-origin: *
  content-length: 194
  date: Tue, 26 Nov 2024 12:44:19 GMT

  {"jsonrpc":"2.0","result":[{"l1BatchNumber":15,"teeType":"sgx","pubkey":null,"signature":null,"proof":null,"provedAt":"2024-11-20T15:43:46.121432Z","status":"failed","attestation":null}],"id":1}
  ```
- requesting TEE proof for a batch with the `generated` status
  ```
  $ curl -i  -X POST -H "Content-Type: application/json"     --data '{"jsonrpc": "2.0", "id": 1, "method": "unstable_getTeeProofs", "params": [28, "sgx"] }' 'http://localhost:3152'
  HTTP/1.1 200 OK
  content-type: application/json; charset=utf-8
  vary: origin, access-control-request-method, access-control-request-headers
  access-control-allow-origin: *
  content-length: 229
  date: Tue, 26 Nov 2024 12:45:27 GMT

  {"jsonrpc":"2.0","result":[{"l1BatchNumber":28,"teeType":"sgx","pubkey":"0506070809","signature":"0001020304","proof":"1011121314","provedAt":"2024-11-20T15:21:16.129128Z","status":"generated","attestation":"0403020100"}],"id":1}
  ```
- requesting TEE proof for a non-existent batch
  ```
  $ curl -i -X POST -H "Content-Type: application/json" --data '{"jsonrpc": "2.0", "id": 1, "method": "unstable_getTeeProofs", "params": [1337, "sgx"] }' 'http://localhost:3152'
  HTTP/1.1 200 OK
  content-type: application/json; charset=utf-8
  vary: origin, access-control-request-method, access-control-request-headers
  access-control-allow-origin: *
  content-length: 36
  date: Tue, 26 Nov 2024 12:46:08 GMT

  {"jsonrpc":"2.0","result":[],"id":1}
  ```

Relevant database entries for the use cases mentioned above:
```
zksync_server_localhost_legacy=# SELECT * FROM tee_proof_generation_details WHERE l1_batch_number IN (14, 15, 28, 1337);
 l1_batch_number |       status        |  signature   |    pubkey    |    proof     | tee_type |         created_at         |         updated_at         |      prover_taken_at
-----------------+---------------------+--------------+--------------+--------------+----------+----------------------------+----------------------------+----------------------------
              14 | permanently_ignored |              |              |              | sgx      | 2023-11-20 15:27:47.281293 | 2024-11-20 15:43:46.112146 | 2024-11-20 15:43:46.106042
              15 | failed              |              |              |              | sgx      | 2024-11-20 15:27:47.287777 | 2024-11-20 15:43:46.121432 | 2024-11-20 15:43:46.115853
              28 | generated           | \x0001020304 | \x0506070809 | \x1011121314 | sgx      | 2024-11-20 12:56:33.055642 | 2024-11-20 15:21:16.129128 | 2024-11-20 14:53:14.25949
(3 rows)
```

[1]: https://github.com/matter-labs/teepot/blob/main/bin/verify-era-proof-attestation/src/main.rs
[2]: https://github.com/matter-labs/teepot/blob/1a8a9f17fa7284f83c41a63d37fe380aef6d550d/bin/verify-era-proof-attestation/src/proof.rs#L15-L21
[3]: https://grafana.matterlabs.dev/goto/unFqf57Hg?orgId=1
@pbeza pbeza force-pushed the tee/pab/error-handling-get-tee-proofs-api branch from 4ca5a10 to facb415 Compare November 26, 2024 12:57
pbeza added a commit to matter-labs/teepot that referenced this pull request Nov 26, 2024
Currently, the [TEE verifier][1] – the tool for continuous SGX
attestation and batch signature verification – is [stuck][2] on batches
that failed to be proven and are marked as `permanently_ignored`. The
tool should be able to distinguish between batches that are permanently
ignored (and should be skipped) and batches that have failed but will be
retried. This PR enables that distinction.

This commit goes hand in hand with the following PR:
matter-labs/zksync-era#3321

[1]: https://github.com/matter-labs/teepot/blob/main/bin/verify-era-proof-attestation/src/main.rs
[2]: https://grafana.matterlabs.dev/goto/unFqf57Hg?orgId=1
pbeza added a commit to matter-labs/teepot that referenced this pull request Nov 26, 2024
Currently, the [TEE verifier][1] – the tool for continuous SGX
attestation and batch signature verification – is [stuck][2] on batches
that failed to be proven and are marked as `permanently_ignored`. The
tool should be able to distinguish between batches that are permanently
ignored (and should be skipped) and batches that have failed but will be
retried. This PR enables that distinction.

This commit goes hand in hand with the following PR:
matter-labs/zksync-era#3321

[1]: https://github.com/matter-labs/teepot/blob/main/bin/verify-era-proof-attestation/src/main.rs
[2]: https://grafana.matterlabs.dev/goto/unFqf57Hg?orgId=1
pbeza added a commit to matter-labs/teepot that referenced this pull request Nov 26, 2024
Currently, the [TEE verifier][1] – the tool for continuous SGX
attestation and batch signature verification – is [stuck][2] on batches
that failed to be proven and are marked as `permanently_ignored`. The
tool should be able to distinguish between batches that are permanently
ignored (and should be skipped) and batches that have failed but will be
retried. This PR enables that distinction.

This commit goes hand in hand with the following PR:
matter-labs/zksync-era#3321

[1]: https://github.com/matter-labs/teepot/blob/main/bin/verify-era-proof-attestation/src/main.rs
[2]: https://grafana.matterlabs.dev/goto/unFqf57Hg?orgId=1
@pbeza
Copy link
Collaborator Author

pbeza commented Nov 26, 2024

JFYI: this commit goes hand in hand with the following PR: matter-labs/teepot#221 (feel free to review it).

@haraldh pls review/merge when you get a chance. Thanks!

@haraldh haraldh added this pull request to the merge queue Nov 26, 2024
Merged via the queue into main with commit 26f630c Nov 26, 2024
32 checks passed
@haraldh haraldh deleted the tee/pab/error-handling-get-tee-proofs-api branch November 26, 2024 18:26
github-merge-queue bot pushed a commit that referenced this pull request Dec 11, 2024
🤖 I have created a release *beep* *boop*
---


##
[25.3.0](core-v25.2.0...core-v25.3.0)
(2024-12-11)


### Features

* change seal criteria for gateway
([#3320](#3320))
([a0a74aa](a0a74aa))
* **contract-verifier:** Download compilers from GH automatically
([#3291](#3291))
([a10c4ba](a10c4ba))
* integrate gateway changes for some components
([#3274](#3274))
([cbc91e3](cbc91e3))
* **proof-data-handler:** exclude batches without object file in GCS
([#2980](#2980))
([3e309e0](3e309e0))
* **pruning:** Record L1 batch root hash in pruning logs
([#3266](#3266))
([7b6e590](7b6e590))
* **state-keeper:** mempool io opens batch if there is protocol upgrade
tx ([#3360](#3360))
([f6422cd](f6422cd))
* **tee:** add error handling for unstable_getTeeProofs API endpoint
([#3321](#3321))
([26f630c](26f630c))
* **zksync_cli:** Health checkpoint improvements
([#3193](#3193))
([440fe8d](440fe8d))


### Bug Fixes

* **api:** batch fee input scaling for `debug_traceCall`
([#3344](#3344))
([7ace594](7ace594))
* **tee:** correct previous fix for race condition in batch locking
([#3358](#3358))
([b12da8d](b12da8d))
* **tee:** fix race condition in batch locking
([#3342](#3342))
([a7dc0ed](a7dc0ed))
* **tracer:** adds vm error to flatCallTracer error field if exists
([#3374](#3374))
([5d77727](5d77727))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

---------

Co-authored-by: zksync-era-bot <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants