Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FLIP 243: Flow EVM Gateway #235

Open
wants to merge 17 commits into
base: main
Choose a base branch
from
Open
Changes from 14 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
311 changes: 311 additions & 0 deletions application/20240103-evm-gateway.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,311 @@
---
status: implemented
m-Peter marked this conversation as resolved.
Show resolved Hide resolved
flip: [243]
m-Peter marked this conversation as resolved.
Show resolved Hide resolved
authors: Ardit Marku (markoupetr@gmail.com), Gregor Gololicic (gregor.gololicic@flowfoundation.org)
sponsor: Jerome Pimmel (jerome.pimmel@dapperlabs.com)
updated: 2024-01-03
---

# FLIP 235: Flow EVM Gateway
m-Peter marked this conversation as resolved.
Show resolved Hide resolved

An implementation of the Ethereum JSON-RPC specification.

## Objective

This proposal outlines the design and implementation of the [Ethereum JSON-RPC specification](https://ethereum.github.io/execution-apis/api-documentation/) into a standalone
service, which aims to provide a gateway to Flow EVM for developers and the
various existing tools.

## Motivation

The [JSON-RPC API](https://ethereum.org/en/developers/docs/apis/json-rpc/) specifies a
uniform set of methods that software applications, such as MetaMask or an Ethereum client,
can use to interact with the Ethereum blockchain, either by reading blockchain data or
sending transactions to the network. To achieve EVM compatibility, it is crucial to
provide a JSON-RPC API, which among other things, facilitates the onboarding of developers
that are already familiar with it and enables interoperability with the existing tools
built on top of the JSON-RPC API.

## User Benefit

Users will be able to use an EVM-compatible client to connect to the Flow EVM network,
by providing the network name, JSON-RPC URL, chain ID, currency etc.
Developers will be able to easily retrieve all sorts of information from Flow EVM,
such as block/transaction/account data, as well as define filters and subscribe
to changes regarding the previous entities. This will allow developers to focus
on building new tools and products, without having to write custom indexers from
scratch.

## Design Proposal

### Context

JSON-RPC is a stateless, light-weight remote procedure call (RPC) protocol.
It defines several data structures and the rules around their processing.
It is transport agnostic in that the concepts can be used within the same
process, over sockets, over HTTP, or in many various message passing
environments. It uses JSON (RFC 4627) as the data format.

### Existing RPC Server implementations

The Geth client offers a generic [RPC Server](https://github.com/ethereum/go-ethereum/blob/master/rpc/server.go) implementation, which will be used in the implementation
of the Flow EVM Gateway as well.

```go
type BlockChainAPI struct {
config *Config
Store *storage.Store
}

func NewBlockChainAPI(config *Config, store *storage.Store) *BlockChainAPI {
return &BlockChainAPI{
config: config,
Store: store,
}
}

// eth_chainId
// Returns the chain ID used for signing replay-protected transactions.
func (api *BlockChainAPI) ChainId() *hexutil.Big {
return (*hexutil.Big)(api.config.ChainID)
}

// eth_blockNumber
// BlockNumber returns the block number of the chain head.
func (api *BlockChainAPI) BlockNumber() hexutil.Uint64 {
latestBlockHeight, err := api.Store.LatestBlockHeight(context.Background())
if err != nil {
panic(fmt.Errorf("failed to fetch the latest block number: %v", err))
}
return hexutil.Uint64(latestBlockHeight)
}

// Create an RPC server.
srv := rpc.NewServer()

apis := []rpc.API{
{
Namespace: "eth",
Service: NewBlockChainAPI(config, store),
},
}

// Register all the APIs exposed by the services
for _, api := range apis {
if err := srv.RegisterName(api.Namespace, api.Service); err != nil {
return err
}
}
```

The above code snippet instantiates a new RPC server and registers a
service object (`*BlockChainAPI`) under the `eth` namespace.

The server can now accept HTTP requests with the following body:

```json
{"jsonrpc":"2.0","id":1,"method":"eth_chainId","params": []}
```

and respond with an appropriate result, such as:

```json
{"jsonrpc":"2.0","id":1,"result":"0x29a"}
```

Under the hood, the RPC server reads the value of the `"method"` key.
In this case `"eth_chainId"`, the part before the `_` is the namespace
(`eth`) and the part after the `_` is the method which should be called
on the registered service object (`ChainId`).

Some sample HTTP calls to the Flow EVM Gateway's JSON-RPC server, can
be seen below:

![Flow EVM Gateway JSON-RPC over HTTP](./2024-01-03-evm-gateway-resources/flow-evm-gateway-curl-http.png)

### Supported transport protocols

There are three transport protocols available in Geth: IPC, HTTP and WebSockets.

For the gateway, we are interested in supporting only JSON-RPC over HTTP and
JSON-RPC over WebSockets. The RPC methods with event subscription are available
only for WebSocket connections.

Again, the Geth client has an implementation of an [httpServer](https://github.com/ethereum/go-ethereum/blob/master/node/rpcstack.go) which is basically a proxy that routes `HTTP` &
`WebSocket` requests to a dedicated `rpc.Server` instance for each of the two
transport mediums. This allows for more fine-grained control over which namespaces
& RPC methods are available on which transport, and allows for better scaling.

Unfortunately, none of the relevant types in the `go-ethereum` package are exported,
so we cannot pull them directly from the `go-ethereum` package. But even if they
were exported, it comes with quite a lot of extras, such as `CORS` / `JWT` / `GZIP`
compression which might be overkill for what we currently need. For the time being,
we have extracted a minified version of it [here](https://github.com/onflow/flow-evm-gateway/blob/aaf3ba3f2ab78321fa377ecf5165a2f7dbc44865/api/server.go).

### Indexing FlowEVM events

Quoting some content from [EVM support](https://github.com/onflow/flips/pull/225) FLIP:

> To better understand the approach proposed in this Flip, consider EVM on Flow as a virtual blockchain deployed to the Flow blockchain at a specific address (e.g., a service account). EVM on Flow functions as a smart contract that emulates the EVM with its own dedicated chain-ID. Signed transactions are inputted, and a chain of blocks is produced as output. Similar to other built-in standard contracts (e.g., RLP encoding), this EVM environment can be imported into any Flow transaction or script.
This is made possible using selective integration of the core EVM runtime without the supporting software stack in which it currently exists on Ethereum. For equivalence we also provide an EVM compatible JSON-RPC API implementation to facilitate EVM on Flow interactions from existing EVM clients.

Since the gateway is not the only entrypoint for interacting with Flow EVM, we need
to track every event emitted from the `EVM` smart contract, consume it and build an
index which reflects the latest state of the chain.

For the indexing, we will be using the Event Streaming API offered from the
`flow-go-sdk`. A sample code snippet can be seen below:

```go
flowClient, err := grpc.NewBaseClient(
grpc.EmulatorHost,
goGrpc.WithTransportCredentials(insecure.NewCredentials()),
)
if err != nil {
logger.Error().Msgf("could not create flow client: %v", err)
}

data, errChan, initErr := flowClient.SubscribeEventsByBlockHeight(
ctx,
latestBlockHeader.Height,
flow.EventFilter{
EventTypes: []string{
"flow.evm.BlockExecuted",
"flow.evm.TransactionExecuted",
},
},
grpc.WithHeartbeatInterval(1),
)

// track the most recently seen block height. we will use this when reconnecting.
// the first response should be for latestBlockHeader.Height
lastHeight := latestBlockHeader.Height - 1
for {
select {
case <-ctx.Done():
return

case response, ok := <-data:
if !ok {
if ctx.Err() != nil {
return // graceful shutdown
}
logger.Error().Msg("subscription closed - reconnecting")
connect(lastHeight+1, 10)
continue
}

if response.Height != lastHeight+1 {
logger.Error().Msgf("missed events response for block %d", lastHeight+1)
connect(lastHeight, 10)
continue
}

logger.Info().Msgf("block %d %s:", response.Height, response.BlockID)
if len(response.Events) > 0 {
store.StoreBlockHeight(ctx, response.Height)
}
for _, event := range response.Events {
logger.Info().Msgf(" %s", event.Value)
}

lastHeight = response.Height

case err, ok := <-errChan:
...
}
}
```

The main events from the `EVM` smart contract are two:
- `flow.evm.BlockExecuted`,
- `flow.evm.TransactionExecuted`

To see the indexer & the events' payloads in action:

![Flow EVM Events](./2024-01-03-evm-gateway-resources/flow-evm-events.png)

### Storage Integrity
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The integrity mechanism outlined below is a good start though we may need to build in more logic to recover missed events. While the goal is to not miss events we should plan for a reality where it will eventually happen.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, certainly. Back-filling of past events is something we should account for.


While consuming events, it is very critical to make sure that they are processed
in a sequential manner, and no events response is missed for a single block.
If this were to happen, the storage index could end up corrupted, and we would
return incorrect data for things such as account balances, nonces etc.

In the code snippet below, we specify a heartbeat interval of `1` and check
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would say this would be good to change in the future as a performance improvement, ideally we would get multiple events and still preserve the order and integrity, however this currently might not be possible yet due to API not providing the heights, but it might be worth linking to an issue and keeping a promise to improve it.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to clarify, even with a heartbeat interval of 1, we still get all the events emitted within the transactions of a single block. You are right though, I guess it will improve performance if we use a larger heartbeat interval when the backend introduces the sequence number field. Quoting the original message from Peter Argue in onflow/flow-evm-gateway#11 (comment)

The SubscribeEvents endpoint has a heartbeatInterval request parameter:
https://github.com/onflow/flow/blob/b7ed2c3e452b5e44561399afc6b5e868e8957117/protobuf/flow/executiondata/executiondata.proto#L140-L148

This is used to set how often the backend will return a response message. If you set it to 1, it will return a response for every block. If the block has no events, the list is empty. If you check the block height from each message increases sequentially, you can verify that you received all messages, and thus all blocks were searched.

The API is currently missing an important sequence number field, which would let you get the same guarantee with larger heartbeatIntervals without receiving a response for every block.

Given missing any events may corrupt your index, I'd recommend using the 1 block heartbeat for now.

that each received events response is for the expected block height, before
committing any changes to the storage.

```go
data, errChan, initErr := flowClient.SubscribeEventsByBlockHeight(
...,
grpc.WithHeartbeatInterval(1),
// this will yield an events response for each block,
// even if it has no events matching the filter criteria.
)

// track the most recently seen block height. we will use this when reconnecting.
// the first response should be for latestBlockHeader.Height
lastHeight := latestBlockHeader.Height - 1
for {
select {
case response, ok := <-data:
...
if response.Height != lastHeight+1 {
logger.Error().Msgf("missed events response for block %d", lastHeight+1)
connect(lastHeight, 10)
continue
}
// make sure that the response is for the expected block height,
// before committing any changes to the storage.
...

// mark the block height of the events response as processed.
lastHeight = response.Height
}
}
```

### Performance Implications

We should have some end-to-end tests, perhaps with the usage of the Flow Emulator.
As well as some benchmarks, to assess the service degradation with regards to the
increase of the storage, the incoming requests and possibly the emitted events.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would also create stress tests that could be run as part of the Flow benchnet, with good metrics on the gateway side we could measure performance. The metrics should also be used in production for monitoring. It would be also worth exploring tracing with OTEL.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very good suggestion 👏

### Dependencies

The Flow EVM Gateway is a standalone service, hosted in its own repository, so
it does not add any dependencies to existing core repositories. It is
build around some of Flow's core repositories, such as `flow-go-sdk` and
`flow-go`, so it depends on their public APIs.
Later on, it would be useful to expose the Flow EVM Gateway from the Emulator
as well, as we have done with the gRPC & REST Access API.

### Tutorials and Examples

A reference implementation addressing parts of this proposal can be found in
this [PR](https://github.com/onflow/flow-evm-gateway/pull/11#issue-2006802251).

### User Impact

The Flow EVM Gateway is a standalone service that complements the Flow EVM
functionality. It should be released at around the same time as Flow EVM,
in order to allow for better beta testing.

## Related Issues

As with any service, we need to think about logging, alerting, monitoring,
storage capacity and throughput.
Dockerizing the gateway would be nice, in order to allow developers to easily
start using it for development and testing.

## Prior Art

As explained above, the [Geth](https://github.com/ethereum/go-ethereum) client
was the main source of inspiration for the work on the JSON-RPC server.

## Questions and Discussion

- Do we expect to have one official EVM Gateway run by the Flow team and
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We expect the community to maintain the EVM Gateways the same way as currently are for AN.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good, I guess I should introduce more flexible config options then.

individual developers/team can also run their own? As is the case with
Access Nodes, for example.