(kurtosis) WIP smoke test for l1 pectra support with dev images

[geoknee]

Manual test will be to run:

cast send $(cast az) --auth 0x8943545177806ED17B9F23F0a21ee5948eCaa776 --rpc-url=localhost:PORT --private-key=BATCHER_PK

where PORT is the l1 EL JSON-RPC endpoint.

Towards #14044

[geoknee]

Using reth here since they apparently have support for EIP7702 (whereas geth does not yet).

[geoknee]

The version used for devnet 5 is shown here https://github.com/ethpandaops/pectra-devnets/blob/6f4e7ebd19d35a10392c45facc053602384e0690/ansible/inventories/devnet-5/group_vars/all/images.yaml#L16

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 42.07%. Comparing base (cf7a37b) to head (f6a982b).

Additional details and impacted files

@@             Coverage Diff             @@
##           develop   #14046      +/-   ##
===========================================
- Coverage    46.53%   42.07%   -4.47%     
===========================================
  Files          989      824     -165     
  Lines        83625    74314    -9311     
===========================================
- Hits         38918    31264    -7654     
+ Misses       41768    40294    -1474     
+ Partials      2939     2756     -183     

Flag	Coverage Δ
cannon-go-tests-32	?
cannon-go-tests-64	?
contracts-bedrock-tests	?

Flags with carried forward coverage won't be shown. Click here to find out more.

see 172 files with indirect coverage changes

[geoknee]

I'm struggling to get kurtosis to launch fully so far.

However, while it is starting up and the L1 Reth node comes up, I did manage to attempt to send a SetCodeTx:

➜  ~ cast send $(cast az) --auth 0x8943545177806ED17B9F23F0a21ee5948eCaa776 --rpc-url=localhost:56662 --private-key=BATCHER_KEY

Error: server returned an error response: error code -32003: EIP-7702 authorization list not supported

EDIT: this is because Pectra did not activate yet. We need to wait a few minutes before it does.

[geoknee]

Update: I got this working end to end just now.

I needed to update the op-node image because there were important fixes around bootnodes / discovery. Also rebased and picked up some fixes to our kurtosis setup, which helped.

Repro steps:

1. Run just pectra-devnet on this branch (it uses reth as the EL client on L1, and dev images which preview our op-node and op-geth releases).
2. Wait until Pectra activates. Should take around 6 or 7 minutes
3. Use cast rpc optimism_syncStatus --rpc-url localhost:59238 | jq -r '.safe_l2.number' to watch the safe head, taking care to use the port which is mapped to 8547 on op-node.
4. Use cast send $(cast az) --auth 0x8943545177806ED17B9F23F0a21ee5948eCaa776 --rpc-url=localhost:59200 --private-key=BATCHER_PK to launch an EIP-7702 transaction. This time use the port that maps to 8545 on L1 reth.
5. Check safe head still progresses (i.e. the client did not crash).

Note that the EIP-7702 transaction was not sent to the batch inbox, nor did it have any batch data in it. An extension would be to run a test that sends it to the batch inbox.

[geoknee]

As a sanity check I also ran with op-node/op-geth images on the develop branch, which does not yet contain the fix we are about to release. Interestingly, the op-node does not panic. But we do see a safe head stall and:

2025-01-30 15:06:15 t=2025-01-30T15:06:15+0000 lvl=warn msg="Engine temporary error" err="temp: failed to fetch L1 block info and receipts: querying block: transaction type not supported"
2025-01-30 15:06:15 t=2025-01-30T15:06:15+0000 lvl=error msg="Engine failed temporarily, backing off sequencer" err="temp: failed to fetch L1 block info and receipts: querying block: transaction type not supported"

[geoknee]

Ideally we would run the test without this flag (or set it to false). However, I get the following error form kurtosis in that case:

Adding service with name 'op-cl-1-op-node-op-geth-op-kurtosis' and image 'us-docker.pkg.dev/oplabs-tools-artifacts/images/op-node:v1.10.4-dev.5'
2025/01/30 15:15:25 Error: error deploying environment: error deploying kurtosis package: execution error: error_message:"An error occurred executing instruction (number 53) at github.com/ethpandaops/optimism-package/src/cl/op-node/op_node_launcher.star[115:38]:\nadd_service(name=\"op-cl-1-op-node-op-geth-op-kurtosis\", config=ServiceConfig(image=\"us-docker.pkg.dev/oplabs-tools-artifacts/images/op-node:v1.10.4-dev.5\", ports={\"http\": PortSpec(number=8547, transport_protocol=\"TCP\", application_protocol=\"http\"), \"metrics\": PortSpec(number=9001, transport_protocol=\"TCP\", application_protocol=\"\"), \"tcp-discovery\": PortSpec(number=9003, transport_protocol=\"TCP\", application_protocol=\"\", wait=None), \"udp-discovery\": PortSpec(number=9003, transport_protocol=\"UDP\", application_protocol=\"\", wait=None)}, files={\"/jwt\": \"op_jwt_file\", \"/network-configs\": \"op-deployer-configs\"}, cmd=[\"op-node\", \"--l2=http://{{kurtosis:11923fd2d31e4506bf6f1ca45a0be3f6:ip_address.runtime_value}}:8551\", \"--l2.jwt-secret=/jwt/jwtsecret\", \"--verifier.l1-confs=1\", \"--rollup.config=/network-configs/rollup-2151908.json\", \"--rpc.addr=0.0.0.0\", \"--rpc.port=8547\", \"--rpc.enable-admin\", \"--l1=http://{{kurtosis:3b4df3952ee1433c985bd46e84bd5d2c:ip_address.runtime_value}}:8545\", \"--l1.rpckind=standard\", \"--l1.beacon=http://{{kurtosis:c154ad4e9e3144efa3b614bc6cdfad58:ip_address.runtime_value}}:4000\", \"--p2p.advertise.ip=KURTOSIS_IP_ADDR_PLACEHOLDER\", \"--p2p.advertise.tcp=9003\", \"--p2p.advertise.udp=9003\", \"--p2p.listen.ip=0.0.0.0\", \"--p2p.listen.tcp=9003\", \"--p2p.listen.udp=9003\", \"--safedb.path=/data/op-node/op-node-beacon-data\", \"--metrics.enabled=true\", \"--metrics.addr=0.0.0.0\", \"--metrics.port=9001\", \"--p2p.sequencer.key={{kurtosis:a735b10ad07e451085a1c36351cad8b7:output.runtime_value}}\", \"--sequencer.enabled\", \"--sequencer.l1-confs=2\", \"--l1.trustrpc=false\"], env_vars={}, private_ip_address_placeholder=\"KURTOSIS_IP_ADDR_PLACEHOLDER\", ready_conditions=ReadyCondition(recipe=PostHttpRequestRecipe(port_id=\"http\", endpoint=\"/\", body=\"{\\\"jsonrpc\\\":\\\"2.0\\\",\\\"method\\\":\\\"opp2p_self\\\",\\\"params\\\":[],\\\"id\\\":1}\", content_type=\"application/json\", extract={\"enr\": \".result.ENR\", \"multiaddr\": \".result.addresses[0]\", \"peer_id\": \".result.peerID\"}), field=\"code\", assertion=\"==\", target_value=200, timeout=\"1m\"), labels={\"ethereum-package.client\": \"op-node\", \"ethereum-package.client-image\": \"oplabs-tools-artifacts-images-op-node_v1.10.4-dev.5\", \"ethereum-package.client-type\": \"beacon\", \"ethereum-package.connected-client\": \"op-geth\", \"ethereum-package.sha256\": \"\"}, tolerations=[], node_selectors={}))\n --- at /home/circleci/project/core/server/api_container/server/startosis_engine/startosis_executor.go:157 (sendErrorAndFail) ---\nCaused by: Unexpected error occurred starting service 'op-cl-1-op-node-op-geth-op-kurtosis'\n --- at /home/circleci/project/core/server/api_container/server/startosis_engine/kurtosis_instruction/add_service/add_service.go:212 (AddServiceCapabilities.Execute) ---\nCaused by: An error occurred waiting for all TCP and UDP ports to be open for service 'op-cl-1-op-node-op-geth-op-kurtosis' with private IP '172.16.0.23'; this is usually due to a misconfiguration in the service itself, so here are the logs:\n== SERVICE 'op-cl-1-op-node-op-geth-op-kurtosis' LOGS ===================================\nt=2025-01-30T15:13:25+0000 lvl=info msg=\"Not opted in to ProtocolVersions signal loading, disabling ProtocolVersions contract now.\"\nt=2025-01-30T15:13:25+0000 lvl=info msg=\"No persisted sequencer state loaded\"\nt=2025-01-30T15:13:25+0000 lvl=info msg=\"Rollup Config\" l2_chain_id=2151908 l2_network=\"unknown L2\" l1_chain_id=3151908 l1_network=\"unknown L1\" l2_start_time=1738249987 l2_block_hash=0x8756b358aa26ffcd047722085596ca0e36ef3d980e5d8b9310ec7f30ed3fb5b8 l2_block_number=0 l1_block_hash=0x2e401419ddfda64941a26edddc4d69d895c2333134e042a1b0bc631778b6840c l1_block_number=8 regolith_time=\"@ genesis\" canyon_time=\"@ genesis\" delta_time=\"@ genesis\" ecotone_time=\"@ genesis\" fjord_time=\"@ genesis\" granite_time=\"@ genesis\" holocene_time=\"@ genesis\" isthmus_time=\"(not configured)\" interop_time=\"(not configured)\" alt_da=false\nt=2025-01-30T15:13:25+0000 lvl=info msg=\"Initializing rollup node\" version=v1.10.4-dev.5-339fdfa1-1738182577\nt=2025-01-30T15:13:25+0000 lvl=info msg=\"Connected to L1 Beacon API, ready for EIP-4844 blobs retrieval.\" version=Lighthouse/v6.0.1-b6d80eb/aarch64-linux\nt=2025-01-30T15:13:25+0000 lvl=info msg=\"Safe head database enabled\" path=/data/op-node/op-node-beacon-data\nt=2025-01-30T15:13:25+0000 lvl=error msg=\"failed to fetch runtime config data\" err=\"failed to fetch unsafe block signing address from system config: failed to fetch proof of storage slot 0x65a7ed542fb37fe237fdfbdd70b31598523fe5b32879e307bae27a0bd9581c08 at block 0x2e401419ddfda64941a26edddc4d69d895c2333134e042a1b0bc631778b6840c: distance to target block exceeds maximum proof window\"\nt=2025-01-30T15:13:35+0000 lvl=error msg=\"failed to fetch runtime config data\" err=\"failed to fetch unsafe block signing address from system config: failed to fetch proof of storage slot 0x65a7ed542fb37fe237fdfbdd70b31598523fe5b32879e307bae27a0bd9581c08 at block 0x1892089c36e20b8004cd960e99e6952b7ed35df176247b9b3fb3abb00867823d: distance to target block exceeds maximum proof window\"\nt=2025-01-30T15:13:45+0000 lvl=error msg=\"failed to fetch runtime config data\" err=\"failed to fetch unsafe block signing address from system config: failed to fetch proof of storage slot 0x65a7ed542fb37fe237fdfbdd70b31598523fe5b32879e307bae27a0bd9581c08 at block 0x099fb59d39a6fa2da062052082270439421012f48593a71282b3b14e9d407f19: distance to target block exceeds maximum proof window\"\nt=2025-01-30T15:13:55+0000 lvl=error msg=\"failed to fetch runtime config data\" err=\"failed to fetch unsafe block signing address from system config: failed to fetch proof of storage slot 0x65a7ed542fb37fe237fdfbdd70b31598523fe5b32879e307bae27a0bd9581c08 at block 0xf256514d7ecbf2d0ea03e2c25d1dac37f36a874f8db4bba7a54acb5856e0adf2: distance to target block exceeds maximum proof window\"\nt=2025-01-30T15:14:05+0000 lvl=error msg=\"failed to fetch runtime config data\" err=\"failed to fetch unsafe block signing address from system config: failed to fetch proof of storage slot 0x65a7ed542fb37fe237fdfbdd70b31598523fe5b32879e307bae27a0bd9581c08 at block 0xf256514d7ecbf2d0ea03e2c25d1dac37f36a874f8db4bba7a54acb5856e0adf2: distance to target block exceeds maximum proof window\"\nt=2025-01-30T15:14:05+0000 lvl=error msg=\"Error initializing the rollup node\" err=\"failed to init the runtime config: failed to load runtime configuration repeatedly, last error: operation failed permanently after 5 attempts: failed to fetch unsafe block signing address from system config: failed to fetch proof of storage slot 0x65a7ed542fb37fe237fdfbdd70b31598523fe5b32879e307bae27a0bd9581c08 at block 0xf256514d7ecbf2d0ea03e2c25d1dac37f36a874f8db4bba7a54acb5856e0adf2: distance to target block exceeds maximum proof window\"\nt=2025-01-30T15:14:05+0000 lvl=info msg=\"stopping node: sequencer already stopped\" latestHead=0x0000000000000000000000000000000000000000000000000000000000000000\nt=2025-01-30T15:14:05+0000 lvl=crit msg=\"Application failed\" message=\"failed to setup: unable to create the rollup node: failed to init the runtime config: failed to load runtime configuration repeatedly, last error: operation failed permanently after 5 attempts: failed to fetch unsafe block signing address from system config: failed to fetch proof of storage slot 0x65a7ed542fb37fe237fdfbdd70b31598523fe5b32879e307bae27a0bd9581c08 at block 0xf256514d7ecbf2d0ea03e2c25d1dac37f36a874f8db4bba7a54acb5856e0adf2: distance to target block exceeds maximum proof window\"\n\n== FINISHED SERVICE 'op-cl-1-op-node-op-geth-op-kurtosis' LOGS ===================================\n --- at /home/circleci/project/core/server/api_container/server/service_network/default_service_network.go:1015 (DefaultServiceNetwork.startRegisteredService) ---\nCaused by: An error occurred while waiting for all TCP and UDP ports to be open\n --- at /home/circleci/project/core/server/api_container/server/service_network/default_service_network.go:1396 (waitUntilAllTCPAndUDPPortsAreOpen) ---\nCaused by: Unsuccessful ports check for IP '172.16.0.23' and port spec '{privatePortSpec:0x40009d4660}', even after '240' retries with '500' milliseconds in between retries. Timeout '2m0s' has been reached\n --- at /home/circleci/project/core/server/api_container/server/service_network/default_service_network.go:1426 (waitUntilPortIsOpenWithTimeout) ---\nCaused by: An error occurred while calling network address '172.16.0.23:9001' with port protocol 'TCP' and using time out '200ms'\n --- at /home/circleci/project/core/server/api_container/server/service_network/default_service_network.go:1458 (scanPort) ---\nCaused by: dial tcp 172.16.0.23:9001: i/o timeout"
exit status 1