Skip to content
This repository has been archived by the owner on Aug 7, 2023. It is now read-only.

Minigeth Diff

Jump to bottom
norswap edited this page May 18, 2022 · 15 revisions

This page tries to explain the changes between Optimism's fork of Geth (which we'll call "L2 geth" or "execution engine", and is itself a very small diff on top of L1 geth) and minigeth.

The current format is that this goes through all commits in the diff in chronological order. Each commits either comprises deletions, or additions/modifications, with the deletions grouped at the start.

Note: It's sometimes useful to refer to the original minigeth history, when it was included in the Cannon repo. The minigeth-files-history branch points to the last commit that has the minigeth directory, so that file history can be explored.

Deletions

All test files have been deleted (i.e. *_test.go and *_fuzz.go).

To generate a list of files and dirs deleted between two commits, you can use the following command:

git diff-tree --diff-filter=D --name-only -t <commit1> <commit2_or_nothing_for_head> -- <dir>

If entire directories have been deleted, you can get their list by using (source):

git diff-tree origin/l2geth-head HEAD | grep 040000 | grep -v -E '^:040000 040000'`

Each bullet below matches a commit name in the diff.

  • "delete unused top-level directories and files"

    • interfaces.go
    • .github/
    • accounts/
    • build/
    • cmd/
    • console/
    • contracts/
    • docs/
    • eth/
    • ethclient/
    • ethstats/
    • event/
    • graphql/
    • internal/
    • les/
    • light/
    • miner/
    • mobile/
    • node/
    • p2p/
    • signer/
    • swarm/
    • tests/

  • "delete files in common directory"

    • common/bitutil/
    • common/compiler/
    • common/fdlimit/
    • common/mclock/
    • common/prque/
    • common/debug.go
    • common/path.go
    • common/test_utils.go

  • "delete files in consensus directory"

    • consensus/clique/
    • consensus/ethash/ (fake_ethash.go added later)
    • consensus/misc/dao.go
    • consensus/misc/forks.go
    • consensus/merger.go

  • "delete files in core directory"

    • core/asm/
    • core/bloombits/
    • core/forkid/
    • core/rawdb/
    • core/types/
    • core/vm/
    • core/state/ — everything excepted
      • core/state/snapshot/snapshot.go
      • core/state/access_list.go
      • core/state/database.go
      • core/state/journal.go
      • core/state/statedb.go
      • core/state/state_object.go
    • core/types/gen_access_tuple.go
    • core/types/gen_header_json.go
    • core/types/gen_log_json.go
    • core/types/gen_receipt_json.go
    • core/types/transaction_marshalling.go
    • core/vm/doc.go
    • core/vm/runtime/
    • core/vm/testdata/
    • core/.gitignore
    • core/block_validator.go
    • core/blockchain.go
    • core/blockchain_insert.go
    • core/blocks.go
    • core/bloom_indexer.go
    • core/chain_indexer.go
    • core/chain_makers.go
    • core/events.go
    • core/gen_genesis.go
    • core/gen_genesis_account.go
    • core/genesis.go
    • core/genesis_alloc.go
    • core/headerchain.go
    • core/mkalloc.go
    • core/state_prefetcher.go
    • core/tx_cacher.go
    • core/tx_journal.go
    • core/tx_list.go
    • core/tx_noncer.go
    • core/tx_pool.go
    • core/types.go

  • "delete files in crypto directory"

    • crypto/ecies/
    • crypto/secp256k1/
    • crypto/signify/
    • crypto/blake2b/ — everything excepted
      • crypto/blake2b/blake2b.go
      • crypto/blake2b/blake2b_generic.go
      • crypto/blake2b/blake2b_ref.go
    • crypto/bls12381/arithmetic_decl.go
    • crypto/bls12381/arithmetic_x86.s
    • crypto/bls12381/arithmetic_x86_adx.go
    • crypto/bls12381/arithmetic_x86_noadx.go
    • crypto/bn256/cloudflare/
    • crypto/bn256/bn256_fast.go
    • crypto/signature_cgo.go

  • "delete files in ethdb directory"

    • ethdb/dbtest/
    • ethdb/leveldb/
    • ethdb/memorydb/

  • "delete files in metrics directory" — everything excepted

    • metrics/metrics.go

  • "delete files in logs directory" — everything excepted

    • log/logger.go

  • "delete files in params directory" — everything excepted

    • params/config.go
    • params/protocol_params.go

  • "delete files in rlp directory"

    • rlp/doc.go
    • rlp/iterator.go
    • rlp/safe.go

  • "delete files in rpc directory" — everything excepted

    • rpc/errors.go
    • rpc/json.go
    • rpc/types.go

  • "delete files in trie directory"

    • trie/proof.go
    • trie/sync.go

Additions & Modifications

  • "modify and add files in core directory"

    • [ADD] core/fake_blockchain.go
      • this replaces the struct in core/blockchain.go and reimplements some of the functions that were implemented in core/blockchain_reader.go
    • core/state_processor.go
      • don't support DAO hardfork
      • write dot to stdout for each tx
    • core/statedb.go
      • remove StartPrefetcher, StopPrefetcher
      • comment out snapshot-related logic
      • comment out remove Get[.*]Proof functions
      • add PrefetchAccount call in deleteStateObject
      • add PrefetchAccount call in getDeletedStateObject
      • rewrite the loop in Commit (TODO understand what this does)
        • necessary because we removed the DB
      • remove meter updates
    • core/state_object.go
      • add PrefetchStorage in GetCommittedState
      • add PrefetchStorage in updateTrie
    • core/state/snapshot.go
      • stub it out entirely
        • TODO: why does the stub need to exist? it builds when removing it, so try if it also runs without it
    • core/state/database.go
      • replace the Database interface with a Database struct that replaces the interface
        • the implementation delegates to the preimage oracle

  • "modify and add files in crypto directory"

    • use blake2b_ref.go for all archs
    • use bn256_slow.go for all archs
    • use signature_nocgo.go for all archs
    • add the btcec secp256k1 implementation
    • for all the above, the alternative implementations were deleted in the crypto directory deletion commit

  • "modify metrics/metrics.go"

    • essentially deletes the file, keeping only var EnabledExpensive = false
      • this disables all expensive metric computations during execution

  • "modify log/logger.go"

    • replaces all logger functions by simple prints

  • "modify params/config.go"

    • adds a IsMerge function to determine if we're after the merge, it gets used in fake_ethash.go
    • this is change I (norswap) made in order to accomodate changes in Geth

  • "modify rlp/unsafe.go"

    • enables it for all archs, the alternative "safe" implem was deleted in the rlp directory deletion commit

  • "modify files in the rpc directory"

    • removes now unused types and implementations in the rpc implementation

  • "modify files in the trie directory"

    • TODO: in stacktrie.go, one line is commented out but not clear why
    • TODO: in trie.go, adds a print upon hashnode deletion that we can probably remove
    • in trie.go, ignores an error
      • this happens when deleting from a "full node" (i.e. a node with multiple children, or a child and a value)
      • ... and in particular when the deletion causes one of its child to be deleted
      • ... and in particular when that causes the full node to only have one child left (not a value)
      • ... which causes the full node to be replaced by another
      • to know which nodes must replace the full node, it is necessary to know whether the remaining child is a short node (aka "extension node") which requires looking up the node (hash) in the db/preimage oracle
      • it is possible that the preimage oracle does not know this hash
        • context: when deleting a key, minigeth in prefetch mode retrieves an absence proof for the key from the state in the next block
          • this happens in updateTrie in state_object.go
        • it is possible that the remaining child is missing from the absence proof, if something under it gets deleted later in block processing
          • the comment added there says that in that case, the child must be a short/extension node, but I don't see/understand why that should be the case
          • the comment also says that the if the node is an extension node, then the error can safely be ignore. This also seems supicious to me, since in that case the node will be nil, the short node check will fail and we'll end up with a short node containing a short node, something that is forbidden.
      • this is the code that causes the bounty bug (unability to fetch preimages from transient nodes) so sort of normal it doesn't make sense + worth investigating thoroughly because we need to fix it anyway
        • it was added in this PR, not by George himself
    • in secure_trie.go
      • comment out GetKey, which is not used, and not supported anymore (because we replaced the db by the preimage oracle, which ironically does not support retrieving the preimage of a "secure key" (which is the hash of an underlying key)
      • part of the logic in Commit is commented out for the same reason
    • in database.go, the whole implementation was swapped out for one that uses the preimage oracle
    • in database.go, GenPossibleShortNodePreimage was also added, which is related to the absence proof mechanism mentionned above
      • as far as I can tell it speculatively generate possible preimages for extension nodes containing a value
        • but as mentionned above, the assumption that the node we need will be an extension node seems mistaken

  • "add consensus/ethash/fake_ethash.go"

    • As the name implies! fake_ethash replaces the original ethhash, which is the consensus implementation, meaning what is responsible to validate the headers
    • In particular, it guts all the validation logic ... I'm not quite sure why this was done, seems like this logic is rather important to the integrity of a block. This definitely needs to go back in for Bedrock.
    • The difficulty calculation logic was retained because difficulty is part of the header and not supplied via the input hash. We don't need this for Bedrock and can rip it off.
      • I (norswap) also patched this for the rebase to return a difficulty of 1 after the merge.

  • "add preimage oracle"

    • this adds two implementation of the preimage oracle
    • embedded_mips.go for MIPS execution
      • all PrefetchXXX functions & related are no-ops
      • the Preimage function write hash at address 0x30001000, reads preimage from address 0x31000000 (more details here)
      • InputHash reads input hash from pre-determined memory location, Output writes output hash to pre-determined memory location
    • preimage.go & prefetch.go for prefetch mode
      • PrefetchXXX functions (in prefetch.go) query a JSON-RPC node (Infura by default) and store the preimages to disk storage
      • Preimage reads the disk storage filed by PrefetchXXX functions
      • InputHash returns the hash initialized from two calls to PrefetchBlock done from the main.go
      • Output just prints whether the transition was good or not (i.e. whether the computed output hash matches the output hash contained within the input committed to by the input hash)
      • TODO: this has a bunch of comments we should clean up
    • apitypes.go contains definitions which are used for parsing JSON-RPC responses
      • SendTxArgs is a copy of ethapi.TransactionArgs where common.Address fields were replaced by common.MixedCaseAddress - why? presumalby because it wouldn't work otherwise, but we should check - ethapi was deleted earlier, could we just resurrect it and just edit the definition there?
      • Header is a structure used to parse headers returned by JSON-RPC answers (which uses SendTxArgs)
      • there is logic to convert SendTxArgs to types.Transaction and Header to types.Header
      • TODO: investigate if we couldn't reuse existing geth code more

  • "add README and main.go"

    • the main routine
    • it validates a block transition, using a set of inputs (see inputs definition here)
      • in MIPS mode, an input hash committing to the inputs is read from memory, where it was included prior to execution
      • in prefetch mode, one block number as command-line argument, and the inputs are retrieved by fetching the block headers for these two blocks (from which the input hash is re-derived)
      • in both cases, the inputs are accessed via the input hash + the preimage oracle
      • besides EVM execution, some validation is also done, but I think it's not quite up to parity with what used to be done in the real ethhash
        • there is a comment that mentions ValidateState from the deleted core/block_validator.go
      • the last step is call Output - the file contains a few TODO that need to be investigated and removed

TODO to improve the diff

  • there are quite a few comments that need to be cleaned up
  • it would be good to comment the code more
  • !! make sure the validation logic is complete
  • address TODOs in the above, mostly it's about seeing if some of the added / leftover stuff is really necessary
  • the rebase introduced new files which by definition didn't conflict, and a few of those are bound to be unused and can be deleted

General

Minigeth

Misc

Clone this wiki locally