diff --git a/README.md b/README.md index 531e21af..4cb13d49 100644 --- a/README.md +++ b/README.md @@ -49,10 +49,9 @@ - [Bugs Reporting & Contributions](#bugs-reporting-contributions) - [Scripts](#scripts) - ## Introduction -The Refiner is a block specimen data processing and transformation framework (Rudder), the purpose of which is validated data transformation. +The Refiner is a block specimen data processing and transformation framework (Rudder), the purpose of which is validated data transformation. Generally, the Refiner has the capability to perform arbitrary transformations over any binary block specimen file, concurrently with other transformations. This enables simultaneous data indexing, with any consumer of the data slicing and dicing the data as they see fit. Such concurrent execution of ethereum blocks (via block specimens), makes it possible to trace, enrich or analyze blockchain data at an unprecedented rate with no sequential bottlenecks (provided each block specimen is its own independent entity and available at a decentralized content address!). @@ -76,7 +75,7 @@ At a very high level, the Refiner locates a source to apply a transformational r ## Architecture -![Rudder Pipeline](./docs/pipeline.png) +![Rudder Pipeline](./docs/pipeline.jpg) The happy path for `rudder` (the refiner) application in the Covalent Network is made up of actor processes spawned through many [Gen Servers](https://elixir-lang.org/getting-started/mix-otp/genserver.html) processes that are loosely coupled, here some maintain state and some don't. The children processes can be called upon to fulfill responsibilities at different sections in the refinement/transformation process pipeline - under one umbrella [Dynamic Supervisor](https://elixir-lang.org/getting-started/mix-otp/dynamic-supervisor.html), that can bring them back up in case of a failure to continue a given pipeline operation. Read more about the components and their operations in the [full architecture document](./docs/ARCH.md). diff --git a/docker-compose-mbase.yml b/docker-compose-mbase.yml index 3679c17b..5af00db6 100644 --- a/docker-compose-mbase.yml +++ b/docker-compose-mbase.yml @@ -1,13 +1,12 @@ -version: '3' -# runs the entire rudder pipeline with all supporting services (including rudder) in docker -# set .env such that all services in docker are talking to each other only services: ipfs-pinner: image: "us-docker.pkg.dev/covalent-project/network/ipfs-pinner:stable" volumes: - ~/.ipfs:/root/.ipfs/ container_name: ipfs-pinner - restart: on-failure + restart: always + labels: + "autoheal": "true" expose: - "4001:4001" - "3000:3000" @@ -22,7 +21,9 @@ services: evm-server: image: "us-docker.pkg.dev/covalent-project/network/evm-server:stable" container_name: evm-server - restart: on-failure + restart: always + labels: + "autoheal": "true" expose: - "3002:3002" networks: @@ -35,15 +36,16 @@ services: container_name: rudder links: - "ipfs-pinner:ipfs-pinner" + - "evm-server:evm-server" # build: # context: . # dockerfile: Dockerfile - restart: on-failure + restart: always depends_on: ipfs-pinner: - condition: service_started + condition: service_healthy evm-server: - condition: service_started + condition: service_healthy entrypoint: > /bin/bash -l -c " echo "moonbase-node:" $NODE_ETHEREUM_MAINNET; @@ -61,5 +63,14 @@ services: networks: - cqt-net + autoheal: + image: willfarrell/autoheal + container_name: autoheal + volumes: + - '/var/run/docker.sock:/var/run/docker.sock' + environment: + - AUTOHEAL_INTERVAL=10 + - CURL_TIMEOUT=30 + networks: - cqt-net: + cqt-net: \ No newline at end of file diff --git a/docker-compose-mbeam.yml b/docker-compose-mbeam.yml index 90b184e8..73a10f38 100644 --- a/docker-compose-mbeam.yml +++ b/docker-compose-mbeam.yml @@ -7,7 +7,9 @@ services: volumes: - ~/.ipfs:/root/.ipfs/ container_name: ipfs-pinner - restart: on-failure + restart: always + labels: + "autoheal": "true" expose: - "4001:4001" - "3000:3000" @@ -22,7 +24,9 @@ services: evm-server: image: "us-docker.pkg.dev/covalent-project/network/evm-server:stable" container_name: evm-server - restart: on-failure + restart: always + labels: + "autoheal": "true" expose: - "3002:3002" networks: @@ -35,15 +39,16 @@ services: container_name: rudder links: - "ipfs-pinner:ipfs-pinner" + - "evm-server:evm-server" # build: # context: . # dockerfile: Dockerfile - restart: on-failure + restart: always depends_on: ipfs-pinner: - condition: service_started + condition: service_healthy evm-server: - condition: service_started + condition: service_healthy entrypoint: > /bin/bash -l -c " echo "moonbase-node:" $NODE_ETHEREUM_MAINNET; @@ -61,5 +66,14 @@ services: networks: - cqt-net + autoheal: + image: willfarrell/autoheal + container_name: autoheal + volumes: + - '/var/run/docker.sock:/var/run/docker.sock' + environment: + - AUTOHEAL_INTERVAL=10 + - CURL_TIMEOUT=30 + networks: cqt-net: diff --git a/docs/ARCH.md b/docs/ARCH.md index a11e222b..653ed8c8 100644 --- a/docs/ARCH.md +++ b/docs/ARCH.md @@ -10,7 +10,7 @@ - [Pipeline Journal](#pipeline-journal) - [Pipeline Telemetry](#pipeline-telemetry) -![Rudder Pipeline](./pipeline.png) +![Rudder Pipeline](./pipeline.jpg) The happy path for `rudder` (the refiner) application in the Covalent Network is made up of actor processes spawned through many [Gen Servers](https://elixir-lang.org/getting-started/mix-otp/genserver.html) processes that are loosely coupled, here some maintain state and some don't. diff --git a/docs/arch.png b/docs/arch.png deleted file mode 100644 index 37431c64..00000000 Binary files a/docs/arch.png and /dev/null differ diff --git a/docs/covalent.jpg b/docs/covalent.jpg deleted file mode 100644 index 54f33bdc..00000000 Binary files a/docs/covalent.jpg and /dev/null differ diff --git a/docs/pipeline.jpg b/docs/pipeline.jpg new file mode 100644 index 00000000..a85cb58d Binary files /dev/null and b/docs/pipeline.jpg differ diff --git a/docs/pipeline.png b/docs/pipeline.png deleted file mode 100644 index 97ffd633..00000000 Binary files a/docs/pipeline.png and /dev/null differ diff --git a/docs/refiner.png b/docs/refiner.png deleted file mode 100644 index 0ca79e5c..00000000 Binary files a/docs/refiner.png and /dev/null differ diff --git a/docs/roadmap.png b/docs/roadmap.png deleted file mode 100644 index 1169361c..00000000 Binary files a/docs/roadmap.png and /dev/null differ diff --git a/lib/rudder/application.ex b/lib/rudder/application.ex index 023a7e78..4d8954f8 100644 --- a/lib/rudder/application.ex +++ b/lib/rudder/application.ex @@ -27,7 +27,12 @@ defmodule Rudder.Application do # See https://hexdocs.pm/elixir/Supervisor.html # for other strategies and supported options - options = [strategy: :one_for_one, name: Rudder.Supervisor] + options = [ + strategy: :one_for_one, + name: Rudder.Supervisor, + max_restarts: 3, + max_seconds: 1200 + ] Supervisor.start_link(children, options) end diff --git a/lib/rudder/evm/block_processor.ex b/lib/rudder/evm/block_processor.ex index 63995d36..41883658 100644 --- a/lib/rudder/evm/block_processor.ex +++ b/lib/rudder/evm/block_processor.ex @@ -45,7 +45,11 @@ defmodule Rudder.BlockProcessor do {:reply, {:error, errormsg}, state} end + {:error, %Mint.TransportError{reason: reason}} when reason in [:econnrefused, :nxdomain] -> + raise "#{inspect(reason)}: is evm-server up?" + {:error, errormsg} -> + Logger.error("error in blockprocessor: #{inspect(errormsg)}") {:reply, {:error, errormsg}, state} end end @@ -74,6 +78,6 @@ defmodule Rudder.BlockProcessor do @impl true def terminate(reason, _state) do - Logger.info("terminating blockprocessor: #{reason}") + Logger.info("terminating blockprocessor: #{inspect(reason)}") end end diff --git a/lib/rudder/ipfs/ipfs_interactor.ex b/lib/rudder/ipfs/ipfs_interactor.ex index 4d99691b..87e353e9 100644 --- a/lib/rudder/ipfs/ipfs_interactor.ex +++ b/lib/rudder/ipfs/ipfs_interactor.ex @@ -35,18 +35,27 @@ defmodule Rudder.IPFSInteractor do content_type = Multipart.content_type(multipart, "multipart/form-data") headers = [{"Content-Type", content_type}, {"Content-Length", to_string(content_length)}] - {:ok, %Finch.Response{body: body, headers: _, status: _}} = + resp = Finch.build("POST", url, headers, {:stream, body_stream}) |> Finch.request(Rudder.Finch) - body_map = body |> Poison.decode!() + case resp do + {:ok, %Finch.Response{body: body, headers: _, status: _}} -> + body_map = body |> Poison.decode!() - end_pin_ms = System.monotonic_time(:millisecond) - Events.ipfs_pin(end_pin_ms - start_pin_ms) + end_pin_ms = System.monotonic_time(:millisecond) + Events.ipfs_pin(end_pin_ms - start_pin_ms) - case body_map do - %{"error" => error} -> {:reply, {:error, error}, state} - %{"cid" => cid} -> {:reply, {:ok, cid}, state} + case body_map do + %{"error" => error} -> {:reply, {:error, error}, state} + %{"cid" => cid} -> {:reply, {:ok, cid}, state} + end + + {:error, %Mint.TransportError{reason: :econnrefused}} -> + raise "connection refused: is ipfs-pinner started?" + + {:error, err} -> + {:reply, {:error, err}, state} end end @@ -56,22 +65,31 @@ defmodule Rudder.IPFSInteractor do ipfs_url = Application.get_env(:rudder, :ipfs_pinner_url) - {:ok, %Finch.Response{body: body, headers: _, status: _}} = + resp = Finch.build(:get, "#{ipfs_url}/get?cid=#{cid}") |> Finch.request(Rudder.Finch, receive_timeout: 150_000_000, pool_timeout: 150_000_000) - end_fetch_ms = System.monotonic_time(:millisecond) - Events.ipfs_fetch(end_fetch_ms - start_fetch_ms) + case resp do + {:ok, %Finch.Response{body: body, headers: _, status: _}} -> + end_fetch_ms = System.monotonic_time(:millisecond) + Events.ipfs_fetch(end_fetch_ms - start_fetch_ms) + + try do + body_map = body |> Poison.decode!() + + case body_map do + %{"error" => error} -> {:reply, {:error, error}, state} + _ -> {:reply, {:ok, body}, state} + end + rescue + _ -> {:reply, {:ok, body}, state} + end - try do - body_map = body |> Poison.decode!() + {:error, %Mint.TransportError{reason: reason}} when reason in [:econnrefused, :nxdomain] -> + raise "#{inspect(reason)}: is ipfs-pinner up?" - case body_map do - %{"error" => error} -> {:reply, {:error, error}, state} - _ -> {:reply, {:ok, body}, state} - end - rescue - _ -> {:reply, {:ok, body}, state} + {:error, err} -> + {:reply, {:error, err}, state} end end diff --git a/lib/rudder/pipeline.ex b/lib/rudder/pipeline.ex index a5cc9ce7..4767eddf 100644 --- a/lib/rudder/pipeline.ex +++ b/lib/rudder/pipeline.ex @@ -45,7 +45,7 @@ defmodule Rudder.Pipeline do start_pipeline_ms = System.monotonic_time(:millisecond) try do - with [_chain_id, block_height, _block_hash, specimen_hash] <- String.split(bsp_key, "_"), + with [_chain_id, _block_height, _block_hash, specimen_hash] <- String.split(bsp_key, "_"), {:ok, specimen} <- Rudder.IPFSInteractor.discover_block_specimen(urls), {:ok, decoded_specimen} <- Rudder.Avro.BlockSpecimen.decode(specimen), {:ok, block_specimen} <- extract_block_specimen(decoded_specimen), diff --git a/lib/rudder/proof_chain/block_specimen_event_listener.ex b/lib/rudder/proof_chain/block_specimen_event_listener.ex index addeea0f..fcf80d42 100644 --- a/lib/rudder/proof_chain/block_specimen_event_listener.ex +++ b/lib/rudder/proof_chain/block_specimen_event_listener.ex @@ -105,8 +105,8 @@ defmodule Rudder.ProofChain.BlockSpecimenEventListener do Rudder.Network.EthereumMainnet.eth_getLogs([ %{ address: proofchain_address, - fromBlock: block_height, - toBlock: block_height, + fromBlock: "0x" <> Integer.to_string(block_height, 16), + toBlock: "0x" <> Integer.to_string(block_height, 16), topics: [@bsp_awarded_event_hash] } ]) @@ -123,12 +123,14 @@ defmodule Rudder.ProofChain.BlockSpecimenEventListener do defp loop(curr_block_height) do {:ok, latest_block_number} = Rudder.Network.EthereumMainnet.eth_blockNumber() - Logger.info("curr_block: #{curr_block_height} and latest_block_num:#{latest_block_number}") if curr_block_height > latest_block_number do + Logger.info("synced to latest; waiting for #{curr_block_height} to be mined") # ~12 seconds is mining time of one moonbeam block :timer.sleep(12_000) loop(curr_block_height) + else + Logger.info("curr_block: #{curr_block_height} and latest_block_num:#{latest_block_number}") end end end diff --git a/lib/rudder/proof_chain/proof_chain_interactor.ex b/lib/rudder/proof_chain/proof_chain_interactor.ex index f1c988e4..ccf76daa 100644 --- a/lib/rudder/proof_chain/proof_chain_interactor.ex +++ b/lib/rudder/proof_chain/proof_chain_interactor.ex @@ -20,18 +20,13 @@ defmodule Rudder.ProofChain.Interactor do GenServer.start_link(__MODULE__, [], name: __MODULE__) end - # "Block height is out of bounds for live sync" - # "Sender is not BLOCK_RESULT_PRODUCER_ROLE" - # "Invalid chain ID" - # "Invalid block height" - # "Session submissions have closed" - # "Max submissions limit exceeded" - # "Operator already submitted for the provided block hash" - # "Insufficiently staked to submit" + defp get_proofchain() do + proofchain_address = Application.get_env(:rudder, :proofchain_address) + Rudder.RPC.PublicKeyHash.parse(proofchain_address) + end defp make_call(data) do - proofchain_address = Application.get_env(:rudder, :proofchain_address) - {:ok, to} = Rudder.PublicKeyHash.parse(proofchain_address) + {:ok, to} = get_proofchain() tx = [ from: nil, @@ -67,6 +62,88 @@ defmodule Rudder.ProofChain.Interactor do Rudder.Wallet.load(Base.decode16!(operator_private_key, case: :lower)) end + defp send_eip1559_signed_tx( + sender, + nonce, + to, + estimated_gas_limit, + data, + proofchain_chain_id, + max_priority_fee_per_gas_hex + ) do + {:ok, block} = Rudder.Network.EthereumMainnet.eth_getBlockByNumber(:latest) + base_fee = block.base_fee_per_gas + "0x" <> max_priority_fee_per_gas_hex = max_priority_fee_per_gas_hex + {max_priority_fee_per_gas, _} = Integer.parse(max_priority_fee_per_gas_hex, 16) + max_fee_per_gas = 2 * base_fee + max_priority_fee_per_gas + + tx = %Rudder.RPC.EthereumClient.TransactionEIP1559{ + type: 2, + nonce: nonce, + to: to, + gas_limit: estimated_gas_limit, + max_fee_per_gas: max_fee_per_gas, + max_priority_fee_per_gas: max_priority_fee_per_gas, + value: 0, + data: data, + chain_id: proofchain_chain_id + } + + Rudder.RPC.EthereumClient.TransactionEIP1559.signed_by(tx, sender) + end + + defp get_eip1559_signed_tx(sender, nonce, to, estimated_gas_limit, data, proofchain_chain_id) do + case proofchain_chain_id do + # case for testing via hardhat node in absence of maxPriorityFeePerGas support + 31_337 -> + {:ok, fee_history} = Rudder.Network.EthereumMainnet.eth_feeHistory() + fee_history_list = Map.to_list(fee_history) + + max_priority_fee_per_gas_hex = + List.last(List.last(Tuple.to_list(List.first(fee_history_list)))) + + send_eip1559_signed_tx( + sender, + nonce, + to, + estimated_gas_limit, + data, + proofchain_chain_id, + max_priority_fee_per_gas_hex + ) + + _ -> + {:ok, max_priority_fee_per_gas_hex} = + Rudder.Network.EthereumMainnet.eth_maxPriorityFeePerGas() + + send_eip1559_signed_tx( + sender, + nonce, + to, + estimated_gas_limit, + data, + proofchain_chain_id, + max_priority_fee_per_gas_hex + ) + end + end + + defp get_legacy_signed_tx(sender, nonce, to, estimated_gas_limit, data, proofchain_chain_id) do + gas_price = Rudder.Network.EthereumMainnet.eth_gasPrice!() + + tx = %Rudder.RPC.EthereumClient.Transaction{ + nonce: nonce, + gas_price: gas_price, + gas_limit: estimated_gas_limit, + to: to, + value: 0, + data: data, + chain_id: proofchain_chain_id + } + + Rudder.RPC.EthereumClient.Transaction.signed_by(tx, sender) + end + @spec submit_block_result_proof(any, any, any, any, any) :: any def submit_block_result_proof( chain_id, @@ -87,8 +164,7 @@ defmodule Rudder.ProofChain.Interactor do ]) sender = get_operator_wallet() - proofchain_address = Application.get_env(:rudder, :proofchain_address) - {:ok, to} = Rudder.PublicKeyHash.parse(proofchain_address) + {:ok, to} = get_proofchain() {:ok, recent_gas_limit} = Rudder.Network.EthereumMainnet.gas_limit(:latest) @@ -102,21 +178,10 @@ defmodule Rudder.ProofChain.Interactor do ) nonce = Rudder.Network.EthereumMainnet.next_nonce(sender.address) - gas_price = Rudder.Network.EthereumMainnet.eth_gasPrice!() - - chain_id = Application.get_env(:rudder, :proofchain_chain_id) - - tx = %Rudder.RPC.EthereumClient.Transaction{ - nonce: nonce, - gas_price: gas_price, - gas_limit: estimated_gas_limit, - to: to, - value: 0, - data: data, - chain_id: chain_id - } + proofchain_chain_id = Application.get_env(:rudder, :proofchain_chain_id) - signed_tx = Rudder.RPC.EthereumClient.Transaction.signed_by(tx, sender) + signed_tx = + get_eip1559_signed_tx(sender, nonce, to, estimated_gas_limit, data, proofchain_chain_id) with {:ok, txid} <- Rudder.Network.EthereumMainnet.eth_sendTransaction(signed_tx) do :ok = Events.brp_proof(System.monotonic_time(:millisecond) - start_proof_ms) diff --git a/lib/rudder/rpc/ethereum_client/codec.ex b/lib/rudder/rpc/ethereum_client/codec.ex index 818ac5b9..bbd95d9d 100644 --- a/lib/rudder/rpc/ethereum_client/codec.ex +++ b/lib/rudder/rpc/ethereum_client/codec.ex @@ -1,5 +1,6 @@ defmodule Rudder.RPC.EthereumClient.Codec do alias Rudder.RPC.EthereumClient.Transaction + alias Rudder.RPC.EthereumClient.TransactionEIP1559 def encode_sha256(nil), do: nil def encode_sha256(%Rudder.SHA256{bytes: bytes}), do: encode_sha256(bytes) @@ -12,7 +13,7 @@ defmodule Rudder.RPC.EthereumClient.Codec do end def encode_address( - %Rudder.PublicKeyHash{format: :ethpub, namespace: 0, bytes: bytes}, + %Rudder.RPC.PublicKeyHash{format: :ethpub, namespace: 0, bytes: bytes}, opts \\ [] ) do if Keyword.get(opts, :raw, false) do @@ -28,43 +29,22 @@ defmodule Rudder.RPC.EthereumClient.Codec do end end - @spec encode_slot_key_path(any) :: <<_::16, _::_*8>> - def encode_slot_key_path(parts) do - slug = - Enum.map(parts, fn - %Rudder.SHA256{bytes: bytes} -> bytes - %Rudder.PublicKeyHash{bytes: bytes} -> bytes - bin when is_binary(bin) -> bin - n when is_integer(n) -> :binary.encode_unsigned(n) - end) - |> Enum.map_join("", &String.pad_leading(&1, 32, <<0>>)) - - # |> Enum.map(&String.pad_leading(&1, 32, <<0>>)) - # |> Enum.join("") - - mem_hash = - case byte_size(slug) do - 0 -> <<0::256>> - 32 -> slug - n when n > 32 -> ExKeccak.hash_256(slug) - end - - mem_hash - |> String.trim_leading(<<0>>) - |> encode_bin() - end - @spec decode_address(nil | bitstring) :: nil - | %Rudder.PublicKeyHash{bytes: bitstring, chain_id: nil, format: :ethpub, namespace: 0} + | %Rudder.RPC.PublicKeyHash{ + bytes: bitstring, + chain_id: nil, + format: :ethpub, + namespace: 0 + } def decode_address(nil), do: nil def decode_address("") do - %Rudder.PublicKeyHash{format: :ethpub, namespace: 0, bytes: <<0::160>>} + %Rudder.RPC.PublicKeyHash{format: :ethpub, namespace: 0, bytes: <<0::160>>} end def decode_address(bytes) when byte_size(bytes) == 20 do - %Rudder.PublicKeyHash{format: :ethpub, namespace: 0, bytes: bytes} + %Rudder.RPC.PublicKeyHash{format: :ethpub, namespace: 0, bytes: bytes} end def decode_address(<<"0x", _::binary>> = bin) do @@ -89,6 +69,15 @@ defmodule Rudder.RPC.EthereumClient.Codec do |> encode_bin() end + def encode_transaction(%TransactionEIP1559{} = tx) do + hex_paylod = + TransactionEIP1559.to_rlp(tx) + |> Base.encode16(case: :lower) + + tx_type = "0x02" + tx_type <> hex_paylod + end + @spec encode_call_transaction(keyword) :: any def encode_call_transaction(call_tx) do call_tx @@ -139,7 +128,7 @@ defmodule Rudder.RPC.EthereumClient.Codec do end defp normalize_hl_call_payload_part(%Rudder.SHA256{bytes: bytes}), do: bytes - defp normalize_hl_call_payload_part(%Rudder.PublicKeyHash{bytes: bytes}), do: bytes + defp normalize_hl_call_payload_part(%Rudder.RPC.PublicKeyHash{bytes: bytes}), do: bytes defp normalize_hl_call_payload_part(other), do: other @spec encode_call_payload(binary | {binary | ABI.FunctionSelector.t(), any}) :: @@ -248,12 +237,9 @@ defmodule Rudder.RPC.EthereumClient.Codec do txs -> {nil, txs} end - mining_cost = - if discovered_on_network.ingest_mining_costs(), - do: decode_qty(block["difficulty"]) - block = [ discovered_on_network: discovered_on_network, + base_fee_per_gas: decode_qty(block["baseFeePerGas"]), hash: decode_sha256(block["hash"]), signed_at: decode_timestamp(block["timestamp"]), namespace: 0, @@ -262,7 +248,6 @@ defmodule Rudder.RPC.EthereumClient.Codec do uncles: decode_uncles(block["uncles"]), extra_data: decode_bin(block["extraData"]), miner: decode_address(block["miner"]), - mining_cost: mining_cost, gas_limit: decode_qty(block["gasLimit"]), gas_used: decode_qty(block["gasUsed"]), transaction_ids: transaction_ids, @@ -369,7 +354,7 @@ defmodule Rudder.RPC.EthereumClient.Codec do end def extract_address_from_log_topic(%Rudder.SHA256{bytes: <<0::96, addr::binary-size(20)>>}), - do: %Rudder.PublicKeyHash{format: :ethpub, namespace: 0, bytes: addr} + do: %Rudder.RPC.PublicKeyHash{format: :ethpub, namespace: 0, bytes: addr} def linearize_log_offsets(logs) do Enum.sort_by(logs, fn log -> diff --git a/lib/rudder/rpc/ethereum_client/ethereum_client.ex b/lib/rudder/rpc/ethereum_client/ethereum_client.ex index 734a87bc..58fa1a1d 100644 --- a/lib/rudder/rpc/ethereum_client/ethereum_client.ex +++ b/lib/rudder/rpc/ethereum_client/ethereum_client.ex @@ -5,6 +5,7 @@ defmodule Rudder.RPC.EthereumClient do use Confex, used_with_opts alias Rudder.RPC.EthereumClient.Codec alias Rudder.RPC.EthereumClient.Transaction + alias Rudder.RPC.EthereumClient.TransactionEIP1559 @default_client_module Rudder.RPC.JSONRPC.HTTPClient @default_client_opts "http://localhost:8545" @@ -49,7 +50,7 @@ defmodule Rudder.RPC.EthereumClient do def sealer do case Keyword.fetch(config(), :sealer) do {:ok, sealer_addr_str} -> - Rudder.PublicKeyHash.parse!(sealer_addr_str) + Rudder.RPC.PublicKeyHash.parse!(sealer_addr_str) :error -> nil @@ -158,6 +159,26 @@ defmodule Rudder.RPC.EthereumClient do ) end + def eth_maxPriorityFeePerGas(opts \\ []) do + call( + :eth_maxPriorityFeePerGas, + opts, + nil + ) + end + + def eth_feeHistory(opts \\ []) do + call( + :eth_feeHistory, + [ + Codec.encode_qty(5), + :latest, + [] + ], + nil + ) + end + def eth_getTransactionByHash(hash, opts \\ []), do: call( @@ -229,7 +250,13 @@ defmodule Rudder.RPC.EthereumClient do case call_tx do %Transaction{} = tx -> call( - # :eth_sendTransaction, + :eth_sendRawTransaction, + [Codec.encode_transaction(tx)], + & &1 + ) + + %TransactionEIP1559{} = tx -> + call( :eth_sendRawTransaction, [Codec.encode_transaction(tx)], & &1 @@ -312,7 +339,7 @@ defmodule Rudder.RPC.EthereumClient do def next_nonce(nil), do: 0 - def next_nonce(%Rudder.PublicKeyHash{} = pkh) do + def next_nonce(%Rudder.RPC.PublicKeyHash{} = pkh) do eth_getTransactionCount!(pkh, :pending) end @@ -335,6 +362,7 @@ defmodule Rudder.RPC.EthereumClient do def eth_getBlockByHash!(hash, opts \\ []), do: unwrap!(eth_getBlockByHash(hash, opts)) def eth_getLogs!(opts \\ []), do: unwrap!(eth_getLogs(opts)) + def eth_maxPriorityFeePerGas!(opts \\ []), do: unwrap!(eth_maxPriorityFeePerGas(opts)) def eth_getTransactionByHash!(hash), do: unwrap!(eth_getTransactionByHash(hash)) def eth_gasPrice!, do: unwrap!(eth_gasPrice()) diff --git a/lib/rudder/rpc/ethereum_client/public_key_hash.ex b/lib/rudder/rpc/ethereum_client/public_key_hash.ex index b9c84900..4afcc998 100644 --- a/lib/rudder/rpc/ethereum_client/public_key_hash.ex +++ b/lib/rudder/rpc/ethereum_client/public_key_hash.ex @@ -1,4 +1,4 @@ -defmodule Rudder.PublicKeyHash do +defmodule Rudder.RPC.PublicKeyHash do defstruct format: :p2pkh, namespace: 0, chain_id: nil, @@ -12,7 +12,7 @@ defmodule Rudder.PublicKeyHash do @zero_address EthereumCodec.decode_address("0x0000000000000000000000000000000000000000") - @spec zero_address :: %Rudder.PublicKeyHash{ + @spec zero_address :: %Rudder.RPC.PublicKeyHash{ bytes: <<_::160>>, chain_id: nil, format: :ethpub, @@ -43,7 +43,7 @@ defmodule Rudder.PublicKeyHash do :error | {:ok, nil - | %Rudder.PublicKeyHash{ + | %Rudder.RPC.PublicKeyHash{ bytes: bitstring, chain_id: nil, format: :ethpub, @@ -57,13 +57,18 @@ defmodule Rudder.PublicKeyHash do @spec parse_raw!(any) :: nil - | %Rudder.PublicKeyHash{bytes: bitstring, chain_id: nil, format: :ethpub, namespace: 0} + | %Rudder.RPC.PublicKeyHash{ + bytes: bitstring, + chain_id: nil, + format: :ethpub, + namespace: 0 + } def parse_raw!(v) do {:ok, pkh} = parse_raw(v) pkh end - @spec new(any, any, any, any) :: %Rudder.PublicKeyHash{ + @spec new(any, any, any, any) :: %Rudder.RPC.PublicKeyHash{ bytes: any, chain_id: any, format: any, @@ -106,7 +111,7 @@ defmodule Rudder.PublicKeyHash do :error | {:ok, nil - | %Rudder.PublicKeyHash{ + | %Rudder.RPC.PublicKeyHash{ bytes: bitstring, chain_id: nil, format: :ethpub, @@ -125,26 +130,26 @@ defmodule Rudder.PublicKeyHash do def equal?(a, b), do: a == b end -defimpl Jason.Encoder, for: Rudder.PublicKeyHash do - @spec encode(%Rudder.PublicKeyHash{}, Jason.Encode.opts()) :: [ +defimpl Jason.Encoder, for: Rudder.RPC.PublicKeyHash do + @spec encode(%Rudder.RPC.PublicKeyHash{}, Jason.Encode.opts()) :: [ binary | maybe_improper_list(any, binary | []) | byte, ... ] - def encode(%Rudder.PublicKeyHash{} = pkh, opts) do - Rudder.PublicKeyHash.as_string!(pkh) + def encode(%Rudder.RPC.PublicKeyHash{} = pkh, opts) do + Rudder.RPC.PublicKeyHash.as_string!(pkh) |> Jason.Encode.string(opts) end end -defimpl String.Chars, for: Rudder.PublicKeyHash do - @spec to_string(%Rudder.PublicKeyHash{}) :: any - def to_string(pkh), do: Rudder.PublicKeyHash.as_string!(pkh) +defimpl String.Chars, for: Rudder.RPC.PublicKeyHash do + @spec to_string(%Rudder.RPC.PublicKeyHash{}) :: any + def to_string(pkh), do: Rudder.RPC.PublicKeyHash.as_string!(pkh) end -defimpl Inspect, for: Rudder.PublicKeyHash do +defimpl Inspect, for: Rudder.RPC.PublicKeyHash do import Inspect.Algebra - @spec inspect(%Rudder.PublicKeyHash{}, Inspect.Opts.t()) :: + @spec inspect(%Rudder.RPC.PublicKeyHash{}, Inspect.Opts.t()) :: :doc_line | :doc_nil | binary @@ -153,10 +158,10 @@ defimpl Inspect, for: Rudder.PublicKeyHash do | {:doc_break | :doc_color | :doc_cons | :doc_fits | :doc_group | :doc_string, any, any} | {:doc_nest, any, :cursor | :reset | non_neg_integer, :always | :break} def inspect( - %Rudder.PublicKeyHash{format: format, namespace: namespace, chain_id: chain_id} = pkh, + %Rudder.RPC.PublicKeyHash{format: format, namespace: namespace, chain_id: chain_id} = pkh, opts ) do - {:ok, str_repr} = Rudder.PublicKeyHash.as_string(pkh, hide_prefix: true) + {:ok, str_repr} = Rudder.RPC.PublicKeyHash.as_string(pkh, hide_prefix: true) str_repr_doc = str_repr_doc(str_repr, opts) diff --git a/lib/rudder/rpc/ethereum_client/transaction.ex b/lib/rudder/rpc/ethereum_client/transaction.ex index b1821644..7c627b38 100644 --- a/lib/rudder/rpc/ethereum_client/transaction.ex +++ b/lib/rudder/rpc/ethereum_client/transaction.ex @@ -15,57 +15,6 @@ defmodule Rudder.RPC.EthereumClient.Transaction do from: nil ] - def parse("0x" <> tx_rlp_hex) do - parse(Base.decode16!(tx_rlp_hex, case: :mixed)) - end - - def parse(tx_rlp_raw) when is_binary(tx_rlp_raw) do - [nonce, gas_price, gas_limit, to, value, data, v, r, s] = ExRLP.decode(tx_rlp_raw) - - v_num = :binary.decode_unsigned(v) - - %__MODULE__{ - nonce: :binary.decode_unsigned(nonce), - gas_price: :binary.decode_unsigned(gas_price), - gas_limit: :binary.decode_unsigned(gas_limit), - to: Rudder.PublicKeyHash.parse_raw!(to), - value: :binary.decode_unsigned(value), - data: data, - v: v_num, - r: r, - s: s, - chain_id: compute_chain_id(v_num) - } - end - - def parse([nonce, gas_price, gas_limit, to, value, data]) do - %__MODULE__{ - nonce: normalize_qty(nonce), - gas_price: normalize_qty(gas_price), - gas_limit: normalize_qty(gas_limit), - to: normalize_address(to), - value: normalize_qty(value), - data: normalize_bin(data) - } - end - - def parse([nonce, gas_price, gas_limit, to, value, data, v, r, s]) do - v_norm = normalize_qty(v) - - %__MODULE__{ - nonce: normalize_qty(nonce), - gas_price: normalize_qty(gas_price), - gas_limit: normalize_qty(gas_limit), - to: normalize_address(to), - value: normalize_qty(value), - data: normalize_bin(data), - v: v_norm, - r: normalize_qty(r), - s: normalize_qty(s), - chain_id: compute_chain_id(v_norm) - } - end - def to_rlpable(%__MODULE__{ nonce: nonce, gas_price: gas_price, @@ -146,12 +95,12 @@ defmodule Rudder.RPC.EthereumClient.Transaction do defp normalize_bin("0x" <> hex), do: Base.decode16!(hex, case: :mixed) defp normalize_bin(bin) when is_binary(bin), do: bin - def normalize_address(%Wallet{address: %Rudder.PublicKeyHash{} = pkh}), do: pkh - def normalize_address(other), do: Rudder.PublicKeyHash.parse_raw!(normalize_bin(other)) + def normalize_address(%Wallet{address: %Rudder.RPC.PublicKeyHash{} = pkh}), do: pkh + def normalize_address(other), do: Rudder.RPC.PublicKeyHash.parse_raw!(normalize_bin(other)) defp term_to_rlpable(nil), do: "" defp term_to_rlpable(0), do: "" - defp term_to_rlpable(%Rudder.PublicKeyHash{bytes: bin}), do: bin + defp term_to_rlpable(%Rudder.RPC.PublicKeyHash{bytes: bin}), do: bin defp term_to_rlpable(data) when is_integer(data), do: :binary.encode_unsigned(data) defp term_to_rlpable(data) when is_binary(data), do: data diff --git a/lib/rudder/rpc/ethereum_client/transaction_eip1559.ex b/lib/rudder/rpc/ethereum_client/transaction_eip1559.ex new file mode 100644 index 00000000..f0992961 --- /dev/null +++ b/lib/rudder/rpc/ethereum_client/transaction_eip1559.ex @@ -0,0 +1,129 @@ +defmodule Rudder.RPC.EthereumClient.TransactionEIP1559 do + alias Rudder.Wallet + + defstruct [ + :type, + :nonce, + :max_priority_fee_per_gas, + :max_fee_per_gas, + :to, + :gas_limit, + value: 0, + data: <<>>, + access_list: [], + signature_y_parity: <<>>, + signature_r: <<>>, + signature_s: <<>>, + chain_id: nil, + from: nil + ] + + def to_rlp(%__MODULE__{ + chain_id: chain_id, + nonce: nonce, + to: to, + gas_limit: gas_limit, + max_fee_per_gas: max_fee_per_gas, + max_priority_fee_per_gas: max_priority_fee_per_gas, + value: value, + data: data, + access_list: access_list, + signature_y_parity: signature_y_parity, + signature_r: signature_r, + signature_s: signature_s + }) do + [ + chain_id, + nonce, + max_priority_fee_per_gas, + max_fee_per_gas, + gas_limit, + to, + value, + data, + access_list, + signature_y_parity, + signature_r, + signature_s + ] + |> Enum.map(&term_to_rlpable/1) + |> ExRLP.encode() + end + + def unsigned_hash(%__MODULE__{ + chain_id: chain_id, + nonce: nonce, + gas_limit: gas_limit, + to: to, + max_fee_per_gas: max_fee_per_gas, + max_priority_fee_per_gas: max_priority_fee_per_gas, + value: value, + data: data, + access_list: access_list + }) do + rlp_encoded = + [ + chain_id, + nonce, + max_priority_fee_per_gas, + max_fee_per_gas, + gas_limit, + to, + value, + data, + access_list + ] + |> Enum.map(&term_to_rlpable/1) + |> ExRLP.encode() + + bin = :binary.encode_unsigned(0x02) <> rlp_encoded + + ExKeccak.hash_256(bin) + end + + def signed_by( + %__MODULE__{chain_id: chain_id} = tx, + %Wallet{ + address: sender_address, + private_key: sender_privkey, + public_key: sender_pubkey + } + ) do + msg_hash = unsigned_hash(tx) + + {:ok, {signature, signature_y_parity}} = ExSecp256k1.sign_compact(msg_hash, sender_privkey) + + <> = signature + + %{ + tx + | signature_y_parity: signature_y_parity, + signature_r: sig_r, + signature_s: sig_s, + from: sender_address + } + end + + defp normalize_qty(nil), do: nil + defp normalize_qty(n) when is_integer(n), do: n + defp normalize_qty("0x" <> hex), do: normalize_qty(hex) + + defp normalize_qty(hex) when is_binary(hex) do + {qty, ""} = Integer.parse(hex, 16) + qty + end + + defp normalize_bin(nil), do: nil + defp normalize_bin("0x" <> hex), do: Base.decode16!(hex, case: :mixed) + defp normalize_bin(bin) when is_binary(bin), do: bin + + def normalize_address(%Wallet{address: %Rudder.RPC.PublicKeyHash{} = pkh}), do: pkh + def normalize_address(other), do: Rudder.RPC.PublicKeyHash.parse_raw!(normalize_bin(other)) + + defp term_to_rlpable(nil), do: "" + defp term_to_rlpable(0), do: 0 + defp term_to_rlpable(%Rudder.RPC.PublicKeyHash{bytes: bin}), do: bin + defp term_to_rlpable(data) when is_integer(data), do: :binary.encode_unsigned(data) + defp term_to_rlpable(data) when is_binary(data), do: data + defp term_to_rlpable([]), do: [] +end diff --git a/lib/rudder/rpc/ethereum_client/wallet.ex b/lib/rudder/rpc/ethereum_client/wallet.ex index c0878e06..8cbd19d4 100644 --- a/lib/rudder/rpc/ethereum_client/wallet.ex +++ b/lib/rudder/rpc/ethereum_client/wallet.ex @@ -28,6 +28,6 @@ defmodule Rudder.Wallet do defp pubkey_to_address(<<4::size(8), key::binary-size(64)>>) do <<_::binary-size(12), raw_address::binary-size(20)>> = ExKeccak.hash_256(key) - Rudder.PublicKeyHash.parse_raw!(raw_address) + Rudder.RPC.PublicKeyHash.parse_raw!(raw_address) end end diff --git a/lib/rudder/rpc/moonbeam_mainnet.ex b/lib/rudder/rpc/moonbeam_mainnet.ex new file mode 100644 index 00000000..35f91d21 --- /dev/null +++ b/lib/rudder/rpc/moonbeam_mainnet.ex @@ -0,0 +1,8 @@ +defmodule Rudder.Network.MoonbeamMainnet do + use Rudder.RPC.EthereumClient, + otp_app: :rudder, + client_opts: Application.get_env(:rudder, :proofchain_node), + chain_id: Application.get_env(:rudder, :proofchain_chain_id), + description: "Moonbeam Foundation Mainnet", + currency: [name: "Glimmer", ticker_symbol: "GLMR"] +end diff --git a/mix.exs b/mix.exs index fc6e4636..ddcc7a31 100644 --- a/mix.exs +++ b/mix.exs @@ -4,7 +4,7 @@ defmodule Rudder.MixProject do def project do [ app: :rudder, - version: "0.2.6", + version: "0.2.8", elixir: "~> 1.14.3", start_permanent: Mix.env() == :prod, deps: deps() @@ -59,7 +59,7 @@ defmodule Rudder.MixProject do {:finch, "~> 0.16.0"}, {:downstream, "~> 1.0"}, {:websockex, "~> 0.4.3"}, - {:multipart, "~> 0.3.1"}, + {:multipart, "~> 0.4.0"}, # static code analysis {:dialyxir, "~> 1.0", only: [:dev], runtime: false}, diff --git a/mix.lock b/mix.lock index d435bcb1..5db40b07 100644 --- a/mix.lock +++ b/mix.lock @@ -40,7 +40,7 @@ "mimerl": {:hex, :mimerl, "1.2.0", "67e2d3f571088d5cfd3e550c383094b47159f3eee8ffa08e64106cdf5e981be3", [:rebar3], [], "hexpm", "f278585650aa581986264638ebf698f8bb19df297f66ad91b18910dfc6e19323"}, "mint": {:hex, :mint, "1.5.1", "8db5239e56738552d85af398798c80648db0e90f343c8469f6c6d8898944fb6f", [:mix], [{:castore, "~> 0.1.0 or ~> 1.0", [hex: :castore, repo: "hexpm", optional: true]}, {:hpax, "~> 0.1.1", [hex: :hpax, repo: "hexpm", optional: false]}], "hexpm", "4a63e1e76a7c3956abd2c72f370a0d0aecddc3976dea5c27eccbecfa5e7d5b1e"}, "mnemonic": {:hex, :mnemonic, "0.3.1", "35da1f5161ecfe1dd068e6e3fa50d93cc5cd7873ef937ae1880d6c71ca509337", [:mix], [{:pbkdf2_elixir, "2.0.0", [hex: :pbkdf2_elixir, repo: "hexpm", optional: false]}], "hexpm", "1681c9be68162742c91228fc406c1f5c34e8474ae0c1e952aaf8fb81efb1ebca"}, - "multipart": {:hex, :multipart, "0.3.1", "886d77125f5d7ba6be2f86e4be8f6d3556684c8e56a777753f06234885b09cde", [:mix], [{:mime, "~> 1.2 or ~> 2.0", [hex: :mime, repo: "hexpm", optional: false]}], "hexpm", "9657783995d2b9b546d9c66e1d497fcb473d813a8a3fb73faf5e411538b1db97"}, + "multipart": {:hex, :multipart, "0.4.0", "634880a2148d4555d050963373d0e3bbb44a55b2badd87fa8623166172e9cda0", [:mix], [{:mime, "~> 1.2 or ~> 2.0", [hex: :mime, repo: "hexpm", optional: false]}], "hexpm", "3c5604bc2fb17b3137e5d2abdf5dacc2647e60c5cc6634b102cf1aef75a06f0a"}, "nimble_options": {:hex, :nimble_options, "1.0.2", "92098a74df0072ff37d0c12ace58574d26880e522c22801437151a159392270e", [:mix], [], "hexpm", "fd12a8db2021036ce12a309f26f564ec367373265b53e25403f0ee697380f1b8"}, "nimble_pool": {:hex, :nimble_pool, "1.0.0", "5eb82705d138f4dd4423f69ceb19ac667b3b492ae570c9f5c900bb3d2f50a847", [:mix], [], "hexpm", "80be3b882d2d351882256087078e1b1952a28bf98d0a287be87e4a24a710b67a"}, "off_broadway_redis": {:hex, :off_broadway_redis, "0.4.3", "e45564bc2e3269771e8d941cae1583536ebd0900d1ff7dfa64559c97b4ae9b04", [:mix], [{:broadway, "~> 0.6.2", [hex: :broadway, repo: "hexpm", optional: false]}, {:redix, ">= 0.11.1 and < 1.1.0", [hex: :redix, repo: "hexpm", optional: false]}], "hexpm", "74a79519f3a1f5bf92130c361f240b9dad076276d84f0785b5eaa62048055daf"}, diff --git a/test/evm/evm_test.exs b/test/evm/evm_test.exs index 3d90e698..0b39379d 100644 --- a/test/evm/evm_test.exs +++ b/test/evm/evm_test.exs @@ -31,12 +31,23 @@ defmodule SupervisionTreeTest do }) end - test "query-ing server at wrong port" do + # test "query-ing server at wrong port" do + # block_id = "1234_f_" + + # specimen = get_sample_specimen!() + # {:ok, bpid} = Rudder.BlockProcessor.start_link(["http://127.0.0.1:3100"]) + # {:error, errormsg} = GenServer.call(bpid, {:process, "dfjkejkjfd"}, 60_000) + # end + + test "handles block processor server error" do block_id = "1234_f_" specimen = get_sample_specimen!() - {:ok, bpid} = Rudder.BlockProcessor.start_link(["http://127.0.0.1:3100"]) - {:error, errormsg} = GenServer.call(bpid, {:process, "dfjkejkjfd"}, 60_000) + {:ok, bpid} = Rudder.BlockProcessor.start_link(["http://evm-server:3002"]) + {:error, error_msg} = GenServer.call(bpid, {:process, "invalid content"}, 60_000) + + assert error_msg == + "ERROR(10): error unmarshalling replica file: invalid character 'i' looking for beginning of value" end def get_sample_specimen!() do