FAQ Hermes v1
Hi Cosmonauts !
This discussion is dedicated to offering answers to frequently asked questions from users. Feel free to ask any question, share and contribute as well !
Multiple instances of Hermes can not access to the same wallet concurrently. They would need to coordinate their actions in order to broadcast transactions. If two instances of Hermes operate on the same chain, they need to access different wallets. If you want to run an instance for each channel between two chains, you will need a different pair of wallets for every instance.
As part of the mode.client, Hermes has a refresh feature.
If this feature is enabled, then Hermes will periodically send a UpdateClient message to all clients that are active.
An active client is defined as a client that an open channel is using. For instance, consider two networks ibc-0 and ibc-1. Suppose there are 3 clients that network ibc-1 is hosting for ibc-0: (1) 07-tendermint-0, 07-tendermint-1, 07-tendermint-2. Suppose there is a single channel open between the two networks: ibc-0 : channel-1 - ibc-1 : channel-2, which is using client 07-tendermint-2 on ibc-1. This means that the other two clients (07-tendermint-0 and 07-tendermint-1) are not in use by any open channel. Upon starting Hermes, those two clients are not considered active and Hermes will not periodically refresh them.
The chains.packet_filter feature will affect which is the set of clients that Hermes considers to be active. If you deny a certain channel, then Hermes will not refresh the clients underlying that channel.
To debug or understand if your instance of Hermes is correctly performing client refresh, there are 2 ways: inspect the Hermes logs, or inspect the workers.
The line you want to look for is as follows:
2022-07-29T09:19:48.012316Z INFO ThreadId(01) spawned client worker: client::ibc-1->ibc-0:07-tendermint-0
This means that Hermes spawned the client worker for the client existing on chain ibc-0 with identifier 07-tendermint-0, which is verifying headers of network ibc-1.
This log line appears only when Hermes starts-up.
This method triggers Hermes to dump the state of all its workers, which can then inspect in the logs. To trigger, send a USR1 signal to the Hermes process. Documented here: https://hermes.informal.systems/help.html#inspecting-the-relayer-state
Briefly, no. The default values in config.toml are not for production, they are just basic heuristics that work most of the time, but in production environments these config options for each chain will require some tuning (depending on the chain). Here are two examples of production configuration (Thanks to CryptoCrew Validators and Notional Labs), which might help:
- https://github.com/clemensgg/RELAYER-dev-crew/blob/main/HERMES/omniflix/relayer-doc.md
- https://github.com/notional-labs/notional/blob/master/relaying/hermes/all-ibc.toml
- https://github.com/cephalopodequipment/config/blob/main/hermes/manual.toml
- https://github.com/bro-n-bro/bro_infra/blob/main/docs/hermes_config.toml
For testnets, it depends on the networks. For Agoric specifically, here are some details: https://github.com/Agoric/agoric-sdk/blob/6b0deac341424da9b1e8badfbf307c37f16b4618/packages/pegasus/demo.md
- Run Hermes debug like below which will print the RPC requests sent by Hermes and then replicate the request manually (eg. using curl)
RUST_LOG='tendermint_rpc=debug,ibc_relayer=debug,ibc_relayer_cli=debug' hermes start
- Check
abci_infoof the full node
curl <rpc_addr>/abci_info
When clearing packets with Hermes, if there are multiple packets which are found but the command doesn't relay any packets, e.g.
This could mean that the node used to retrieve the packets doesn’t have a history long enough.
Verify how old the packets are by first retrieving the sequence number of a packet using:
hermes query packet pending-acks --chain likecoin-mainnet-2 --channel channel-3 --port transfer
Once the sequence number is retrieved, the Tx hash needs to be fetched by going to the following link: https://mainnet-node.like.co/rpc/tx_search?query=%22send_packet.packet_sequence=%27<sequence_number>%27%22
Once you have the Tx hash, you can use mintscan to get information on the transaction, e.g. https://www.mintscan.io/likecoin/txs/<Tx hash>
If the packet is indeed too old, one solution would be to use an Archival Node to clear the pending packets.
As of Hermes v1.2, for more information refer the Uncleared packets section added to the Hermes guide, under Troubleshooting, https://hermes.informal.systems/advanced/troubleshooting/cross-comp-config.html.
File permalink https://github.com/informalsystems/hermes/blob/master/guide/src/advanced/troubleshooting/cross-comp-config.md
You can use Hermes to relay IBC packets between Wasm contracts on separate networks. External teams maintain excellent documentation on how to achieve this. Please visit:
- https://github.com/0xekez/cw-ibc-example/#background
- https://github.com/public-awesome/ics721/tree/main
If you're a module/network developer, this is a great point to start: https://github.com/CosmWasm/cosmwasm/blob/main/IBC.md
Hermes has a misbehaviour detection mechanism. By default, this is disabled. Setting it to true will make Hermes monitor all the IBC clients underlying all channels on which it is actively relaying packets.
At a high level, misbehaviour detection works as follows:
- given a IBC client X
- suppose client X is hosted on network Hub
- suppose client X verifies headers of network Osmosis
- given a header H1 at height
hthat client X installed (this header appears in events that Hub nodes are emitting on the Websocket, which Hermes receives) - given header H2 of height
hthat a full node of Osmosis reports to have - Hermes cross-checks that H1 == H2
- if the two headers are equal, all good
- if they're not equal, that's an evidence of misbehaviour, concretely there was a fork of Osmosis, of the client X was tricked to install a bad header, hence client X should be frozen