The purpose of this index is to provide an overview of the testing approaches to be implemented by Ziggurat. It is intended to evolve as the framework matures, leaving room for novel cases and extensions of existing cases, as called for by any protocol idiosyncrasies that may come to light during the development process.
Some test cases have been consolidated when similar behaviour is tested with differing messages. The final implementation of these cases will be subject to factors such as node setup and teardown details, test run time (and potentially runtime) constraints, readability and maintainability.
Some of these tests can be executed against the same node instance without resetting or clearing the cache to cut down on node setup and teardown times. Other tests are intended for use where a new peer is needed to clean the slate for a deterministic output.
For load testing, "reasonable load" and "heavy load" will need to be defined.
The tests can be run with cargo test once Ziggurat is properly configured and dependencies (node instance to be tested) are satisfied. See the README for details.
Tests are grouped into the following categories: conformance, performance, and resistance. Each test is named after the category it belongs to, in addition to what's being tested. For example, c001_handshake_when_node_receives_connection is the first conformance test and tests the handshake behavior on the receiving end. The full naming convention is: id_part_t(subtest_no)_(message type)_(extra_test_desc).
The conformance tests aim to verify the node adheres to the network protocol. In addition, they include some naive error cases with malicious and fuzzing cases consigned to the resistance tests. Most cases in this section will only require a socket standing in for the connected peer and a full node running in the background.
These tests verify the proper execution of a handshake between a node and a peer as well as some simple error cases.
These tests verify the node responds with the correct messages to requests and disconnects in certain trivial non-fuzz, non-malicious cases. These form the basic assumptions necessary for peering and syncing.
These tests aim to evaluate the proper behaviour of a node when receiving unsolicited messages post-handshake.
These tests evaluate the node's basic peering properties by verifying the data included in the messages are in accordance with the peering status of the node.
These tests evaluate the node's basic syncing properties for transactions and blocks by verifying the data included in the message payloads are in accordance with the ranges provided by the peer.
The performance tests aim to verify the node maintains a healthy throughput under pressure. This is principally done through simulating load with synthetic peers and evaluating the node's responsiveness. Synthetic peers will need to be able to simulate the behaviour of a full node by implementing handshaking, message sending and receiving.
These tests are intended to verify the node remains healthy under "reasonable load". Additionally these tests will be pushed to the extreme for resistance testing with heavier loads.
These tests are meant to explore the impact of malicious network use against a node.
The amount of load and its frequency could be modulated to provide a comprehensive verification of the node's behaviour under different conditions (including synchronized requests from different peers and other worst case scenarios).
The resistance tests are designed for the early detection and avoidance of weaknesses exploitable through malicious behaviour. They attempt to probe boundary conditions with comprehensive fuzz testing and extreme load testing. The nature of the peers in these cases will depend on how accurately they needs to simulate node behaviour. It will likely be a mixture of simple sockets for the simple cases and peers used in the performance tests for the more advanced.
The fuzz tests aim to buttress the message conformance tests with extra verification of expected node behaviour when receiving corrupted or broken messages. Our approach is targeting these specific areas and we anticipate broadening these test scenarios as necessary:
- Messages with any length and any content (random bytes).
- Messages with plausible lengths, e.g. 24 bytes for header and within the expected range for the body.
- Metadata-compliant messages, e.g. correct header, random body.
- Slightly corrupted but otherwise valid messages, e.g. N% of body replaced with random bytes.
- Messages with an incorrect checksum.
- Messages with differing announced and actual lengths.
The test index makes use of symbolic language in describing connection and message sending directions.
| Symbol | Meaning |
|---|---|
-> A |
Ziggurat's synthetic node sends a message A to Rippled |
<- B |
Rippled sends a message B to Ziggurat's synthetic node |
>> C |
Ziggurat's synthetic node broadcasts a message C to all its peers |
<< D |
Rippled broadcasts a message D to all its peers |
<> |
Signifies a completed handshake, in either direction |
The node correctly performs a handshake from the responder side.
->
-> public key & session signature
<- public key & session signature
Assert: the node’s peer count has increased to 1 and the synthetic node is an established peer.
The node correctly performs a handshake from the initiator side.
<-
<- public key & session signature
-> public key & session signature
Assert: the node’s peer count has increased to 1 and the synthetic node is an established peer.
1. The node responds with `pong` message for `ping`.
<>
-> ping message with random `sequence` number
<- pong response with the same `sequence` number
2. The node sends `ping` message within the specified timeout. A synthetic node responds with correct `pong` and ensures
that it was not disconnected.
<>
<- ping request
-> pong response
Assert: The nodes stay connected.
3. The node receives an unsolicited `pong` message and disconnects sender.
<>
-> pong response with made up `sequence` number
Assert: the node disconnects sender after the specified timeout.
The node responds with mtLEDGER_DATA for mtGET_LEDGER with different iType types.
iType types used here are LiBase and LiAsNode.
<>
-> mtGET_LEDGER (iType)
<- mtLEDGER_DATA
The node requests mtGET_PEER_SHARD_INFO_V2 after connection and handshake.
<>
<- mtGET_PEER_SHARD_INFO_V2
The node should *NOT* send any messages after connection if there was no handshake.
The test waits for the predefined amount of time, ensuring no messages were received.
The node should respond with transaction details after receiveing mtGET_OBJECTS / OtTransactions request.
Normally the node does not respond with transaction details if the transaction is not in its cache. In this test we first
query for transaction details via rpc, then via peer protocol.
<>
-> mtGET_OBJECTS with r#type == OtTransactions
<- mtTRANSACTIONS
The node should query for the transaction object after receiving a mtHAVE_TRANSACTIONS packet.
<>
-> mtHAVE_TRANSACTIONS
<- mtGET_OBJECTS
The node should ignore the squelch message for its validator public key.
<>
<- mtPROPOSE_LEDGER (node public key 1)
-> mtSQUELCH (node public key 1)
<- mtPROPOSE_LEDGER (node public key 1)
Assert: the synthetic node has continued receiving mtPROPOSE_LEDGER messages.
The node should send mtSTATUS_CHANGE message containing ledger info.
This message should be sent without any explicit requests.
To ensure ledger correctness, the test asks for its information via RPC first
and compares the results with the mtSTATUS_CHANGE payload.
<>
<- mtSTATUS_CHANGE with the correct ledger hash and sequence
The node should relay a mtGET_PEER_SHARD_INFO_V2 message to connected peers
if it contains a valid key type and relay counter is higher than 0.
Connection scenario:
Synthetic Node 1 <> Rippled <> Synthetic Node 2
This test checks whether Synthetic node 2 receives the mtGET_PEER_SHARD_INFO_V2
message sent from Synthetic Node 1 to the Ripple node.
The node should not relay a mtGET_PEER_SHARD_INFO_V2 message to connected peers
if it contains unsupported key type.
Connection scenario:
Synthetic Node 1 <> Rippled <> Synthetic Node 2
This test ensures that Synthetic node 2 does not receive the mtGET_PEER_SHARD_INFO_V2
message sent from Synthetic Node 1 to the Ripple node.
The node should not relay a mtGET_PEER_SHARD_INFO_V2 message to connected peers
if the message's relay count is equal 0.
Connection scenario:
Synthetic Node 1 <> Rippled <> Synthetic Node 2
This test ensures that Synthetic node 2 does not receive the mtGET_PEER_SHARD_INFO_V2
message sent from Synthetic Node 1 to the Ripple node.
The node should not relay a mtGET_PEER_SHARD_INFO_V2 message to connected peers
if the message's relay count is above relay limit (currently 3).
Connection scenario:
Synthetic Node 1 <> Rippled <> Synthetic Node 2
This test ensures that Synthetic node 2 does not receive the mtGET_PEER_SHARD_INFO_V2
message sent from Synthetic Node 1 to the Ripple node.
The node should send a mtVALIDATORLISTCOLLECTION message containing at least one validator
with a correct public key and a non-empty manifest.
<>
<- mtVALIDATORLISTCOLLECTION with at least one validator with a correct public key and a non-empty manifest.
Deploy a multi-node network setup with nodes 1, 2 and 3.
Connect a synthetic node to Node 1.
Let A be a list of node public keys 1, 2 and 3.
Let B be a list of node public keys 2 and 3.
<> with Node1
<< mtPROPOSE_LEDGER (A)
-> mtSQUELCH (B)
<< mtPROPOSE_LEDGER (A)
Assert: A synthetic node receives only mtPROPOSE_LEDGER messages with a key from node 1
after squelching node public keys belonging to nodes 2 and 3 (B).
The node sends mtMANIFESTS after the handshake.
<>
<- mtMANIFESTS
The node sends mtENDPOINTS after the handshake.
<>
<- mtENDPOINTS
Nodes in the testnet should relay a mtTRANSACTION message to connected peers.
Connection scenario:
RPC call > Rippled 1 <> Rippled 2 <> Synthetic Node
This test checks whether the synthetic node receives the mtTRANSACTION
message containing details from the RPC call.
Nodes in the testnet should broadcast a mtHAVE_SET message to connected peers after a transaction.
Connection scenario:
Submit transaction via RPC call > Rippled 1 <> Rippled 2 <> Synthetic Node
This test checks whether the synthetic node receives the mtHAVE_SET message.
The node sends mtVALIDATION after the handshake.
<>
<- mtVALIDATION
The node should respond with mtREPLAY_DELTA_RESPONSE to mtREPLAY_DELTA_REQ.
During this test a feature 'ledgerReplay' is enabled. This requires two actions:
1. Enabling the feature in the config file (option `[ledger_replay]` set to `1`).
2. Adding `ledgerreplay=1` to the `X-Protocol-Ctl` header during the handshake.
<>
-> mtREPLAY_DELTA_REQ
<- mtREPLAY_DELTA_RESPONSE
The node should respond with mtPEER_SHARD_INFO_V2 to mtGET_PEER_SHARD_INFO_V2 when
sharding is enabled.
<>
-> mtGET_PEER_SHARD_INFO_V2
<- mtPEER_SHARD_INFO_V2
The node should connect with other nodes in its cluster and exchange public keys.
<-
<- mtCLUSTER with public keys
The node should respond with mtPROOF_PATH_RESPONSE to mtPROOF_PATH_REQ.
<>
-> mtPROOF_PATH_REQ
<- mtPROOF_PATH_RESPONSE
A synthetic node sends a mtVALIDATORLIST message with both master and signature public keys, correctly serializing a manifest and validator blob to the node. To verify the node has received the message, another synthetic node awaits a mtVALIDATORLISTCOLLECTION message from the node with the same validator blob sent by the first synthetic node in its mtVALIDATORLIST message.
<>
-> mtVALIDATORLIST with master and signing public keys and a correctly serialized manifest and validator blob.
Assert: sequence number in the validator list and public key in the validator match what was sent.
The node behaves as expected under load from other peers.
1. Establish a node and synthetic peers.
2. Begin simulation.
3. Introspect node health and responsiveness through peers (latency, throughput). This could be done using `Ping`/`Pong` messages.
There can be different errors during testing: broken pipes to signal established but suddenly lost connections,
InvalidData errors to indicate that connection couldn't be established at all or timeout errors to indicate
that data has not been received in timely manner.
The node sheds or rejects connections when necessary.
1. Establish a node.
2. Connect and handshake synthetic peers until peer threshold is reached.
3. Expect connections to be dropped.
The node behaves as expected under load when other peers are requesting transaction details.
1. Establish a node and synthetic peers.
2. Begin simulation.
3. Introspect node health and responsiveness through peers (latency, throughput) when requesting transaction data.
Each peer is requesting transaction details using transaction hash obtained earlier through RPC.
The node rejects a handshake when the 'User-Agent' (for initiation side) or 'Server' (from responder side) header
is too long (8192 bytes in this test).
The node should be able to accept connections after such a request.
These tests attempt a handshake with long 'User-Agent'/'Server' headers and ensures that the connection
is rejected. Then, it attempts a normal connection and ensures that the connection is established.
The node rejects various random bytes post-handshake.
The test sends random bytes of variable length (between 1 and 65536) 20 times in a row.
The random values are generated using rand_chacha for performance benefits and ease of use.
<>
-> random bytes
Assert: The node is disconnected after sending random bytes
The node rejects the handshake when there is error an in any of handshake field:
1. Bit flip in a public_key.
2. Bit flip in a shared_value (which later results in an invalid session signature).
3. "Connection" HTTP field is incorrect.
4. "Connect-As" HTTP field is incorrect.
5. "Crawl" HTTP field is incorrect.
6. "X-Protocol-Ctl" HTTP field is incorrect.
7. "Network-Time" HTTP field is incorrect.
8. "Upgrade" HTTP field is incorrect.
9. "User-Agent" HTTP field is incorrect.
10. "Closed-Ledger" HTTP field is incorrect.
11. "Previous-Ledger" HTTP field is incorrect. It was discovered that a valid string results with rejected handshake
with a message "Malformed handshake data (2)".
12. Extra header checks: Checks for rippled bahaviour when sending duplicate headers. It was found that in such case
rippled will not drop the connection.
The node rejects various random bytes pre-handshake.
-> random bytes
Assert: The node is disconnected after sending random bytes