End-to-end integration tests for Zeekoe

[plaidfinch]

The goal is to develop a script for end-to-end testing zkChannels lifecycle (ie establish, pay, and close) on

• sandbox
• testnet
• mainnet

Update 1/10/22: The current version of the test script executes many code paths, but it does not effectively compare them to the expected outcomes. In particular, it does not distinguish "happy paths" from test cases that should fail, and it does not react to non-abortive failures in the chain watchers (#338).

I propose an updating testing framework in Rust that uses the standard test conventions (e.g. lives in test/) and solves these issues. Like the existing script, it would

• have to run in an environment with an accessible on-chain network
• spin up and monitor a merchant server, customer watcher as sub-processes (or, depending on Write spec for RPC Protocol #322, using an RPC request).

Each test is described as a struct with a list of pairs of an operation (e.g. establish, pay, pay all, expiry) and an expected outcome (customer status, merchant status, whether an error should be raised).

The tests will run the setup phase once, then run each test. Every test needs to monitor the output from the three processes (server, watcher, and operation) and identify errors relevant to itself; this may become easier with #91. It will also check the expected outcome against the current status (e.g. by calling list for each party).

The tests cannot be run in parallel with the current Tezos instantiation per #282, #212. We can prevent this with a testing flag. If we do eventually parallelize testing, we will need to define a way for the test to identify errors that belong to itself, as the server and watcher may be generating errors for multiple tests in parallel.

Ideally, this will confer the following benefits:

• greater nuance in determining whether the correct error was thrown (e.g. by the correct party? at the correct step in the set of operations?)
• more easily extended to a chain-agnostic environment (and to testing on-chain error cases), since we could build a mocked un-networked testing chain fully in Rust after we define a chain trait Create chain-agnostic ArbiterClient trait #346
• easier to scan test coverage: they can be named and explicitly defined using a consistent format
• easier to build out malicious party scenarios Test malicious-customer pay scenarios #327: we could switch out the honest party sub-process for a mocked malicious party and define the specific expected error cases

[indomitableSwan]

@jakinyele what is the current status of this? Are we moving this to milestone 4?

[jakinyele]

Correct, we won't have any automated scripts to support end-to-end tests until milestone 4. Will create a new issue for testing goal specifically for milestone 3.