End-to-end XCM Test

[mustermeiszer]

This idea is lingering around in my mind for a really long right now, and Nuno and I thought, that the newly opened discussions section is a great starter to get us talking about this.

The Why

Currently, when we test our connectors logic we are "just" ensuring that the connector messages decode and encode correctly and that we actually submit a correct XCM message (I hope the last part is correct @NunoAlexandre).

The Problem

• The problem here is that we have no coverage about the other side of XCM. Although this is currently not an issue, the more complex our integration will get -- imagine connectors connecting to multiple chains, Moonbeam, Ethereum, Acala -- our knowledge of when and if we break stuff is really.

• The development speed is really slow. The time it takes to actually test a new setup -- Centrifuge chain using new contract addresses, and Moonbeam contracts actually updated -- takes roughly half a day. This will slow us down in the long run with increased complexity

The Solution

I want to integrate the actual Moonbeam runtime in our integration test and use FUDGE to submit real XCM messages from our chain via the relay chain to Moonbeam (locally, of course, no network).

The Pros

• Faster development cycle
• Logs at the console
• End-to-end

The Cons

• Harder to maintain
  We need to keep a fork of moonbeam maintained
• CI pipeline gets heavier
• Does it actually increase our coverage?
  This depends on how much we depend on a live-state and how good we can mirror this one in the tests

[lemunozm]

Does it actually increase our coverage?

This is a really good question. With this proposed end-to-end testing which exact part of our code are we testing that we are not already testing in our tests?
If we want to test the whole path in centrifuge-chain, could we do this integration test "mocking" the Moonbeam behavior instead of using it?

[mustermeiszer]

I think we could create a mock runtime that contains the same xcm configuration and the same evm pallet as moonbeam does.

BUT we also need to have moonbeam as a dependency in this case, so it is additional overhead without any gain.

If the evm pallet would be separate I would consider it an option.

[NunoAlexandre]

The main goal of this discussion is to implement actual end-to-end xcm tests with the real runtimes that will be interacting in production so mocking a runtime is a disservice to that goal. Let's rather explore the options @mikiquantum left below 🙏

[mikiquantum]

I am onboard on some sort of real integration test for connectors or any xcm-based business flow.
I have a few concerns though:

• Upgrades of Polkadot dependencies in our chain code now depends (even more) on other projects updating at the same cadence as us. We cannot expect this from any project, and we cannot block our releases because of that. Might be a good idea to take this into its own xcm-integrations-test repository.
• Assuming that at certain point in time we have successful runs, the third party project will move at a different pace than us (lets assume faster) and our test would become obsolete (since we didnt update their deps at the same speed) and therefore we would be testing again only from our point of view of the world, as before.
• In the case of connectors, it will not be just enough to send a successful xcm message but we would need to configure the evm pallet, deploy our connectors contract and then verify that the e2e bidirectionally works. This requires specific fixture data and test setup beforehand. Moreover requires deeper knowledge on the target project setup and most importantly keeping up with it.

The last two points, makes me think that having a live environment maintained by each project and have e2e automated tests that run regularly might be more practical than having to maintain all the above over time.

[NunoAlexandre]

The last two points, makes me think that having a live environment maintained by each project and have e2e automated tests that run regularly might be more practical than having to maintain all the above over time.

Great point. That live environment already exists, the Moonbase Alpha Testnet. So we could in fact consider just writing JS-based automated tests that we would run against our chain and Moonbase on that testnet to trigger calls and verify their impact.

[mustermeiszer]

The comment from Miguel makes me wonder. From my perspective I think we need two things:

1. live environment to run e2e tests on
2. xcm playground to increase development speed

The e2e would only run on releases maybe or when specifically triggered, but assume everything works. The playground is not mandatory to be maintained at the same speed as the chain repo but is useful for actually increasing the development speed by having everything local with fudge.

WDYT?

[NunoAlexandre]

I have been diving into Acala's chopsticks and their e2e-tests repo that is meant to be used by the community as a hub for testing XCM transfers across the different chains.

A problem with using Fudge for this purpose is that it's a Rust SDK and we wouldn't be able to run tests across different versions of the same chain or across chains if they are at different Polkadot versions.

One USP of chopsticks is that 1) it allows for multiple versions of Polkadot (i.e, we can test Centrifuge at Polkadot 0.9.39 against, say, Moonbeam at Polkadot 0.9.42) since it forks the desired chains and runs them locally, against which we can run tests, and 2) it allows for wasm override so we could for example verify that we can fork our latest Centrifuge chain's state and run the same set of tests but using the wasm of an upcoming release to ensure that everything would work just the same.

I am first looking into using e2e-tests to move all our XCM transfer tests into and clean up a lot of the xcm testing boilerplate from our chain and actually have those tests use the real chains and not mockers for the "other" chains. Running into some issues here which I am trying to figure out together from Bryan from Acala.

Afterwards I would like to play more barebonds with chopsticks directly on a new dedicated Centrifuge repo where we could have more Centrifuge specific tests, the main purpose at first being Connectors; so we could use chopsticks to fork Centrifuge and Moonbeam for example, deploy Connectors into the Moonbeam evm, and run our full e2e connectors tests for all the flows and edge cases.