Scylla CDC Connector Tests

[Bouncheck]

Currently CDC connector is severely lacking in tests department. This results in unnecessarily long processes of manual testing, even when introducing the smallest of changes. And even after such testing it is hard to tell if it was extensive enough.

What kind of tests would be useful:

Unit tests – Currently there are 0 unit tests. First step should be covering at least the critical parts. Would greatly help with quickly checking nothing important is breaking with new changes.

Integration tests – Here debezium-connector-cassandra may be a good reference. We could similarly set up Scylla using Testcontainers and java-driver. After that thoroughly check if the CDC tables are translated into Kafka messages correctly, without yet sending them.

E2E – The most expensive type, but would save a lot of pain from setting up every component manually. It would be good to at least have 1 setup with recent Scylla version and one type of Kafka cluster.

Other nice-to-haves:

Stress tests – Mainly to see if anything breaks only under load. We could also think later about tracking performance.

“Nemesis” type of tests – Scenarios where we intentionally throw a wrench in-between important operations. For example: does the connector correctly resume if we crash it after processing pre-image event but before processing the insert it is related to? Will the offset be correct and pre-image reread upon restart?

Other debezium connectors that may be used as a references:
https://github.com/debezium/debezium-connector-spanner
https://github.com/debezium/debezium-connector-jdbc (This is a sink connector)
https://github.com/debezium/debezium-connector-cassandra
https://github.com/debezium/debezium-examples (Not a connector repo, but has some end-to-end examples)

[roydahan]

For the E2E and stress tests we can/should use SCT infrastructure and intergrate it with our current tests.
We can either open a new task here or in "Qa-tasks" repo and assign it to @fruch, so we can plan it for next sprints.