Integration and E2E Tests

[dmathieu]

We currently have a couple of tests that are intended to be e2e.

I feel like it would be nice to formalize a bit more what we mean by integration and E2E tests.
Here is my definition proposal:

Note: I know I am phrasing things in a very assertive way here. This is not meant to force my own thinking on everybody.
I will update this original message based on comments in the discussion, so that this original message can reflect any consensus we (hopefully) get to in the end.

Test types definitions

E2E Tests

E2E Tests are, as their name states, end to end. So they should run the full binary produced by this service, and never use any mocks or fake factories.
They should therefore be able to access external AWS, GCP or a functional K8 cluster (for kubernetes, that may happen locally with tools such as kind though).
This means in order to run them, credentials are very likely to be required, and possibly some setup.

As an example, you can see some of the official kubernetes azure cloud provider e2e test: https://github.com/kubernetes-sigs/cloud-provider-azure/blob/master/tests/e2e/node/vmss.go

These tests are quite expensive, and can't test every use case (we can't force the cloud provider to generate errors). So they are only meant to ensure the API contract hasn't changed.
They don't need to run on every push to a branch, but they probably should run at least once before going to main.
They could run only once a PR has been approved, or only as part of the merge queue for example.

Integration tests

Integration tests are meant to ensure a module synchronizes properly with every other module it uses.
It shouldn't use mocks, but it also shouldn't call external services (connectivity shouldn't be required, nor should any credentials).
Calls to external services could be abstracted away with in-memory factories (though in that case, we're not testing the actual HTTP call anymore), or with tools such as go-vcr.

These tests are less expensive than E2E ones, though they shouldn't test every edge case either (unit tests are there for that). They should ensure the code can run properly, provided the external services' API contract doesn't change.

Our approach in inputrunner

At the time of the original writing of this discussion, our existing e2e tests should probably actually be integration ones, as they ensure the inputs can run, but they remove any calls to external APIs.

Setting up integration tests is quite easier than e2e ones, which require much more boilerplate. So we should start with those, and setup/use an e2e framework later on.