Reworking E2E tests for Machine Controller to support external CCM

[ahmedwaleedmalik]

Kubernetes maintainers have long been discouraging the usage of In-tree cloud controller manager(CCM) in favor of external CCM. In-tree providers are planned to be removed from the Kubernetes core and we are already affected with:

• OpenStack on k8s v1.26+
• AWS on k8s v1.27+

For now, we are ignoring or skipping e2e tests for the affected versions but this is not ideal/acceptable, it's important to ensure that the e2e tests are reliable and up-to-date.

Proposed solution

Switch from KIND to KKP for machine-controller E2E tests.

1. Create a dedicated KKP setup(or dedicated seed) for machine-controller. Connectivity for nodes would work out of the box and we won't need any hacks for CCM or CNI.
2. Creating user clusters for e2e tests. We have multiple options for that; we should evaluate and choose:
   1. Long-standing user clusters per cloud provider with no machine deployments. MC will create the required machine deployments for tests and then clean them up.
   2. When running tests, create a user cluster per cloud provider. We can check if a user cluster for the provider already exists and then just use it.
   3. Dedicated user cluster per CI job; this is not ideal due to resource constraints and the cost incurred.
3. Run our test suite to check that the machine came up and was ready. We don't need conformance tests etc. here.

This would help solve #1462 as well.

NOTE: Proper cleanup jobs should be in place right from the start to avoid broken/cancelled jobs leading to orphaned resources

Alternate solution

• Stick to KIND and deploy external CCM for each cloud provider when it's required
• Create single node clusters using KubeOne to run the tests

[embik]

Will also fix #1620.

[kubermatic-bot]

Issues go stale after 90d of inactivity.
After a furter 30 days, they will turn rotten.
Mark the issue as fresh with /remove-lifecycle stale.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[ahmedwaleedmalik]

/remove-lifecycle stale

[ahmedwaleedmalik]

With #1709, we are dropping support for Kubernetes 1.25 and that makes the e2e tests for OpenStack no-op. Since they require external CCM.

[ahmedwaleedmalik]

In-tree providers have been disabled by default via kubernetes/kubernetes#117503.

[kubermatic-bot]

Issues go stale after 90d of inactivity.
After a furter 30 days, they will turn rotten.
Mark the issue as fresh with /remove-lifecycle stale.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[embik]

/remove-lifecycle stale