Run CI jobs in container

[demisx]

Currently, we execute all jobs directly on self-hosted runners. This leads to a side effect where each runner has state left over from the previously executed job. We try to clean it up in the common "Reset Runner" action right now, but it has its own drawbacks:

1. It's a manual process, so it's easy to miss something
2. Leftover state leads to intermittent errors which are hard to replicate and fix
3. The reset runner cannot be used as the first step, because the repo that hosts this runner needs to be checked out beforehand. And if the error is in the checkout action itself we are looking at catch 22 (ex. Fix GitHub check out error #1133)

In order to make CI process more reliable, I recommend considering running each job in a container, so each runner starts off with a clean state every single time. There is also another benefit of running in a container - we can create and use our own images with all preinstalled tools and libraries. Makes CI even faster.

Tasks:

• Conduct research on running CI jobs in a container, caveats associated with it
• Move shared runner jobs to container
• Move self-hosted runner jobs to container
• Remove debug steps
• Delete test rust file
• Disable Reset Runner common action (globally)
• Remove HOME: /root env var?
• Evaluate the benefit of pre-building our own images with pre-installed tools and libs
• Switch from root to runner user

References:

• Running jobs in a container
• Docker Container Jobs in GitHub Actions
  • Caveats of Running Jobs in CI Container
• Using Docker Containers In Jobs - GitHub Actions
• Docker run non root

[demisx]

@wilwade @saraswatpuneet I think this is the next logical step in Frequency CI evolution. Let me know what your thoughts are on this.

[demisx]

Here is my plan of attack. So far we are running two types of executors:

1. "Self-hosted runners" for compute intensive jobs (8 total)
2. "GitHub shared runners" for everything else

• I am going to attempt to run ALL jobs in a container, regardless of the executor. All jobs will run under non-privileged user runner now (I'm worried that self-hosted runners run them as root right now).
• ubuntu-20.04 official docker image will be the base image.
• Required tools and libs will be pre-installed into the image to, hopefully, reduce CI execution run times further.
• Alpine image is a backup in case we see poor performance from the Ubuntu based image.
• The jobs running on the shared runners are simpler and will be converted to use containers first, in its own PR.
• A separate issue Harden GitHub Actions #1152 has been created to further improve security of GH actions, especially since this is a public repo and community is allowed to fork it.

[demisx]

The calc-code-coverage job fails with an error when running in a container. Filed the following bug actions-rs/tarpaulin#24.