T340226 reproducible builds

[rti]

Addressed in this PR

This PR became a bit more complex as it turned out to make lots of sense to do some of the "heavy lifting" tasks in the build system all at once instead of distributing them in several PRs.

Specifically, this PR changes:

• Component versions are specified in variables.env which should makes builds pretty™ reproducible, this is true for git checkouts and docker images https://phabricator.wikimedia.org/T340226
  • definitely not an reproducible on a byte by byte level, e.g. debian security patches will sneak into docker base images
• Flatten file hierarchy, i.e. remove the Docker folder https://phabricator.wikimedia.org/T343873
• Move build logic into Dockerfile to leverage docker builds caching https://phabricator.wikimedia.org/T345564
  • this removes all custom git and downloads caching logic
  • this removes all traces of composer cache https://phabricator.wikimedia.org/T346882
• Put docker builds first, tarballs are extracted from the docker builds now
• Builds are now versioned right after build, docker images get version tags and latest, tarballs get a version string
• Remove the builder container, everything is done in docker build now which is containerized anyway
• ./build.sh now has a slightly different interface, see README https://phabricator.wikimedia.org/T345564
  • exporting to tarballs and dumping docker images as tarballs needs to be explicitly set via options now
  • without those options, builds can run way faster now, especially when parts are cached already
• wdqs-frontend is based on debian now, not alpine for consistency (we use debian in most places anyway)
• Do not write any temp files into the repo during build https://phabricator.wikimedia.org/T347053
• Remove the build metadata mechanism, as all versions are specified upfront now https://phabricator.wikimedia.org/T347084
• Working towards removing the Makefile, it is only used for test and lint now https://phabricator.wikimedia.org/T346037

❗ We probably want to backport this to 1.39 and 1.40

Open Questions

• Some base images change, when we backport this to 1.39 and 1.40 we might introduce instabilities. How can we prevent this?
• How does the cache handle commit sha and docker image url changes?

[lorenjohnson]

So excited for this one. My favourite set of changes to date! I approve and based on use so far think its doing everything it should. 🎉