[CI] Windows wheel builds are slower after refactor

[amoeba]

Describe the bug, including details regarding any error messages, version, and platform.

@kszucs reports on Zulip that Windows wheel builds in CI are taking longer now (~2hr) than before (~30min). From the logs it looks the increase in running time is due to the job building the Docker image instead of pulling from cache.

We recently refactored the Windows wheel images to use a new base in #45442 so that probably broke something.

Component(s)

Continuous Integration