Threaded FS IO probably should be async now?

[rami3l]

Being able to use async where relevant across the codebase makes it possible to use async constructs where relevant. rustup has many places where that can be useful: downloading channel metadata, distribution content, unpacking that content to disk, are work that could usefully proceed in parallel - and async provides a good abstraction for that.

We already have a sophisticated disk IO layer that accomodates various OS latency-inducing behaviours, and adapting that to async without any regressions could be very interesting too - but for now, it co-exists nicely with an async core.
#3367

We are still using an async-unaware threadpool here, and it's known to produce problems (#3125):

rustup/src/diskio/threaded.rs

Lines 1 to 6 in fa4ae32

	/// Threaded IO model: A pool of threads is used so that syscall latencies
	/// due to (nonexhaustive list) Network file systems, virus scanners, and
	/// operating system design, do not cause rustup to be significantly slower
	/// than desired. In particular the docs workload with 20K files requires
	/// very low latency per file, which even a few ms per syscall per file
	/// will cause minutes of wall clock time.

Anyway, using RUSTUP_IO_THREADS=1 to limit concurrency feels a bit off. Will migrating to async help?

[djc]

IMO the code we have for this looks very complicated/scary. Rewriting it looks like a project, would probably be hard to know if we bring over all the desired properties. async could be helpful but won't make this easy by itself IMO.

[rami3l]

Ah, I do remember the days when installing Rust took a long time to finish: #1876

(cc the original author, if I'm not mistaken, @rbtcollins)