Change the default registry flavour from conservative to eager

[KristofferC]

I really think the current default flavour is wrong. I would argue that 99.99% of users do not care if they download a package / artifact from the Pkg server or GitHub. They much rather have the latest version and there are quite frequent questions about why a user gets an old version installed, even though it was x hours since it got registered. As it is right now, if the package server processes a package slowly, all registered versions after that are blocked for everyone which imo is quite unacceptable. The recent reported blockage was ~18 hours from what I recently read which imo is not really acceptable.

In special cases, where you are e.g. behind a fire wall so you cannot talk to GitHub you want to use the conservative version since otherwise you might fail the download. But this is the special case so imo it makes more sense for these people to have to set some option.

cc @StefanKarpinski, @DilumAluthge .

[StefanKarpinski]

As a technical matter, this is the wrong way to do this. The client should never set a default for server options—the default is to set no value and have the server pick a default. In fact, specific server options should never appear anywhere in the Pkg code. The right way to change this default is to change the server to pick eager when the client doesn't set an explicit preference. That change would immediately affect all clients (except those with an explicit preference set). This is why you don't hard code these things into the client—then you have to wait for a new release and for people to start using it, which would take months. When the server controls the defaults we can change them immediately for everyone. That's also why the JULIA_PKG_SERVER_REGISTRY_PREFERENCE doesn't appear anywhere in the Pkg code; instead we just send any environment variable that starts with JULIA_PKG_SERVER_ to the server as a header. That way we can introduce new server options at any time and allow people with any recent Julia version to start using them immediately.

As a social matter, I vehemently disagree with changing the default to eager. The vast majority of users don't care if the registry they're using is a few hours old, or even a day old. They'll never even realize it—they don't ever look at the upstream registry and they just get package updates when they get them. The only people who really care about getting package updates promptly are developers of registered packages, and they have to be in contact with us because they have to register their packages, which gives them a mechanism to learn about eager mode if this affects them. But defaulting to conservative is necessary for people behind restrictive firewalls not to have a terrible experience. That number of people is conservatively some 10% of users. And unlike registered package developers, who inherently need to be in some contact with us, people behind firewalls tend to be the people we're least likely to ever hear from, who will just give up in frustration because from their perspective the Julia package manager is just constantly broken.

I also feel that we should not, in general, regularly be putting people in a position to need to get packages and artifacts from upstream sources, which may be down or broken, and which we have no insight into the state of (not to mention it undermining the accuracy of our download statistics). We should keep everything flowing through package servers as much as possible, which is exactly what the conservative default does. I think that's a good principle for everyone, not just people behind firewalls.

[StefanKarpinski]

Also, @staticfloat and I have talked about an approach that would basically eliminate the lag between the eager and conservative registries: the idea is to prepare all the source and artifact tarballs before allowing package registration to succeed, as part of a CI step. After all, if we can't get the correct tarball or there's something wrong with it that would cause tarball creation to reject it (e.g. it has a symlink attack in it), then we shouldn't allow registration in the first place. If those tarballs are made before registration succeeds and available to storage servers immediately, then the processing lag between eager and conservative should disappear since all the work is done before the registry is updated.

[DilumAluthge]

the idea is to prepare all the source and artifact tarballs before allowing package registration to succeed, as part of a CI step

Cross-ref: JuliaRegistries/General#76297