Package Distributions

[nlf]

Epic

A common problem in the npm ecosystem is the need to distribute pre-compiled binary assets. To do this today involves either a postinstall script in the package that fetches the asset that corresponds to the current system, or publishing each binary asset as its own package in npm and using optional dependencies on all of them, or potentially a combination of those approaches.

None of these alternatives is ideal. A postinstall script means that users cannot use the --ignore-scripts flag or the script won't run. The use of a series of optional dependencies requires the consumer to attempt to require every one of them in sequence until they find one that's installed (or don't!). It also means that the package-lock.json will only include the optional dependency that was actually installed as part of this particular installation.

Exit criteria

1. A package should have a means of defining distributions, a means of mapping system parameters to specific sub-packages.
2. Distribution sets should be fully included in the package-lock.json in order to eliminate churn and correctly lock all distributions available rather than only the one that successfully installed on the current host.
3. Only one matching distribution should actually get installed by default.
4. Packages at any depth in the tree should be able to request specific distributions of their dependencies.

Tasks

• review existing RFC
• answer open questions from RFC
• review prior art of yarn's package variants proposal - [Feature] RFC: Package Variants yarnpkg/berry#2751
• investigate potential backwards compatibility
• determine how the consuming package knows which (if any) distribution is available (require them all?)
• contact some projects who could benefit from distributions and discuss their needs
• ask ourselves if we're solving the right problem, do we want to make the current workarounds easier or do we want to create a brand new flow?
• rewrite existing RFC to align with data gathered here

Immediately actionable items

• [RRFC] Add libc fields to select optionalDependencies should be installed or skipped rfcs#438

Feature ideas

• ability to install additional non-matching distributions to allow builds to be performed on a different type of host
• more fields to match against (see libc rfc above) possibly allowing all of the fields in process.versions, known requests for os, os-version, cpu, libc, libc-version, runtime (node/electron/deno/browser)
• fall through that allows a source package to be installed and built when no match is available
• more types of distributions, i.e. esm vs commonjs, slim (omits docs), locales, possibly arbitrary tag selectors?
• ability to specify distributions as arbitrary files that are fetched and placed within the package's own directory
• overrides should work for specifying a variant of a third party dependency at any depth
• allow for providing integrity checksums (if distributions are each their own npm packages we more or less get this for free)

Supporting documentation

• RFC: Package Distributions rfcs#519

[nlf]

I've reached a point where I can't decide if this is actually a new feature or if it's really just some enhancements to optional dependencies

If we frame it as the latter, we would need to always include all optional deps and their own dependencies in the ideal tree instead of only those that are installable on the current system. We would also have to allow installing deps that do not match the current system to allow for builds and production environments to run on different platforms, sort of like cross compiling. Potentially we would want to add something to allow specifying "at least one of the following optional dependencies must be installed" as well.

This scoping is a lot more narrow, but maybe it solves the general use cases well enough that that's a good thing.

[ljharb]

@nlf i don't think optional dep features would address the "full vs minimal" file set tho :-/

[nlf]

@nlf i don't think optional dep features would address the "full vs minimal" file set tho :-/

this is true, we would need a different solution for that. there's a difference between "platform specific dependencies" and "variations of the same package". i think the original proposal here was trying to address both of those needs and that it may be better to split them into two concerns. optional deps address the first, but that second concern would have to be addressed in a different way.

[ljharb]

One of the motivating use cases tho was indeed variations of the same package - it's just that for that, they're platform-specific. The way they hack around it right now is via platform-specific optional deps, but that doesn't seem like a clean approach to endorse.