-
Notifications
You must be signed in to change notification settings - Fork 11
What would decentralized publishing on IPFS for npm actually look like? #58
Comments
I was going to open an issue on ipfs-desktop about the possibility of bundling a fork of npm with ipfs support added via pacote that would enable some of these ideas via the flick of a switch in IPFS desktop. But thinking about it, npm-on-ipfs and the pacote patch come at the problem from two different angles and I suspect the current implementation of npm-on-ipfs would not directly benefit from the pacote patch being merged (or available in a fork). This is because npm-on-ipfs acts as a transparent IPFS proxy for registry.npmjs.org, allowing individual users to opt-in to loading data from registry.npmjs.org over IPFS. It doesn't force other contributors to the same project to also use npm-on-ipfs. The pacote patch on the other hand does force any other contributors to use IPFS as well, take the following package.json for example: {
"name": "my-module",
"dependencies": {
"big-number": "ipns://Qmfoo^1.1.0",
}
}
I'd like to propose a slightly different approach inspired by ipfs-companion. ipfs-companion-lib (or http-or-ipfs)ipfs-companion the browser extension does a neat thing where if it see's the browser loading a http url where the query path starts with It also only upgrades requests if it detects that the user has an active IPFS node, either via an embedded one, IPFS desktop or a go-ipfs daemon. If there was a low level javascript http library (perhaps called This would then enable users with IPFS support to enjoy loading modules from their neighbors without directly adding IPFS hashes to their package.json files. I suspect DNSLink or It doesn't look like registry.js.ipfs.io currently publishes a DNSlink record but that'd be a good thing to add. Then you could set registry.js.ipfs.io as your config in .npmrc and any http requests to that registry would be upgraded to IPFS for users running a node locally whilst continuing to work over http for other users. Eventually the hope would be that registry.npmjs.org would also have a DNSLink and registry.js.ipfs.io could be retired. It'd also mean that for self-published packages, rather than using the pacote patch style {
"name": "my-module",
"dependencies": {
"big-number": "https://ipfs.io/ipns/Qmfoo^1.1.0",
}
} One other thing to note here, currently the npm-on-ipfs infrastructure uses js-ipfs and does not connect to the same DHT as go-ipfs, so trying to load tarballs and packuments through the ipfs gateway (http://ipfs.io/ipfs/bafybeihmelfwcg664jeznipvrx2qzc6acrme5ztea2r4rljdqzrj72bx6u) doesn't find anything. |
There are quite a few things to unpack here..
This should be trivial to implement - I can't find a detailed spec for packuments but there's an overview on the packote repo.
True, but you can have more memorable strings via DNSLink and it's better than having them in the source code. If we get Not sure how it would look though, maybe:
Also more unsure of how we'd deal with scoped packages (e.g.
This is definitely a concern but one I hope we can solve outside of package managers. For example, if the
Where
I think this is ok. We already pull deps from
Not as it stands - it's more aimed at the Level 3 use case above. My gut feeling is that Level 4 adds a significant amount of complexity - it'd probably be good for teams working on large apps but may be overkill for small projects/libs. The IPFS Companion approach is interesting - the You've also made enough of an HTTP request to get the
Or any DHT for that matter. It's coming in 0.36.0 though - if it doesn't we can convert
How do you see this working? We could monkey-patch node internals but you'd have to require it somehow. Also it seems a bit weird to co-opt the Ermergerd there's a lot here, sorry for the stream of consciousness. The way I see
|
I'm been doing some real world testing with this today, although instead of using @achingbrain's pacote patch, I just used the local http gateway provided by the ipfs daemon. Level 1This works as expected:
Install it:
It results in the local gateway url to the tarball being added to package.json as the version: {
"name": "testing",
"dependencies": {
"base62": "http://localhost:8080/ipfs/QmeuiRzbjhnyghoADPCrmPpQYgLj4eG94TWVcedXp6S2mg"
}
}
and a similar package-lock.json file: {
"name": "testing",
"requires": true,
"lockfileVersion": 1,
"dependencies": {
"base62": {
"version": "http://localhost:8080/ipfs/QmeuiRzbjhnyghoADPCrmPpQYgLj4eG94TWVcedXp6S2mg",
"integrity": "sha512-YtkASiZCn90Bd78+tLHSSxtrUekRLUchUhtPCp46vONCJEHJrXOJc97x6Ipdfos0xKIw/9fVBo1adBV5MAkhEA=="
}
}
} Level 2Similar to level 1 but with an ipns name. Generate a tarball and add it to IPFS:
Then publish it under an ipns:
Install it:
I couldn't get that last stage to complete, it timed out every attempt, ipns is still too slow to recommend anyone use it. That said, I expect to get the same results as with Level 1 just with a slightly different url. This also has the existing downside of not allowing users to control which version they are installing and will cause integrity checks with lockfiles, so not a good idea to use for demos either. Level 3Turns out this does not work at all, although not because of IPFS but a misunderstanding in how npm installation arguments work. The theory was that you could publish a "packument" to IPFS and then pass the address to npm along with a version and it would just work. Testing with a regular packument from npmjs.org over http reveals the problem:
If you pass anything that looks like a url (or any other address that pacote already supports like git://) then npm is expecting to find a single folder of a release with a package.json in the root. In other words, you can't pass a packument to Level 3.1 (previously level 4)So if level 3 doesn't work, what does? As far as I can tell the only way to pass a packument to
One catch at the moment is that the
fun fact: npm uses a different module for making registry requests than pacote: npm-registry-fetch, so @achingbrain's pacote patch won't automatically add ipfs support here too But we can use the local http gateway provided by ipfs 🎉 The command If we take the packument from https://registry.js.ipfs.io/bignumber and drop it in an empty folder called
Then add that folder to ipfs:
We can then use the CID folder as the root registry over the local gateway:
Which results in the following package.json: {
"name": "testing",
"dependencies": {
"bignumber": "^1.1.0"
}
} and package-lock.json: {
"name": "testing",
"requires": true,
"lockfileVersion": 1,
"dependencies": {
"bignumber": {
"version": "1.1.0",
"resolved": "https://registry.js.ipfs.io/bignumber/-/bignumber-1.1.0.tgz",
"integrity": "sha1-5qsKdD2l8+oBjlwXWX0SH3howVk="
}
}
} But there's an issue there, the resolved url in package.json is a http url to registry.js.ipfs.io, which won't use ipfs to load at all. Let's update the packument to point to the actual tarball hash on ipfs via the local gateway and readd that to ipfs: {
"_id":"bignumber",
"name":"bignumber",
// ...
"versions":{
"1.1.0":{
"name":"bignumber",
"version":"1.1.0",
// ...
"dist":{
"shasum":"e6ab0a743da5f3ea018e5c17597d121f7868c159",
"tarball":"http://localhost:8080/ipfs/QmeERhULUu7nwgLvxFrnnvJViKqwUxpmswWKzg41ensSYk"
}
}
},
// ...
} The updated package-lock.json: {
"name": "testing",
"requires": true,
"lockfileVersion": 1,
"dependencies": {
"bignumber": {
"version": "1.1.0",
"resolved": "http://localhost:8080/ipfs/QmeERhULUu7nwgLvxFrnnvJViKqwUxpmswWKzg41ensSYk",
"integrity": "sha1-5qsKdD2l8+oBjlwXWX0SH3howVk="
}
}
} You can now turn off your wifi and successfully run the installation command again and it "just works™️" 🎂 This would be a good place to use ipns, publishing a name for the folder means you can publish new versions (add another version to the packument) or add new packuments to the folder, re-add the whole folder and update the ipns. This is very similar to how npm-on-ipfs actually works, except uses the local gateway rather than running it's own web server proxy. In theory if you connect to the npm-on-ipfs swarm and fetch the
There's a couple ugly bits when it comes to this being more widely used:
There also needs to be a tool to generate a packument as writing one by hand is very error prone! extra ipfs-npm commandsSome potential commands that could be built into the
And a couple extra commands that I thought of whilst experimenting:
|
Some thoughts:
This will conflict with
Having this sort of thing end up in your lock file doesn't bode well for repeatable builds as things will break if ports change or the daemon isn't running on the current machine, etc, which is why Maybe if the pacote PR gets merged we can rewrite them to
I like the simplicity of using the local gateway as the registry but it means we can't rewrite the lock file post-install. Then again if the pacote PR gets merged, and the registry uses |
I thought I'd take a stab at making a script that implements a certain amount of Level 3.1 (previously Level 4), here's what I came up with: https://github.com/andrew/ipfs-npm-republish First npm install it: Then change into a directory with a package-lock.json and run: This will do the following steps:
The code probably isn't the best and it doesn't handle unexpected errors very well but when it works you do end up with a valid npm registry hosted on IPFS 🎉 Things it doesn't do:
It's also small steps towards the ideas in #52 around "portable packages", where you could use ipfs-npm-republish to create a micro-registry for a package and one for an application, then have a function that can merge those two micro-registries together, enabling someone to add the package to the application and generate a new package-lock.json without relying on a traditional registry or proxy at all. |
Next step I'm going to look at doing is republishing for a package, which is subtly different than what ipfs-npm-republish currently does, which is to republish only the dependencies of an application. In this case you'll be able to do:
Then you'll have everything you need to do a command line installation of react from that one micro-registry |
As of v1.0.3 you can now republish individual modules directly from npmjs.org:
|
As of v1.0.6 of ipfs-npm-republish you can now publish directly to IPFS without needing to first publish to npmjs.org. Running Aside from some performance issues on very large dependency trees, the last major area to cover is "update", which would in-effect take an existing IPFS micro-registry, check upstream for dependency updates and then publish a new micro-registry with the updated dependencies. N.b. "update" in this sense is a little different to using IPNS for a micro-registry as it'll need to support packages from sources outside of IPFS, micro-registries published with IPNS should be auto-updating without any extra work. This will require storing some more extra metadata in the packuments, which I'm planning on modelling on some of the ideas from #22, being able to follow the providence of the packages to get back to their original publish location, either on a traditional registry or hopefully the IPNS of the micro-registry they were published to. |
v1.0.8 of ipfs-npm-republish is out, includes some significant refactoring:
I spoke to @olizilla yesterday about having IPFS desktop install npm-on-ipfs via a microregistry. This would be pretty neat, IPFS Desktop wouldn't need to include ipfs-npm-republish as a runtime dependency, it can just store a CID (or eventually an IPNS name) to the npm-on-ipfs micro-registry created by ipfs-npm-republish. There are a couple of bugs before that can work seemlessly:
Any help on those two bugs would be great! |
There was some talk of decentralized publishing for npm-on-ipfs on the IPFS GUI call yesterday as a potentially interesting area to explore.
After exploring some decentralized package manager ideas I thought I'd try to apply some of them to an npm implementation.
Assuming ipfs client support can be added to npm or yarn and that IPNS isn't slow (or DNSLink is used), how does one go about publishing a package directly to IPFS and then having someone else consume it?
One key aim here is to not require hosted infrastructure, can we just use standard IPFS tooling and keep the logic in the clients?
Also of note, none of these solve the discovery problem of "where do I find the decentrally published packages", which likely would be an opt-in search engine where publishing tools announce the cid of new published packages to be indexed.
Level 1
The simplest option would be to add the a tarball directly to ipfs:
ipfs add big-number-1.1.0.tgz
Then any user could specify it as a dependency in package.json:
Downsides to this:
Level 2
To add an update mechanism to level one, you could use ipns:
ipfs name publish $(ipfs add big-number-1.1.0.tgz)
Then any user could specify it as a dependency in package.json:
Then when publishing a new version, just update ipns to point to the new one:
ipfs name publish $(ipfs add big-number-1.1.1.tgz)
Downsides to this:
Level 3
To allow end users to control which version they are installing, publish a modified index file (eg bignumber) along side the tarball.
Rather than adding a new field
cid
, like npm-on-ipfs does, instead make thetarball
field the cid of that versions tarball:As in level 2, using an IPNS name for the index will allow for an update mechanism:
ipfs name publish $(ipfs add bignumber)
Then any user could specify it as a dependency in package.json:
Missing parts:
Downsides:
Level 4
Publishing a small registry directly to IPFS.
npm-on-ipfs has already shown how you can setup an npm registry on ipfs, by publishing multiple versions indexes (packuments) and adding them to a folder (or mfs), an IPFS address can then be used as a registry rather than a the location for one particular package.
small registries could be collections of useful modules for individual personal use, but the most likely use case is per-project registries that contain all the dependencies for that application.
Command to point your local npm client at an ipfs registry
npm config set registry ipns://QmT8wUy9CV3FHbjQeDG49ZL7at24u5fynBrkjum4MznNA7
The setting can also be added to a
.npmrc
file which can be commited in a repository so that all users of that project use the same registry:Then packages can be referenced in your package json using regular names:
This has the added benefit that all sub-dependencies will also be pulled from IPFS automatically, but will require the publisher of that registry to ensure they have added all required versions of every sub-dependency to that IPFS folder before using it.
Missing parts:
Downsides:
Note:
The text was updated successfully, but these errors were encountered: