Merge pull request #1970 from ipfs/ipfs-web

ipfs in web apps guide

.github/styles/pln-ignore.txt

@@ -133,6 +133,9 @@ mainnet
 markdown(lint)
 markdownlint
 merkle
+merklizing
+merkleizing
+merkleization
 metadata('s)
 metamask
 minimalistic
@@ -145,6 +148,8 @@ multiaddrs
 multibase
 multicast
 multicodec
+multicodecs
+multicodec(s)
 multiformats
 multihash
 multihashes
@@ -222,9 +227,11 @@ testground
 testnet
 toolkits
 trustlessly
-uncensorable
+trustlessness
+uncensorable 
 undialable
 uniswap
+unixfs
 unreachability
 untrusted
 upgradeability

.github/workflows/markdownlint-action.yml

@@ -7,8 +7,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: markdownlint-cli
-        uses: nosborn/github-action-markdown-cli@v3.3.0
+        uses: nosborn/github-action-markdown-cli@v3.4.0
         with:
           files: .
           config_file: .markdownlint.jsonc
-

.markdownlint.jsonc

@@ -175,7 +175,7 @@
 
   "MD033": {
     // Allowed elements
-    "allowed_elements": []
+    "allowed_elements": [""]
   },
 
   // MD034/no-bare-urls - Bare URL used
@@ -302,4 +302,4 @@
       "//"
     ]
   }
-}
+}

docs/.vuepress/config.js

@@ -256,12 +256,12 @@ module.exports = {
               ]
             },
             {
-              title: 'IPFS in the browser',
+              title: 'IPFS on the web',
               sidebarDepth: 1,
               collapsable: true,
               children: [
+                '/how-to/ipfs-in-web-apps',
                 '/how-to/address-ipfs-on-web',
-                '/how-to/browser-tools-frameworks'
               ]
             },
             {
@@ -278,7 +278,7 @@ module.exports = {
               collapsable: true,
               children: [
                 '/how-to/gateway-best-practices',
-                '/how-to/gateway-troubleshooting'
+                '/how-to/gateway-troubleshooting',
               ]
             },
             {

docs/.vuepress/redirects

@@ -40,6 +40,7 @@
 /guides/guides/addressing/ /how-to/address-ipfs-on-web
 /guides/guides/install/ /install
 /how-to/host-single-page-site /how-to/websites-on-ipfs/single-page-website
+/how-to/browser-tools-frameworks /how-to/ipfs-on-the-web
 /how-to/troubleshoot-file-transfers /how-to/troubleshooting
 /how-to/run-ipfs-inside-docker /install/run-ipfs-inside-docker
 /install/command-line-quick-start/ /how-to/command-line-quick-start

docs/concepts/lifecycle.md

@@ -5,25 +5,30 @@ description: Learn about the lifecycle of data in IPFS.
 
 # The lifecycle of data in IPFS
 
-- [1. Content-addressable representation](#1-content-addressable-representation)
-- [2. Pinning](#2-pinning)
-- [3. Retrieval](#3-retrieval)
-- [4. Deleting](#4-deleting)
+- [1. Content-addressing](#1-content-addressing)
+- [2. Providing](#2-providing)
+- [3. Retrieving](#3-retrieving)
 - [Learn more](#learn-more)
 
-## 1. Content-addressable representation
+## 1. Content-addressing / Merkleizing
 
-The file is transformed into a content-addressable representation using a CID. The basic idea is that this representation makes files and directories **content-addressable** via CIDs by chunking files into smaller blocks, calculating their hashes, and constructing a [Merkle DAG](./merkle-dag.md).
+The first stage in the lifecycle of data in IPFS is to address it by CID. This is a local operation that takes arbitrary data and encodes it so it can be addressed by a CID.
 
-## 2. Pinning
+The exact process depends on the type of data. For files and directories, this is done by constructing a [UnixFS](./file-systems.md#unix-file-system-unixfs) [Merkle DAG](./merkle-dag.md). For other data types, such as dag-cbor, this is done by encoding the data with [dag-cbor](https://ipld.io/docs/codecs/known/dag-cbor/) which is hashed to produce a CID.
+
+For example, merkleizing a static web application into a UnixFS DAG looks like this, where the whole application is addressed by the CID in the top block (`bafy...jomu`):
+
+![UnixFS Dag](./images/unixfs-dag-diagram.png)
+
+## 2. Providing
 
 In this stage, the blocks of the CID are saved on an IPFS node (or pinning service) and made retrievable to the network. Simply saving the CID on the node does not mean the CID is retrievable, so pinning must be used. Pinning allows the node to advertise that it has the CID, and provide it to the network.
 
 - **Advertising:** In this step, a CID is made discoverable to the IPFS network by advertising a record linking the CID and the server's IP address to the [DHT](./dht.md). Advertising is a continuous process that repeats typically every 12 hours. The term **publishing** is also commonly used to refer to this step.
 
 - **Providing:** The content-addressable representation of the CID is persisted on one of web3.storage's IPFS nodes (servers running an IPFS node) and made publicly available to the IPFS network.
 
-## 3. Retrieval
+## 3. Retrieving
 
 In this stage, an IPFS node fetches the blocks of the CID and constructs the Merkle DAG. This usually involves several steps:
 
@@ -35,13 +40,13 @@ In this stage, an IPFS node fetches the blocks of the CID and constructs the Mer
 
 - **Local access:** Once all blocks are present, the Merkle DAG can be constructed, making the file or directory underlying the CID successfully replicated and accessible.
 
-## 4. Deleting
+<!-- ## 4. Deleting
 
 At this point, the blocks associated with a CID are deleted from a node. **Deletion is always a local operation**. If a CID has been replicated to other nodes, it will continue to be available on the IPFS network.
 
 :::callout
 Once the CID is replicated by another node, it is typically advertised to DHT by default, even if it isn't explicitly pinned.
-:::
+::: -->
 
 ## Learn more
 

docs/how-to/ipfs-in-web-apps.md

@@ -0,0 +1,127 @@
+---
+title: IPFS in web applications
+description: How to develop applications that use IPFS on the Web including addressing by CID, merkleizing, retrieval, providing, and CAR files with Helia.
+---
+
+# IPFS in web applications and resource-constrained environments
+
+In this guide you will learn how to use IPFS using JavaScript/TypeScript in web applications, including addressing data with CIDs, retrieval by CID, working with CAR files, and the nuances of providing.
+
+For this, you will use [Helia](https://github.com/ipfs/helia), the most actively maintained implementation of IPFS in TypeScript for use on the web.
+
+> **Note:** this guide is focused solely on using IPFS for data within a web application. It does _not_ cover using IPFS for static website distribution with IPFS Gateways.
+
+## Challenges with IPFS on the web
+
+IPFS allows you to fetch data by CID from multiple providers without being reliant on a single authoritative server.
+
+However, making all of this work on the web is tricky due to networking constraints. Browsers impose many restrictions on web apps, for example, opening TCP/UDP connections is not possible. Instead, web apps are constrained to HTTP, WebSockets, WebRTC, and most recently WebTransport.
+
+There are good reasons for this like security and resource management, but ultimately, it means that using IPFS on the web is different to native binaries.
+
+## Key IPFS operations: Addressing, Retrieving, and Providing
+
+As a developer, IPFS exposes three main operations for interacting with the network:
+
+- **Addressing data with CIDs** (also known as **merkleizing**): taking arbitrary data and encoding so its addressable by CID. For example, given a directory of files, merkleizing it so it can be addressed and retrieved by CID.
+- **Retrieving data by CID**: given a CID, IPFS finds providers (peers who share the block), connects to them, fetches the blocks, and verifies that the retrieved data is what the CID represents.
+- **Providing data by CID**: making data addressed by a CID retrievable by other peers, either by running a node or with a pinning service.
+
+## Addressing data by CID
+
+As mentioned above, the first step in the [lifecycle of data in IPFS](../concepts/lifecycle.md) is to address it by CID.
+
+When addressing data by [CIDs](../concepts/glossary.md#cid) you will need to choose:
+
+- [hash function](../concepts/glossary.md#hash). For use in browsers, the default and recommended hash function is `sha2-256` which is also the default for [Helia](https://github.com/ipfs/helia).
+- [multicodec](../concepts/glossary.md#multicodec), which is the format of the data you are addressing and is used to help decode data. CIDs support a wide range of multicodecs, but for most intents and purposes, you will likely either want use:
+  - [UnixFS](../concepts/file-systems.md#unix-file-system-unixfs) for files and directories.
+  - [dag-cbor](../concepts/glossary.md#dag-cbor) for json-like structured data with binary encoding. DAG-CBOR is an extension of CBOR that adds a "link" type for CIDs, allowing for the creation of interlinked CBOR objects (which can be used to form larger linked data structures).
+
+### CID Determinism
+
+One important thing to note is that **the same data can result in different CIDs** depending on a number of factors, including the hash function, and the multicodec you use. **This is especially true for files**, where the same file, hash function and multicodec can still result in different CIDs depending on the different options that UnixFS supports.
+
+See the [forum discussion on CID profiles](https://discuss.ipfs.tech/t/should-we-profile-cids/18507) and the [DASL](https://dasl.ing/) initiative for more for more information on the nature of this problem and how the community is addressing it.
+
+For a visual demonstration of this, try the [DAG Builder](https://dag.ipfs.tech/), which visualises how files are addressed by CID with UnixFS and demonstrates how the same file can result in different CIDs, depending on the different options that UnixFS supports.
+
+### Example: Addressing an object by CID with dag-cbor
+
+For example, to address an object by CID with the `dag-cbor` multicodec and `sha2-256` hash function, you can use the following code using [Helia](https://github.com/ipfs/helia):
+
+<iframe height="300" style="width: 100%;" scrolling="no" title="Addressing an object by CID with Helia and dag-cbor" src="https://codepen.io/2color/embed/xbKpJKx?default-tab=js%2Cresult" frameborder="no" loading="lazy" allowtransparency="true" allowfullscreen="true">
+  See the Pen <a href="https://codepen.io/2color/pen/xbKpJKx">
+  Addressing an object by CID with Helia and dag-cbor</a> by Daniel Norman (<a href="https://codepen.io/2color">@2color</a>)
+</iframe>
+
+### Example: Addressing a file by CID with UnixFS
+
+<iframe height="500" style="width: 100%;" scrolling="no" title="Addressing an image by CID with Helia and UnixFS" src="https://codepen.io/2color/embed/zxONqPj?default-tab=js%2Cresult" frameborder="no" loading="lazy" allowtransparency="true" allowfullscreen="true">
+  See the Pen <a href="https://codepen.io/2color/pen/zxONqPj">
+  Addressing an image by CID with Helia and UnixFS</a> by Daniel Norman (<a href="https://codepen.io/2color">@2color</a>)
+</iframe>
+
+## Retrieval
+
+From a high level, there are several ways to retrieve data with IPFS in web applications:
+
+- Using the [Verified Fetch](https://www.npmjs.com/package/@helia/verified-fetch) library, which was modelled after the [Fetch API](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API) and returns [Response](https://developer.mozilla.org/en-US/docs/Web/API/Response) objects, with the main difference being that it allows you to fetch data by CID, abstracting away the details of content routing, transports and retrieval. For more examples and background see the [release blog post](https://blog.ipfs.tech/verified-fetch/).
+- Using the [Helia](https://github.com/ipfs/helia/) library, which is the foundation for the `@helia/verified-fetch` library, and provides a more comprehensive and modular API for interacting with the IPFS network, beyond just retrieval.
+- Using public recursive gateways, e.g. `ipfs.io` with HTTP. This is not recommended for most use cases, because it forgoes the verifiability and trustlessness enabled by content addressing. Granted, it might be the easiest way to retrieve data in a web application, but is also the most fraught with security and centralization concerns.
+
+### Example: Image retrieval with Verified Fetch
+
+<iframe height="500" style="width: 100%;" scrolling="no" title="Fetch an image on IPFS Mainnet @helia/verified-fetch" src="https://codepen.io/2color/embed/QWXKZGx?default-tab=js%2Cresult" frameborder="no" loading="lazy" allowtransparency="true" allowfullscreen="true">
+  See the Pen <a href="https://codepen.io/2color/pen/QWXKZGx">
+  Fetch an image on IPFS Mainnet @helia/verified-fetch</a> by Daniel Norman
+</iframe>
+
+## Providing data
+
+For data to be retrievable by other peers on [IPFS Mainnet](../concepts/glossary.md#ipfs-mainnet) it will need to be uploaded to a pinning service or an IPFS node.
+
+When possible, it's best to rely on client-side merkleization to address data by CID and then upload it to a pinning service or a node. [CAR files](#car-files) are a great way to do this, though they are not supported by all pinning services.
+
+### You probably don't want to provide data from a browser
+
+Browsers make for lousy servers. It's difficult to make a Web page a server, i.e. allow network incoming connections from other computers. WebRTC is the only exception, however, it has many caveats, and doesn't work in all networks.
+
+For this reason, you should never count on providing data from a browser to work.
+
+Instead, you should provide data from a long-running server that runs reliably and has a public IP. That can be a Kubo node that you run, or a [pinning service](../concepts/persistence.md#pinning-services).
+
+### CAR files
+
+The Content Archive format is a way of packaging up content addressed data into archive files that can be easily stored and transferred over the network. You can think of them like TAR files that are designed for storing collections of content addressed data.
+
+**So why would you want to use CAR files?**
+
+One of the main reasons is related to [CID determinism](#cid-determinism). As mentioned above, the same data can result in different CIDs, which can make it difficult to verify data without its content addressed representation. By packaging up the data into a CAR file, you can upload the CAR to multiple pinning services and nodes knowing they are providing the same CIDs
+
+CAR files are a great way to store content-addressed data in a way that is easy to transport and store, and Helia (and other implementations) allow you to both export and import any data you've addressed by CID into a CAR file.
+
+<iframe height="300" style="width: 100%;" scrolling="no" title="CAR export with Helia and dag-cbor" src="https://codepen.io/2color/embed/EaYoegX?default-tab=js%2Cresult" frameborder="no" loading="lazy" allowtransparency="true" allowfullscreen="true">
+  See the Pen <a href="https://codepen.io/2color/pen/EaYoegX">
+  CAR export with Helia and dag-cbor</a> by Daniel Norman (<a href="https://codepen.io/2color">@2color</a>)
+</iframe>
+
+At the time of writing, not all pinning services support CAR files, but it is a feature that is being added to more and more services. Therefore, it is a good idea to check the documentation for the pinning service you are using to see if it supports CAR files.
+
+## Conclusion
+
+This guide has covered the essential aspects of using IPFS in web applications:
+
+- The main operations: addressing/merkleizing data with CIDs, retrieving data, and providing data.
+- The challenges and limitations of using IPFS in browser environments.
+- Practical examples using modern tools like [Helia](https://github.com/ipfs/helia) and [Verified Fetch](https://www.npmjs.com/package/@helia/verified-fetch).
+- Best practices for handling data persistence through pinning services and CAR files.
+
+When building web applications with IPFS, remember these key takeaways:
+
+1. Use client-side merkleization (addressing) when adding new data to IPFS, but rely on pinning services or IPFS nodes for providing data.
+1. Be mindful of CID determinism when working with files and structured data.
+1. Consider using CAR files where possible for storage and transport of content-addressed data.
+1. Use Verified Fetch for simple retrieval or Helia for more complex IPFS interactions.
+
+By following these guidelines, you can reap the benefits of IPFS while working within the constraints of the web.

mlc_pull_req_config.json

@@ -7,5 +7,10 @@
             }
         }
     ],
+    "ignorePatterns": [
+        {
+            "pattern": "^[^/]+$"
+        }
+    ],
     "aliveStatusCodes": [200, 206, 429]
 }