v1.0.0

1.0.0 (2023-03-20)

⚠ BREAKING CHANGES

• The programmatic API has changed significantly, see the README for new streaming API. The CLI "commands" like --pack have changed to pack (i.e. without dashes) but are largely very similar. In the CLI, CAR files written to stdout or piped to another program (i.e. not written to disk using --output) will not have a root CID in the CAR header. Minimum Node.js version for the CLI has changed to 18.

Features

• streaming CAR packing (#148) (5f5c466)

CLI Migration Guide

Packing files into a CAR

Before:

ipfs-car --pack path/to/file/or/dir --output path/to/write/a.car

After:

ipfs-car pack path/to/file/or/dir --output path/to/write/a.car

Packing a file into a CAR, without wrapping in a directory

Before:

ipfs-car --pack path/to/files --wrapWithDirectory false --output path/to/write/a.car

After:

ipfs-car pack path/to/files --no-wrap --output path/to/write/a.car

Unpacking files

Before:

ipfs-car --unpack path/to/my.car --output /path/to/unpack/files/to

After:

ipfs-car unpack path/to/my.car --output /path/to/unpack/files/to

Listing root CIDs

Before:

ipfs-car --list-roots path/to/my.car

After:

ipfs-car roots path/to/my.car

Listing block CIDs

Before:

ipfs-car --list-cids path/to/my.car

After:

ipfs-car blocks path/to/my.car

Listing files

Before:

ipfs-car --list path/to/my.car

After:

ipfs-car ls path/to/my.car

Listing files with CIDs

Before:

ipfs-car --list-full path/to/my.car

After:

ipfs-car ls path/to/my.car --verbose

Generating CAR CID

Before:

ipfs-car --hash path/to/my.car

After:

ipfs-car hash path/to/my.car

API Migration Guide

Packing files into a CAR

The ipfs-car module now uses web streams. Note that due to the streaming nature of DAG and CAR generation the programmatic API does not produce CAR files with any roots in the CAR header.

import { createFileEncoderStream, CAREncoderStream } from 'ipfs-car'

const file = new Blob(['Hello ipfs-car!'])
const carStream = createFileEncoderStream(file).pipeThrough(new CAREncoderStream())

// carStream.pipeTo(somewhereWritable)

Obtaining the root CID

The root CID is the CID of final block generated by the file/directory encoder stream. Use a transform stream to record the CID of the last block generated:

import { createFileEncoderStream, CAREncoderStream } from 'ipfs-car'

const file = new Blob(['Hello ipfs-car!'])
let rootCID

await createFileEncoderStream(file)
  .pipeThrough(new TransformStream({
    transform (block, controller) {
      rootCID = block.cid
      controller.enqueue(block)
    }
  }))
  .pipeThrough(new CAREncoderStream())
  .pipeTo(new WritableStream())

console.log(rootCID.toString())

Adding root CIDs to the CAR header

If you need root CIDs in the CAR header, there are two approaches you can use:

1. Buffer all the DAG blocks, then encode with known root:

   import { createFileEncoderStream, CAREncoderStream } from 'ipfs-car'
   
   const file = new Blob(['Hello ipfs-car!'])
   const blocks = []
   
   // buffer the output
   await createFileEncoderStream(file)
     .pipeTo(new WritableStream({ write: b => blocks.push(b) }))
   
   const rootCID = blocks.at(-1).cid
   const blockStream = new ReadableStream({
     pull (controller) {
       if (blocks.length) {
         controller.enqueue(blocks.shift())
       } else {
         controller.close()
       }
     }
   })
   
   await blockStream
     .pipeThrough(new CAREncoderStream([rootCID])) // pass root to CAR encoder
     .pipeTo(new WritableStream())

2. Write to disk with placeholder CID, then update after DAG is completely generated (Note: Node.js only):

   import fs from 'fs'
   import { Writable } from 'stream'
   import { CarWriter } from '@ipld/car/writer'
   import { CID } from 'multiformats/cid'
   import { createFileEncoderStream, CAREncoderStream } from 'ipfs-car'
   
   // Root CID written in CAR file header before it is updated with the real root CID.
   const placeholderCID = CID.parse('bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi')
   
   const file = new Blob(['Hello ipfs-car!'])
   let rootCID
   
   await createFileEncoderStream(file)
     .pipeThrough(new TransformStream({
       transform (block, controller) {
         rootCID = block.cid
         controller.enqueue(block)
       }
     }))
     .pipeThrough(new CAREncoderStream(placeholderCID))
     .pipeTo(Writable.toWeb(fs.createWriteStream('path/to/my.car')))
   
   // update roots in CAR header
   const fd = await fs.promises.open(opts.output, 'r+')
   await CarWriter.updateRootsInFile(fd, [rootCID])
   await fd.close()

Unpacking files from a CAR

This is no longer provided by this library, but is easy to do with @ipld/car and ipfs-unixfs-exporter modules:

import { CarIndexedReader } from '@ipld/car/indexed-reader'
import { recursive as exporter } from 'ipfs-unixfs-exporter'

const reader = await CarIndexedReader.fromFile('path/to/my.car')
const roots = await reader.getRoots()

const entries = exporter(roots[0], {
  async get (cid) {
    const block = await reader.get(cid)
    return block.bytes
  }
})

for await (const entry of entries) {
  if (entry.type === 'file' || entry.type === 'raw') {
    console.log('file', entry.path, entry.content)
  } else if (entry.type === 'directory') {
    console.log('directory', entry.path)
  }
}