compressing and handling FormData within streams

[nkitku]

I want to use form data as an archive for files for an experiment since form data is available in both Node.js and browsers and without any additional dependencies.

I plan to add multiple files to the form data, and compress some text-type files within the stream. For this, I need to add extra meta-information like content encoding. I thought of using an object in JSON format as the key in the form data. Essentially, I'll prepare an array of objects for meta and data keys. The meta key will be an object containing path, name, size, and optionally content encoding etc. I'll then convert this array into FormData, converting the meta object to JSON and placing it in the key, and the data in the second parameter.

const dataToSend = [
  {
    meta: {
      path: 'path/to/file',
      name: 'file.txt',
      size: 100,
      encoding: 'br',
    },
    data: File,
  },
  ...
];

const formData = new FormData();
dataToSend.forEach(({ meta, data }) => {
  formData.append(JSON.stringify(meta), data);
});

However, I want to extract it on the server using stream to handle large archives efficiently, and I'm not sure how to achieve this with form data. I can't find any documentation on how to extract the form data as a stream. I can only find examples of how to extract it as a buffer or string. I want to avoid this since it will require me to load the entire file into memory, which is not ideal for large files.

i used the following code to extract the form data but it is not a stream:

const body = [];
for await (const chunk of req) body.push(chunk);
const type = req.headers['content-type'];
const formData = await new Response(new Blob(body, { type })).formData();

for now I have two questions:

1. How to compress a file stream before adding it to the form data?
2. On the backend, how to write the FormData directly from the request stream to a file writeable stream and also if file is compressed, decompress within the stream?

I am also thinking to store form-data as a file some thing like tar file and later can read and extract any required file from it. and also add or delete files from it. but i am not sure how to do it without loading the whole file into memory. any help will be appreciated. thanks

[metcoder95]

You can open a ReadeableStream that reads the file on disk, pipe it to a Transform stream that reads and compresses the chunks, and finally create a Blob out of the outcome of the Transform stream.

I imagine something like

const { Redeable } = require('node:stream');
const { pipeline } = require('node:stream/promises');
const { blob } = require('node:stream/consumers');
const fs = require('node:fs');
const zlib = require('node:zlib');

const readable = new Readeable();

async function run() {
  await pipeline(
    fs.createReadStream('archive.txt'),
    zlib.createGzip(),
    redeable,
  );
  
const blobResult = await blob(redeable)
formData.append(blobResult);
}

run().catch(console.error);

To decompress and consume, you can do the exact opposite of the flow, start by passing the redeable stream from the request, pass it to an uncompress stream, and open an fs.createWriteStream for depositing the chunks.

Disclaimer
My example is pretty rough, haven't tested, you might need to do a lot of tweaks to it 🙂

[nkitku]

thx, but I am looking for web/browser compatible way

[metcoder95]

See Compression and Decompression streams. The theory is the same, just the application will difer