How to work with CAR files with NestJS

What is a CAR File?

A CAR file is a representation of a set of blocks in a DAG (Directed Acyclic Graph) structure. It's widely used in decentralized storage protocols like IPFS for its ability to store data efficiently while ensuring content addressability via CIDs (Content Identifiers).

Steps to Compact Multiple Files into a CAR File

1. Prepare the Files
   Convert the files into byte arrays or readable streams.
   Ensure the files are properly labeled with metadata if needed.

2. Use ipfs-car or nft.storage Tools
   Tools like ipfs-car are libraries to create, manipulate, and encode CAR files.

3. Create a CAR Encoder
   Use the createFileEncoderStream, createDirectoryEncoderStream, or CAREncoderStream to encode the files into CAR format.

Implementation Example

Here’s how to implement it in a NestJS service using ipfs-car:

File controller:

import { Controller, Post, UploadedFiles, UseInterceptors } from '@nestjs/common';
import { FilesInterceptor } from '@nestjs/platform-express';
import { FileService } from './file.service';

@Controller('file')
export class FileController {
  constructor(private readonly fileService: FileService) {}

  @Post('upload-car')
  @UseInterceptors(FilesInterceptor('files'))
  async uploadFiles(@UploadedFiles() files: Express.Multer.File[]) {
    const rootCID = await this.fileService.packFilesIntoCar(files);
    console.log(`Root CID: ${rootCID}`);
    return { rootCID };
  }
}

File Service:

import { Injectable } from '@nestjs/common';
import { Blob } from 'nft.storage';
import { CAREncoderStream } from 'ipfs-car';

@Injectable()
export class FileService {
  constructor() {}

  public async packFilesIntoCar(files: Express.Multer.File[]): Promise<string> {
    let rootCID = '';

    try {
      const blobs = files.map((file) => {
        const fileBytes = new Uint8Array(file.buffer);
        return new Blob([fileBytes], { type: file.mimetype });
      });

      // Start encoding files into CAR format
      const encoder = new CAREncoderStream();

      // Add all files to encoder
      for (const blob of blobs) {
        const fileStream = blob.stream();
        const writer = encoder.getWriter();
        for await (const chunk of fileStream) {
          writer.write(chunk);
        }
        writer.close();
      }

      const rootBlock = await encoder.getRootBlock();
      rootCID = rootBlock.cid.toString();

      return rootCID;
    } catch (error) {
      throw new Error(`Error creating CAR file: ${error.message}`);
    }
  }
}

Explanation

File Controller:

Exposes an endpoint to upload files (/upload-car).

• Intercepts files via FilesInterceptor (Multer middleware).
• Calls the packFilesIntoCar method from the FileService. File Service:

Converts files to Blob objects.

• Uses CAREncoderStream to encode files into a CAR file.
• Returns the root CID of the CAR archive.

CAREncoderStream:

• Writes file blocks to the CAR archive. Generates a root CID for referencing the CAR file.

Benefits

• Data Integrity: Each block in the CAR file has its unique CID.
• Compatibility: CAR files are natively supported in IPFS and Filecoin.
• Efficiency: Combines multiple files into a single compact archive.

Additional Enhancements

• Save the CAR file locally: Use Node.js fs module to write the CAR stream into a file.

• Upload CAR to Decentralized Storage: Integrate with nft.storage, Web3.Storage, or similar APIs.

• Add Metadata: Include additional metadata like file names, timestamps, or descriptions to make the archive more useful.