NodeJS: create zip, but on disk, not in memory

Question

I need to create a zip file with around 1GB of data (but that amount will grow in the future). Initially I built this with AdmZip, but since our VM doesn't have a whole lot of memory, (nor does it need it, except for when it's creating this zip file, so I don't want to scale up), it runs out of memory and crashes.

I then rebuilt the process in order to use command line zip, but after getting that all set up and releasing to the server, I had to find out the hard way that that command is not available on an Azure WebApp.

So now I need to rebuild it again. Can anyone recommend me the right node library/libraries to do zipping and unzipping on disk instead of in memory?

(the zip file format is not a hard requirement, so if any other file formats are better for this, I'm open to suggestions!)

Answer 1

I'd have a look at Archiver JS, this is pretty powerful, allowing a streaming interface to .zip files, for example:

const fs = require('fs');
const archiver = require('archiver');

const zipFile = archiver('zip', { zlib: { level: 9 }});

zipFile.on('warning', error => {
    console.log('warning:', error)
});

zipFile.on('error', error => {
    console.error('error occurred :', error)
});

const writeStream = fs.createWriteStream(__dirname + '/test.zip');
zipFile.pipe(writeStream);

// Append test file
zipFile.append(fs.createReadStream('file1.txt'), { name: 'file_1.txt' });

// Append another file...
zipFile.append(fs.createReadStream('file2.txt'), { name: 'file_2.txt' });

zipFile.finalize();

Full docs are here: https://www.archiverjs.com/docs/archiver