Can you pass meta data along with a stream?

Question

When I pipe something like an image file through a stream is there any way to send an meta object along with it?

My server gets sent an image from a user. The image gets pushed through a set of streams that perform various actions.

The final stream emits a data event, it passes the resulting image buffer into a callback but I lose all context for the user. I need to keep the resulting image tied to the user's id and some other meta data.

Ideal:

stream.on('data', function(img, meta){
   ...
})

Thanks for any possible solutions!

Answer 1

In short, no, there's nothing built into Node.js to support including metadata with streams. You do have some other options, though, including:

1. You could use a closure to track the meta data separately from the stream. For example:

   function handleImage(imageStream) {
     var meta = {...};
     imageStream.pipe(otherStreams).on('data', function(image) {
        // you now have `image` and `meta` variables at your disposal here.
     }
   }
   

   The downside of this is that the metadata is not available to your otherStreams.

   This is a good solution if your other streams are third-party code outside of your control, of if they don't need to know about the metadata.

2. You could do something similar to HTTP headers, where all the data up to a certain point is meta data, and everything after it is the image. (In HTTP, the deliminator is wherever \n\n occurs first.) All of your streams in the chain have to know about this and handle it though.

   If you know your metadata will always be in one chunk and none of your streams split or merge chunks, then you could simplify this a bit and just say that the first (or last) chunk is always metadata.

3. Switch to an object stream like Amoli mentioned in his answer. Here you would pass {image: imgData, meta: {...}}. You would then have to update your other streams to expect this format.

   The main downside of this method, though, is that you either have to pass the metadata multiple times, cache it somewhere for each stream that needs it, or pass your entire image as one chunk (which kind of kills the entire point of "streams"). And, from what I've been told, node.js can optimize text/binary streams better than object streams. So, this probably isn't a good approach for your situation.

4. https://github.com/dominictarr/mux-demux might be helpful here. It combines multiple streams into one, so you could have separate image and meta streams. I'm not sure how well it would work for your situation though. You'd probably need to update all of your streams to be aware of it.

I know I said that all but the first option require modifying the other streams, but there is a way around that: you could create a generic "stream wrapper" that splits up the image and meta data and passes just the image data through the main stream, and has the meta data bypass it and go on to the next one down the chain. This gets ugly fast though, so probably not the best idea.

Answer 2

Basically, whenever you want to read or write any objects which are not strings or buffers, you’ll need to put your stream into objectMode

Example (source):

function S3Lister (s3, options) {
  options || (options = {});
  stream.Readable.call(this, { objectMode : true });

  this.s3 = s3; // a knox-like client.
  this.marker = options.start;
  this.connecting = false;
  this.ended  = false;
}

util.inherits(S3Lister, stream.Readable);

We set the stream to use objectMode as we want to return not just data but also some metadata.

For more information:

• Node.js Docs stream object mode
• An introduction to nodes streams