Stream file content to the client using nextjs app router

Question

I am trying to stream a file content to the client using nextjs app router and route handler.

Currently, I have the following code and use polling on the client to refresh the content but would like to use streaming instead.

// app/api/route.ts
async function getLogs(lines: number): Promise<string> {
  const conn = new ssh2.Client();

  return new Promise((resolve, reject) => {
    conn.connect(sshConfig);
    conn.on("ready", function () {
      console.log("Client :: ready");
      conn.exec(
        `tail -${lines} ${filePath}`,
        function (err, stream) {
          if (err) throw err;
          stream
            .on("close", function (code: string, signal: string) {
              conn.end();
            })
            .on("data", function (data: string) {
              resolve(data);
            })
            .stderr.on("data", function (data) {
              reject(data);
            });
        }
      );
    });
  });
}

export async function GET(request: Request) {
  const { searchParams } = new URL(request.url);
  const lines = searchParams.get("lines") ?? "10";
  const data = await getLogs(Number(lines));

  return Response.json({ logs: data.toString() });
}

The nextjs doc has the following example but I'm not sure how to adapt it to my use case. I thought about using tail -f and yielding instead of resolving promise but could not figure how to make generator works with callback functions. Also, how should I consume the stream in the client?

// https://developer.mozilla.org/docs/Web/API/ReadableStream#convert_async_iterator_to_stream
function iteratorToStream(iterator: any) {
  return new ReadableStream({
    async pull(controller) {
      const { value, done } = await iterator.next()
 
      if (done) {
        controller.close()
      } else {
        controller.enqueue(value)
      }
    },
  })
}
 
function sleep(time: number) {
  return new Promise((resolve) => {
    setTimeout(resolve, time)
  })
}
 
const encoder = new TextEncoder()
 
async function* makeIterator() {
  yield encoder.encode('<p>One</p>')
  await sleep(200)
  yield encoder.encode('<p>Two</p>')
  await sleep(200)
  yield encoder.encode('<p>Three</p>')
}
 
export async function GET() {
  const iterator = makeIterator()
  const stream = iteratorToStream(iterator)
 
  return new Response(stream)
}

Answer 1

Streaming files from Route Handlers is mostly about turning an fs.ReadStream that you get when reading the file, to a ReadableStream from the web platform.

If you happen to be able to define a ReadableStream directly (for instance you are generating the stream from an algorithm rather than opening an existing file) you can skip the conversion.

This is a very short summary from my complete article on the topic How to stream files from Next.js Route Handlers

Step 1: from fs.ReadStream to iterator

// Syntax taken from 
// https://github.com/MattMorgis/async-stream-generator
// itself taken from 
// https://nextjs.org/docs/app/building-your-application/routing/router-handlers#streaming
// probably itself taken from 
// https://nodejs.org/api/stream.html
async function* nodeStreamToIterator(stream: fs.ReadStream) {
    for await (const chunk of stream) {
        yield new Uint8Array(chunk);
    }
}

Step 2: from iterator to ReadableStream

function iteratorToStream(iterator: AsyncIterator<Uint8Array>) {
  return new ReadableStream({
    async pull(controller) {
      const { value, done } = await iterator.next();
      if (done) {
        controller.close();
      } else {
        controller.enqueue(value);
      }
    },
  });
}

Step 3: reusable helper to convert files to streams:

export function streamFile(path: string): ReadableStream {
    const nodeStream = fs.createReadStream(path);
    const data: ReadableStream = iteratorToStream(
        nodeStreamToIterator(
            nodeStream
        )
    )
    return data
}

Step 4: the final endpoint

 // File "app/api/serve-file/route.ts"
export function GET() {
 const stats = await fs.promises.stat(filePath);
 const stream: ReadableStream = streamFile(filePath)
 return new Response(stream, {
   status: 200,
   headers: new Headers({
    "content-disposition": 
    `attachment; filename=${path.basename(
       filePath
     )}`,
    "content-type": "application/zip",
    "content-length": stats.size + "",
 })
}