Getting a Readable from getObject in AWS S3 SDK v3

Question

I'm in the process of converting a library that uses v2 of the AWS S3 SDK to v3.

My library is an implementation of an interface of another library. The other library offers the interface to abstract file storage (e.g. have storage in the local file system or the cloud).

At the moment I have this function (v2 SDK):

public async getFileStream(filename: string): Promise<Readable> {
  return this.s3.getObject({
                Bucket: this.bucket,
                Key: filename,
            })
            .createReadStream();
}

The new v3 SDK doesn't have createReadStream anymore. I've checked various sites on the Internet and they all suggest to use .Body.transformToWebStream().pipe. I can't do that as I need to return a Readable.

I've tried doing it like this (v3 SDK):

import { Readable } from 'node:stream';

public async getFileStream(filename: string): Promise<Readable> {
   return Readable.fromWeb(
       (await this.s3.getObject({
                    Bucket: this.bucket,
                    Key: filename,
              })
        ).Body.transformToWebStream()
   );
}

I get the following error:

Argument of type 'ReadableStream<any>' is not assignable to parameter of type 'import("stream/web").ReadableStream<any>'.
  Type 'ReadableStream<any>' is missing the following properties from type 'ReadableStream<any>': values, [Symbol.asyncIterator]ts(2345)

It looks like the native Node ReadableStream<any> and S3's ReadableStream<any> are not the same.

So how can I get a Readable with the new AWS S3 v3 SDK?

Answer 1

After wasting about a day of effort finally managed to get the correct typings for GetObjectCommand in SDK v3. The reason you are actually getting this error is because the s3 sdk client is written to support both browser and nodejs streams, but these streams are not interchangeable. So they are using StreamingBlobPayloadOutputTypes from https://www.npmjs.com/package/@smithy/types/v/2.2.0 to provide support for browser and nodejs environments. So you have to narrow down the returned types by using the correct typing when constructing the s3 client.

import { NodeJsClient } from "@smithy/types";
import { GetObjectCommand, S3Client } from "@aws-sdk/client-s3";

const s3Client = new S3Client({}) as NodeJsClient<S3Client>;

According to the docs from @smithy/types

This is mostly relevant to operations with streaming bodies such as within the S3Client in the AWS SDK for JavaScript v3.

Because blob payload types are platform dependent, you may wish to indicate in your application that a client is running in a specific environment. This narrows the blob payload types. Blockquote

Once you have defined s3 Client as above then you can see that awaited s3.getObject now directly return NodeJS.ReadableStream in Body.

    public async downloadFile(path: string): Promise<NodeJS.ReadableStream> {
        logger.info("starting download from s3", { path });

        const body = (
            await this.s3Client.send(
                new GetObjectCommand({
                    Bucket: config.s3.bucketName,
                    Key: `${BASE_PATH}/${path}`,
                }),
            )
        ).Body;
        return body;
    }