Simplifying AWS SDK GO v2 testing/mocking [closed]

Question

Amazon has a super useful article describing how to unit test AWS SDK Go v2. I understand their motivation to depart from the "old" way of unit testing the v1 API, which I think aligns with this CodeReviewComment.

However, I'm running into a situation that is causing some havoc for me.

Let's say I'm calling s3.HeadObject() and, if some arbitrary condition is met, I then call s3.GetObject(). So, I design my API so that I can create client APIs for each of these operations separately (in pseudoGo).

type HeadObjectAPIClient interface {
    HeadObject(ctx context.Context, params *s3.HeadObjectInput, optFns ...func(*s3.Options)) (*s3.HeadObjectOutput, error)
}

func GetHeadObject(api HeadObjectAPIClient, params...) (*s3.HeadObjectOutput, error) {
    /* setup logic */
    return api.HeadObject( /* params */ )
}

type GetObjectAPIClient interface {
    GetObject(ctx context.Context, params *s3.GetObjectInput, optFns ...func(*s3.Options)) (*s3.GetObjectOutput, error)
}

func GetObject(api GetObjectAPIClient, params...) (*s3.GetObjectOutput, error) {
    /* setup logic */
    return api.GetObject( /* params */ )
}

func DoSomething(headAPI HeadObjectAPIClient, getAPI GetObjectAPIClient, params...) error {
    headOutput, err := GetHeadObject(headAPI, ...params...)
    if err != nil {
        return err
    }
    // process headOutput then ...
    getOutput, err := GetObject(getAPI, ...params...)
    if err != nil {
        return err
    }
    // process getOutput
}

So, the DoSomething method is taking two APIs in addition to other parameters. My natural inclination is to just make some kind of struct:

type DoSomethingService struct {
    headAPI HeadObjectAPIClient
    getAPI GetObjectAPIClient
}

func (s DoSomethingService) DoSomething(params...) error {
    headOutput, err := GetHeadObject(s.headAPI, ...params...)
    if err != nil {
        return err
    }
    // process headOutput then ...
    getOutput, err := GetObject(s.getAPI, ...params...)
    if err != nil {
        return err
    }
    // process getOutput
}

Maybe that is ok ... but then I ask, well why not just make en embedded interface and avoid the struct:

type DoSomethingAPIClient interface {
    HeadObjectAPIClient
    GetObjectAPIClient
}

func DoSomething2(api DoSomethingAPIClient, params...) error {
    headOutput, err := GetHeadObject(api, ...params...)
    if err != nil {
        return err
    }
    // process headOutput then ...
    getOutput, err := GetObject(api, ...params...)
    if err != nil {
        return err
    }
    // process getOutput
}

I can probably think of a couple more ways to make this happen, but I think I've made my point.

I'm looking for a good strategy to enable testing/mocking, while avoiding proliferating client interfaces for every single call into the SDK.

Answer 1

Yes, you're absolutely right, testing AWS SDK v2 is too way verbose.The main issue is that AWS removed interfaces from service clients (like s3.Client), so we can't just mock them directly like we did in v1. That forces us to stub entire clients, which is a pain.

In my opinion, it's better to Wrap AWS clients in a small interface and mock that instead.

Because of using struct instead of interfaces in AWS SDKv2 you cannot mock s3.Client directly.

How to fix? Instead of testing against s3.Client, define a minimal interface for only the methods you need:

type S3API interface {
    PutObject(ctx context.Context, params *s3.PutObjectInput) (*s3.PutObjectOutput, error)
}

type S3Client struct {
    Client *s3.Client
}

func (s *S3Client) PutObject(ctx context.Context, params *s3.PutObjectInput) (*s3.PutObjectOutput, error) {
    return s.Client.PutObject(ctx, params)
}

Now in your real code, you should depend on S3API not s3.Client which makes your mocking simpler.

With the interface in place, we don’t need AWS SDK stubs anymore. We can just do this:

type MockS3 struct{}

func (m MockS3) PutObject(ctx context.Context, params *s3.PutObjectInput) (*s3.PutObjectOutput, error) {
    if *params.Bucket == "fail-bucket" {
        return nil, errors.New("mocked AWS error")
    }
    return &s3.PutObjectOutput{}, nil
}

And you can do this magic everywhere like your entire code, and you might use a robust level of abstraction which not depend on AWS SDK.

See this:

type MyUploader struct {
    s3Client S3API
}

func (u *MyUploader) Upload(ctx context.Context, bucket, key string, body []byte) error {
    _, err := u.s3Client.PutObject(ctx, &s3.PutObjectInput{
        Bucket: &bucket,
        Key:    &key,
        Body:   body,
    })
    return err
}

With this setup, your service doesn’t care whether it’s using a real AWS client or a mock—it just calls PutObject().