SQS to ECS (Fargate) or SQS to Lambda to ECS

Question

This question is more of an architectural question about how best to build an ETL pipeline. At present, I have an AWS Lambda that is pinged via SQS. But processing data can take a little more than 15 minutes (runtime limit of AWS) and using the sam build for deployment results in a .zip larger than 250MB, therefore it throws an error. Therefore an alternative to AWS Lambda is needed. The alternatives what I'm seeing so far are:

SQS -> ECS (Fargate) SQS -> Lambda -> ECS (Fargate)

I don't find any hints to the pro/cons of both options, and what is generally a preferred option. Any tips on how to approach this?

Answer 1

I don't find any hints to the pro/cons of both options, and what is generally a preferred option. Any tips on how to approach this?

As soon as you start running up against the 15 minute limits, or the deployment file size limits of Lambda, then it's probably time to move to ECS Fargate.

Please note that when you configured SQS as an event source for Lambda, behind the scenes AWS is actually running a process that polls the SQS queue for messages, and then calls your Lambda function with those messages. SQS never "pushes" messages anywhere, it is a "pull" service.

With that in mind, when moving to ECS you would need to change your code so that it polls SQS for messages to process. In general the way to configure this is to have a docker image that, when spawned as a container simply polls the SQS queue forever, processing any messages it receives. Then have ECS auto-scaling configured based on the number of messages in the SQS queue. When there are 0 messages in the queue, ECS can scale down to 0 running tasks. When there are messages in the queue, ECS can scale up to however many number of tasks you wish to run in order to process the messages. There's no need for Lambda in this setup.