How to scale a GKE deployment according to an AWS SQS queue size

Question

Might be a strange one to ask, but found myself with:

• An AWS Simple Queue that holds messages to process.
• A deployment in a Kubernetes cluster on Google Cloud (GKE) that processes this queue.

I want to scale the deployment according to the queue size. A simple logic for example:

• Queue size = 0 => deploy 3 pods
• Queue size 0 > 10 > 1000 => deploy 20 pods
• Queue size < 1000 => deploy 100 pods

Turns out that this isn't such a simple task, and I'm looking for ideas.

I tried to achieve this via the Horizontal pod autoscaler, but it looks like an impossible task.

My best idea is an AWS Lambda that monitors the queue (by messages or a cron schedule), and updates the Kubernetes deployment via API. The easy part was monitoring the queue size and getting the desired scale for the deployment, but I'm not managing to physically control the deployment size via the AWS Lambda.

TL:DR, I would like to achieve kubectl functionality (scale deployment), but via an external lambda running node.js code, while authenticating to my google cloud platform, And it seems really tricky as well. There are a few client libraries, but none of them really documents how to authenticate and connect to my cluster. I even thought about running the bash script from my deployment system - but running that through a lambda function using node.js 'exec' seems very very wrong.

Am I missing an easier way?

Answer 1

There's a project called Keda: https://keda.sh/docs/2.0/scalers/aws-sqs/. It supports horizontal scaling basing on a bunch of queue types. SQS is supported.

To securely access SQS/CloudWatch from a GKE one can use https://github.com/doitintl/gtoken which lets you assume AWS role from a GKE. Or in a simpler and less secure way - dedicated AWS user with periodic keys rotation. Also look at https://cloud.google.com/pubsub/docs/overview, perhaps you can replace your SQS to stay in one stack.