Why would SQS ApproximateNumberOfMessagesVisible & ApproximateAgeOfOldestMessage go up even when Received/Deleted metrics match Sent metrics?

Question

In the CloudWatch Metrics graph below, the purple line is ApproximateNumberOfMessagesVisible and red line is ApproximateAgeOfOldestMessage. They are trending up even when NumberOfMessagesReceived (orange)/NumberOfMessagesDeleted (green) match NumberOfMessagesSent (blue).

How is this possible?

In my code, I process the message in a new thread and therefore the message is almost immediately deleted from the queue. (This is not good practice in production but this is a load testing script so I don't expect or care about exceptions)

sqsClient.receiveMessage(queueUrl).getMessages().forEach(msg -> {
    pool.execute(() -> handleSqsMessage(msg));
    sqsClient.deleteMessage(queueUrl, msg.getReceiptHandle());
});

Answer 1

If the approximateAgeOfOldestMessage is increasing then it indicates that there is a poison pill. A poison pill is a malformed message which is unable to get processed by the consumer. What is your redrive policy ? You will have to set the max-receive-count to a smaller value (say 3 for example). After the message is received 3 times by the consumer, if it was not able to process/delete it will be moved to dead letter queue. You can then analyze this poison pill.

If the number of visible messages are increasing consistently, it indicates that your consumer is unable to catch up and messages are piling up in the queue. This is not necessarily a bad sign but shouldn't be very large. Seems ok to me. You can increase the number of consumers to bring it down.

https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html#sqs-dead-letter-queues-when-to-use https://aws.amazon.com/message-queue/features/