How do you monitor Azure Event Hub consumer lag?

Question

We have been trying monitor consumer lag on Event Hub partitions.

I have looked in the azure portal and the Event Hub Api's but so far found nothing.

Was wondering if anyone has tried this and if so could you point me in the right direction.

Thanks.

Answer 1

You can compare the Message Sequence of the current message being processed, against the last sequence number of the message received for a partition. The difference between these numbers is 'how far behind' the latest message your processing has fallen. To get the details of the last message received in a partition, you need to access a PartitionContext object.

How I've implemented this is with Azure Function and a Custom Metric in Application Insights. As a batch of messages is received, I calculate the difference between the values and write the metric, which allows me to track this in Grafana and raise alerts when required.

I wrote a medium article on how to achieve this here - https://medium.com/@dylanm_asos/azure-functions-event-hub-processing-8a3f39d2cd0f

Answer 2

There a few different solutions for this:

Use Event Hub Premium and Dedicated Tiers

As written by makhdumi and zhangyunbo, Event Hub offers a built-in metric for the lag in the Premium and Dedicated SKUs. The way this works is that you configure Diagnostic Settings for your Event Hub Namespace or cluster and send the Application Metrics Logs category to a Log Analytics Workspace. Here is the official documentation for this. I have written a how-to, including an example for monitoring this metric using an Azure Monitor alert, which you can find here.

Use an app from the Azure Marketplace

If you are on the Basic or Standard tier and you do not want to implement and maintain this yourself (see below), you can use this offer from the Azure Marketplace: Lag Metrics for Event Hubs.

It automatically retrieves all Event Hubs of a namespace, creates lag metrics for all consumer groups and sends them to Application Insights / Log Analytics Workspace. An example for deploying this and for configuring Alert rules can be found on GitHub.

Disclaimer: I am the author of this app

Implement the metric yourself

See Dylan Morley's answer and 0xR's comment for example implementations in C# and Typescript. There is also a newer (2023) blog post from Microsoft that also explains this, including an example in C#.