Application Insights - How to set Custom Operation Id

Question

In our current on-prem setup we have 20+ .net core 3.1 API apps (separate ASP.NET Core API Apps). We have started migrating 2 APi app to Azure App Service tagged with a single Application Insights instance.

In On-Prem, we use some other log framework which the rest of the 18 Apps. All these API apps talk to each other and all the logs are tied to some unique_id in on-prem.

Now, for the apis which is in Azure, we need to leverage the same unique_Id and co-relate everything.

In order to achieve it, I started exploring the functionality of setting a same Operation Id for the 2 apps which are hosted in azure.

Created TelemetrInitializer in both the APIs. and if i set Operational Id as shown below in both the APIs, it's works. All the logs are tied to Single Operation Id "12345"

      telemetry.Context.Operation.Id = "12345";

However, as it is obvious to make the Operation Id to be dynamic, I have changed it to the below in my First API

  telemetry.Context.Operation.Id = "CR" + Guid.NewGuid().ToString();

So, the next challenge is, I need to tie this new Operation Id in my second API's TelemetryInitiializer. In order to achieve that I tried to grab the Request-Id Header in the TelemetryInitializer of 2nd API. It's always NULL.

Is there a way to achieve this?

Thanks, Praveen Sreeram.

Answer 1

tldr: This is possible by disabling the built in dependency tracking in .NET Core and App Insights and handling it on your own. In most cases, the best thing to do is let .NET Core and App Insights do the tracking.

I uploaded a simple WebAPI app with the code I'm going to go over to Github: https://github.com/SamaraSoucy-MSFT/customoperationid

There are two things that need to be overridden to get both the headers and App Insights to get the custom operation Id. The first is the Activity the wraps the HttpClient as that controls the correlation headers. The second is the dependency tracing in App Insights.

It is possible to disable Actions completely in your HttpClients, but to minimize side effects, you can just remove the one in the client by setting Activity.Current = null

var operationId = "CR" + Guid.NewGuid().ToString();
var url = "https://www.microsoft.com";
using (var client = new HttpClient())
{
    using (var requestMessage =
        new HttpRequestMessage(HttpMethod.Get, url))
    {
        //Makes the headers configurable
        Activity.Current = null;

        //set correlation header manually
        requestMessage.Headers.Add("Request-Id", operationId);
        await client.SendAsync(requestMessage);
    }
}

The next step is to remove the App Insights default tracking for this request. Again, you can disable dependency tracking completely, or you can filter out the default telemetry for this request. Processors are registered inside the Startup class just like initializers.

services.AddApplicationInsightsTelemetryProcessor<CustomFilter>();

public class CustomFilter : ITelemetryProcessor
{
    private ITelemetryProcessor Next { get; set; }

    // next will point to the next TelemetryProcessor in the chain.
    public CustomFilter(ITelemetryProcessor next)
    {
        this.Next = next;
    }

    public void Process(ITelemetry item)
    {
        // To filter out an item, return without calling the next processor.
        if (!OKtoSend(item)) { return; }

        this.Next.Process(item);
    }

    // Example: replace with your own criteria.
    private bool OKtoSend(ITelemetry item)
    {
        var dependency = item as DependencyTelemetry;

        if (dependency == null) return true;

        if (dependency.Type == "Http"
            && dependency.Data.Contains("microsoft.com")
            //This key is just there to help identify the custom tracking
            && !dependency.Context.GlobalProperties.ContainsKey("keep"))
        {
            return false;
        }
        return true;
    }
}

Finally, in the method that makes the remote call, you need to inject a telemetry client and call TelemetryClient.TrackDependency()

var operationId = "CR" + Guid.NewGuid().ToString();

//setup telemetry client
telemetry.Context.Operation.Id = operationId;
if (!telemetry.Context.GlobalProperties.ContainsKey("keep"))
{
    telemetry.Context.GlobalProperties.Add("keep", "true");
}
var startTime = DateTime.UtcNow;
var timer = System.Diagnostics.Stopwatch.StartNew();
//continue setting up context if needed

var url = "https:microsoft.com";
using (var client = new HttpClient())
{
    //Makes the headers configurable
    Activity.Current = null;

    using (var requestMessage =
        new HttpRequestMessage(HttpMethod.Get, url))
    {
        //Makes the headers configurable
        Activity.Current = null;

        //set header manually
        requestMessage.Headers.Add("Request-Id", operationId);
        await client.SendAsync(requestMessage);
    }
}

//send custom telemetry 
telemetry.TrackDependency("Http", url, "myCall", startTime, timer.Elapsed, true);