Datadog logs with C# and .NET

Question

I am trying to post Datadog-logs from my C# application. I managed to send my logs with the desired structure using Postman, but I just can't figure out how to achieve the same goal from the code. What I tried:

1. Using DogStatsD - but I don't want to install an agent, I'd much rather use the Datadog REST API to just post my logs.
2. Using Serilog.Sinks.Datadog.Logs - Which seems to be pretty easy to use, but I can't figure out how this works, and whether it is possible to change the log structure or not. By default, there are MessageTemplate and Properties fields in the resulting json. I'd like to be able to send my own structure in one message, rather then use the MessageTemplate. Is that possible?

The desired log to be seen in Datadog UI Logs Section:

{
    hostname: myHost
    myStuff {   
    item1: item_val1
    item2: item_val2
    }
    otherStuff: oh wow this is cool
    service: MyService
}

Here's what I sent using Postman to achieve this result:

URL: https://http-intake.logs.datadoghq.com/v1/input

Headers:

DD-API-KEY: my_api_key
Content-Type: application/json

Body:

{
    "ddsource": "mySource",
    "ddtags": "myTag: myVal, myValWithoutTag",
    "hostname": "myHost",
    "message": {
        "myStuff":
        {
            "item1": "item_val1",
            "item2": "item_val2"
        },
        "otherStuff": "oh wow this is cool"
    },
    "service": "MyService"
}

Is it possible to achieve the same (or even similar) result using datalog serilog sinks? If not, how can I achieve this result in C#?

Here is what I tried from the code:

var config = new DatadogConfiguration(url: "intake.logs.datadoghq.com", port: 443, useSSL: true, useTCP: true);
using (var log = new LoggerConfiguration().WriteTo.DatadogLogs(
        "<myApiKey>",
        source: "mySource",
        service: "myService",
        host: "myHost",
        tags: new string[] {"myTag:myVal", "myValWithoutTag"},
        configuration: config
    ).
    CreateLogger())
{
    var messageTemplate = "{message}";
    var message = new
    {
        myStuff = new
        {
            item1 = "item_val1",
            item2 = "item_val2"
        }
    };

    log.Information(messageTemplate, message);
}

With the undesired result in Datadog UI Logs section:

{
    host: myHost
    level: Information
    MessageTemplate: {message}
    Properties: {   
    message: { myStuff = { item1 = item_val1, item2 = item_val2 } }
    }
    service: myService
    Timestamp: 2021-05-17T00:13:14.2614896+03:00
}

The tags part did work, and also the host and service parts are the same. I don't mind the level and Timestamp additions, but I'd love to change the body to behave like in the Postman example (just the message as JSON). So my questions are:

• Is it possible to control the message body format using Datadog Serilog sinks?
• Is there any good alternative that I didn't try? (except for writing my own client, which is what I'm leaning towards)
• Can anyone explain to me how it works? I can't figure out the concept of the sink. Can anyone explain how it works? And why is there no actual REST HTTP client for this task?

Thanks!

Answer 1

You really no need use Agent for sending and customizing logs to Datadog. Its pretty cool for providing traces and other metrics. Unfortunately, MessageTemplate could`t be changed in Datadog logging configuration. Only put MessageTemplate like message in Serilog:

var message = "Service started";
logger.Information(message);

Properties you can add 2 ways. Using Serilog Enrich:

var config = new DatadogConfiguration(url: "intake.logs.datadoghq.com", port: 443, useSSL: true, useTCP: true);
using (var log = new LoggerConfiguration()
    .Enrich.WithProperty(item1, item_val1)
    .WriteTo.DatadogLogs(
      "<myApiKey>",
      source: "mySource",
      service: "myService",
      host: "myHost",
      tags: new string[] {"myTag:myVal", "myValWithoutTag"},
      configuration: config
).
CreateLogger())

Or push property in LogContext by middleware:

using System.Threading.Tasks;
using Microsoft.AspNetCore.Http;
using Serilog.Context;

namespace MyProject.Middlewares
{
  public class LogPropertyMiddleware
  {
    private readonly RequestDelegate _next;

    public LogUserInfoMiddleware(RequestDelegate next)
    {
        _next = next;
    }

    public async Task InvokeAsync(HttpContext context)
    {
        LogContext.PushProperty("item1", "item_val1");

        await _next.Invoke(context);
    }
  }
}

Add middleware to startup:

public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
   .....
   .....
   app.UseMiddleware<LogPropertyMiddleware>();
   .....
}