Exclude Logs from Datadog Ingestion

Question

I have a kubernetes cluster that's running datadog and some microservices. Each microservice makes healthchecks every 5 seconds to make sure the service is up and running. I want to exclude these healthcheck logs from being ingested into Datadog.

I think I need to use log_processing_rules and I've tried that but the healthcheck logs are still making it into the logs section of Datadog. My current Deployment looks like this:

apiVersion: apps/v1
kind: Deployment
[ ... SNIP ... ]
spec:
  replicas: 2
  selector:
    matchLabels:
      app: my-service
  template:
    metadata:
      labels:
        app: my-service
        version: "fac8fb13"
      annotations:
        rollme: "IO2ad"
        tags.datadoghq.com/env: development
        tags.datadoghq.com/version: "fac8fb13"
        tags.datadoghq.com/service: my-service
        tags.datadoghq.com/my-service.logs: |
          [{
            "source": my-service,
            "service": my-service,
            "log_processing_rules": [
              {
                "type": "exclude_at_match",
                "name": "exclude_healthcheck_logs",
                "pattern": "\"RequestPath\": \"\/health\""
              }
            ]
          }]

and the logs coming out of the kubernetes pod:

$ kubectl logs my-service-pod

{
  "@t": "2022-01-07T19:13:05.3134483Z",
  "@m": "Request finished HTTP/1.1 GET http://10.64.0.80:5000/health - - - 200 - text/plain 7.5992ms",
  "@i": "REDACTED",
  "ElapsedMilliseconds": 7.5992,
  "StatusCode": 200,
  "ContentType": "text/plain",
  "ContentLength": null,
  "Protocol": "HTTP/1.1",
  "Method": "GET",
  "Scheme": "http",
  "Host": "10.64.0.80:5000",
  "PathBase": "",
  "Path": "/health",
  "QueryString": "",
  "HostingRequestFinishedLog": "Request finished HTTP/1.1 GET http://10.64.0.80:5000/health - - - 200 - text/plain 7.5992ms",
  "EventId": {
    "Id": 2,
    "Name": "RequestFinished"
  },
  "SourceContext": "Microsoft.AspNetCore.Hosting.Diagnostics",
  "RequestId": "REDACTED",
  "RequestPath": "/health",
  "ConnectionId": "REDACTED",
  "dd_service": "my-service",
  "dd_version": "54aae2b5",
  "dd_env": "development",
  "dd_trace_id": "REDACTED",
  "dd_span_id": "REDACTED"
}

EDIT: Removed 2nd element of the log_processing_rules array above as I've tried with 1 and 2 elements in the rules array.

EDIT2: I've also tried changing log_processing_rules type to INCLUDE at match in an attempt to figure this out:

"log_processing_rules": [
  {
    "type": "include_at_match",
    "name": "testing_include_at_match",
    "pattern": "somepath"
  }
]

and I'm still getting the health logs in Datadog (in theory I should not as /health is not part of the matching pattern)

Answer 1

All of these answers are correct in their own ways, but my specific issue was that the Datadog annotations for the source and service were not properly quoted:

        ad.datadoghq.com/my-service.logs: |
          [{
            "source": "my-service",    # Needs Quotes
            "service": "my-service",   # Needs Quotes
            "log_processing_rules": [
              {
                "type": "exclude_at_match",
                "name": "exclude_healthcheck_logs",
                "pattern": "\"RequestPath\": \"\/health\""
              }
            ]
          }]