How to add custom tag to default metrics using opentelemetry collector and opentelemetry Java agent

Question

I have a Spring Boot plugin which has 3 custom metrics connected to a main microservice and I have already exposed its metrics with the help of OpenTelemtry collector and OpenTelemtry Java agent. But the issue is that I need to add custom tags to the generated default metrics in OpenTelemtry. Is there any solution for this?

]

I tried the solutions present in Sumologic docs but it didn't work for me, and the custom tag configuration through your code option isn't specified enough.

Answer 1

If the attributes are not dependent on the specific application, you can add them through the collector itself.

Here is an example that adds the "deployment.environment" standard attribute to every telemetry passing through the collector (imagine this is your "staging collector", in this example):

receivers:
  otlp:
    protocols:
      grpc:

processors:
  batch:
    timeout: 10s

  resource:
    attributes:
    - key: deployment.environment
      value: "staging"
      action: upsert

exporters:
  datadog:
    api:
      key: ${DATADOG_APIKEY}

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [resource, batch]
      exporters: [datadog]
    metrics:
      receivers: [otlp]
      processors: [resource, batch]
      exporters: [datadog]
    logs:
      receivers: [otlp]
      processors: [resource, batch]
      exporters: [datadog]

You can look at the documentation for the "resource" processor here:

• OpenTelemetry Collector - Resource Processor

Answer 2

When you add spring-boot-starter-actuator to your project, by default, you have http_server_requests_second_count and a lot of other useful metrics published on the endpoint .../actuator/prometheus.

Here are two places where you can add your metrics if it fits your use-case:

• Assuming you are using actuator, in your spring boot app, put in the property file:

management:
  metrics:
    tags:
      custom_metric: value

For example, you may want to add a tag that represent the environment where your app is running. application.qa.yml:

management:
  metrics:
    tags:
      environment: dev

application.preprod.yml:

management:
  metrics:
    tags:
      environment: preprod

• You can add it inside the opentelemetry-collector configuration. For that, you have to use opentelemetry-collector-contrib not the base opentelemetry-collector. (removed unnecessary configs)

receivers:
  ...
  

processors:
  metricstransform:
    transforms:
      - include: .*
        match_type: regexp
        action: update
        operations:
          - action: add_label
            new_label: environment
            new_value: preprod
  batch:

exporters:
  ...

service:
  pipelines:
    traces:
      ...

    metrics:
      ...
      processors: [metricstransform, batch]
      ...

Refer OTEL docs for latest update https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/metricstransformprocessor#configuration