OpenTelemetry SDK: Failed to export spans. The request could not be executed. Full error message: FRAME_SIZE_ERROR: 4740180

Question

I'm trying to export the spans to HoneyComb through Otel-Collector

Using below Otel-collector configs I'm able to POST test Trace to HoneyComb UI as below

curl -i http://localhost:4318/v1/traces -X POST -H "Content-Type: application/json"  -d @span.json

My Otel-Collector versions that I tried are as below

receivers:
  otlp:
    protocols:
      grpc:
      http:

exporters:
  logging:
   loglevel: debug
  otlphttp:
    endpoint: "https://api.honeycomb.io"
    headers:
      "x-honeycomb-team": "YOUR_API_KEY"
      "x-honeycomb-dataset": "YOUR_DATASET"

processors:
  batch:

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [logging, otlphttp]

OR

receivers:
  otlp:
    protocols:
      grpc:
      http:

exporters:
  logging:
   loglevel: debug
  otlp:
    endpoint: "http://api.honeycomb.io:443" # even simply api.honeycomb.io:443
    headers:
      "x-honeycomb-team": "YOUR_API_KEY"
      "x-honeycomb-dataset": "YOUR_DATASET"

processors:
  batch:

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [logging, otlp]

docker-compose.yaml is as below

version: "2"
services:

  # Collector
  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    restart: always
    command: ["--config=/etc/otel-collector-config.yaml", "${OTELCOL_ARGS}"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    ports:
      - "1888:1888"   # pprof extension
      - "8888:8888"   # Prometheus metrics exposed by the collector
      - "8889:8889"   # Prometheus exporter metrics
      - "13133:13133" # health_check extension
      - "4343:4317"   # OTLP gRPC receiver
      - "4318:4318"   # OTLP http receiver

OpenTelemetry implementation is

Resource resource = Resource.getDefault().merge(Resource.builder().put(SERVICE_NAME, MY_SERVICE_NAME).build());

        OtlpGrpcSpanExporter otlpGrpcSpanExporter = OtlpGrpcSpanExporter.builder().setEndpoint("http://localhost:4318").build();

        SdkTracerProvider sdkTracerProvider = SdkTracerProvider.builder()
                .addSpanProcessor(BatchSpanProcessor.builder(otlpGrpcSpanExporter).build())
                .setResource(resource)
                .build();

        openTelemetry = OpenTelemetrySdk.builder().
                .setTracerProvider(sdkTracerProvider)
                .buildAndRegisterGlobal();

Tracer tracer = openTelemetry.getTracer("My Tracer");

span = tracer.spanBuilder(messageType + " handler span").startSpan();
span.setAttribute("messageType", messageType);

//some logic

span.end();

I'm seeing below error in my application. Not sure what is I'm missing here?

Jul 06, 2023 12:52:04 PM io.opentelemetry.exporter.internal.grpc.GrpcExporterUtil logUnimplemented
SEVERE: Failed to export metrics. Server responded with UNIMPLEMENTED. This usually means that your collector is not configured with an otlp receiver in the "pipelines" section of the configuration. If export is not desired and you are using OpenTelemetry autoconfiguration or the javaagent, disable export by setting OTEL_METRICS_EXPORTER=none. Full error message: unknown service opentelemetry.proto.collector.metrics.v1.MetricsService



Jul 06, 2023 12:52:08 PM io.opentelemetry.sdk.internal.ThrottlingLogger doLog
SEVERE: Failed to export spans. The request could not be executed. Full error message: FRAME_SIZE_ERROR: 4740180

Answer 1

There are two errors reported by your application that I can see. The first: Server responded with UNIMPLEMENTED happens when you attempt to use the gRPC exporter with a server expecting HTTP. Change your collector endpoint in the application to use port 4317 to use gRPC.

The second error indicates that the message you are attempting to send to the collector is too large. Either adjust the settings of your batch span processor in the application to send more frequently, adjust your collector's OTLP receiver configuration to increase max_recv_msg_size_mib, or enable compression on the exporter to reduce the size of the exported payload.