[out_opentelemetry] Missing mandatory 'te: trailers' header causes gRPC requests to be rejected by strict servers

[Watson1978]

Bug Report

Describe the bug
The out_opentelemetry plugin, when running in gRPC mode,
it does not send the te: trailers HTTP/2 header.

While some gRPC implementations might be lenient,
strict implementations (such as the Ruby grpc gem used by Fluentd) reject requests missing this header,
treating them as a protocol violation.

This results in a situation where the HTTP/2 connection appears successful (HTTP 200 OK),
but the gRPC call fails internally.

To Reproduce

1. Configure Fluentd with fluent-plugin-opentelemetry (which uses the Ruby grpc server).
2. Configure Fluent Bit to send logs/metrics to Fluentd using out_opentelemetry with Grpc On.
3. Send OTLP data from sample app.

graph LR
    App[Sample App]
    
    subgraph "Fluent Bit (v4.2.2)"
        FB_In(Input: opentelemetry<br>Port: 8080)
        FB_Out(Output: opentelemetry<br>Mode: gRPC)
        FB_In --> FB_Out
    end

    subgraph "Fluentd (Ruby)"
        FD_In(Source: opentelemetry<br>grpc bind: 0.0.0.0:4317)
    end

    App -- "OTLP Payload" --> FB_In
    FB_Out -- "gRPC over HTTP/2<br>(Port: 4317)<br>[Header te: trailers]" --> FD_In

Loading

sample app

require 'bundler/inline'
gemfile do
  source 'https://rubygems.org'
  gem 'opentelemetry-sdk'
  gem 'opentelemetry-metrics-sdk'
  gem 'opentelemetry-exporter-otlp'
  gem 'opentelemetry-exporter-otlp-metrics'
  gem 'opentelemetry-exporter-otlp-logs'
  gem 'opentelemetry-logs-api'
  gem 'opentelemetry-logs-sdk'

  gem 'logger'
end

require 'opentelemetry-sdk'
require 'opentelemetry-logs-sdk'

ENV['OTEL_EXPORTER_OTLP_LOGS_ENDPOINT'] = 'http://localhost:8080/v1/logs'


# Create a LoggerProvider
logger_provider = OpenTelemetry::SDK::Logs::LoggerProvider.new
# Create a batching processor configured to export to the OTLP exporter
processor = OpenTelemetry::SDK::Logs::Export::BatchLogRecordProcessor.new(OpenTelemetry::Exporter::OTLP::Logs::LogsExporter.new)
# Add the processor to the LoggerProvider
logger_provider.add_log_record_processor(processor)
# Access a Logger for your library from your LoggerProvider
logger = logger_provider.logger(name: 'my_app_or_gem', version: '0.1.0')

# Use your Logger to  emit a log record
logger.on_emit(
  timestamp: Time.now,
  severity_text: 'INFO',
  body: 'Thuja plicata',
  attributes: { 'cedar' => true },
)

logger_provider.shutdown

Configuration for Fluentd

<source>
  @type opentelemetry
  tag opentelemetry

  <grpc>
    bind 0.0.0.0
    port 4317
  </grpc>

  <http>
    bind 0.0.0.0
    port 4318
  </http>
</source>

<match **>
  @type stdout
</match>

Configuration for Fluent bit

[SERVICE]
    Log_Level debug

[INPUT]
    Name opentelemetry
    Host 0.0.0.0
    Port 8080

[OUTPUT]
    Name opentelemetry
    Match *
    Host 0.0.0.0
    Port 4317
    Grpc On

Expected behavior
According to the gRPC over HTTP/2 specification, the te header with value trailers is mandatory for requests.

Call-Definition → Method Scheme Path [Authority] TE [Timeout] Content-Type [Message-Type] [Message-Encoding] [Message-Accept-Encoding] [User-Agent]

Ref. https://github.com/grpc/grpc/blob/master/doc/PROTOCOL-HTTP2.md

Screenshots

Your Environment

• Version used: 4.2.2
• Configuration:
• Environment name and version (e.g. Kubernetes? What version?):
• Server type and version:
• Operating System and version:
• Filters and plugins:

Additional context

[Watson1978]

When I add the code to send te: trailers like following,
Fluentd can receive the data expectedly.

diff --git a/plugins/out_opentelemetry/opentelemetry.c b/plugins/out_opentelemetry/opentelemetry.c
index aac8e48af..00b9960e6 100644
--- a/plugins/out_opentelemetry/opentelemetry.c
+++ b/plugins/out_opentelemetry/opentelemetry.c
@@ -342,6 +342,9 @@ int opentelemetry_post(struct opentelemetry_context *ctx,
 
     if (request->protocol_version == HTTP_PROTOCOL_VERSION_20 &&
         ctx->enable_grpc_flag) {
+        // -----------
+        // Add te header
+        flb_http_request_set_header(request, "te", 2, "trailers", 8);
 
         grpc_body = cfl_sds_create_size(body_len + 5);
 

[edsiper]

hi @Watson1978 thanks for reporting this problem and providing a workaround, taking a look now

[edsiper]

fixed in #11347 (will be available in v4.2.3)

[Watson1978]

Thanks!!