@lquerel My initial thoughts are adding the OTLP/HTTP protocol support to the existing OTLP/gRPC receiver by sharing the core infrastructure while adapting to the HTTP/1.1 protocol characteristics. The goal is to have feature parity with Go implementation, while maintaining our zero-deserialization optimization. Still high level, but tried to have some rough sketch on integration:

  +-------------------------------------------------------------+
  |                    OTLPReceiver Node                        |
  |                     (Per-Core Instance)                     |
  +-------------------------------------------------------------+
                                  |
            +---------------------+--------------------+
            |                                          |
       +----v----+                              +-----v-----+
       |  gRPC   |                              |   HTTP    |
       |  :4317  |                              |   :4318   |
       +----+----+                              +-----+-----+
            |                                          |
            | tonic::transport                         | hyper::server::conn
            | (HTTP/2 framing)                         | (HTTP/1.1)
            |                                          |
       +----v--------------------+                +----v------------------+
       | OtlpBytesCodec          |                | HttpHandler           |
       | * copy_to_bytes()       |                | * Limited::collect()  |
       | * NO decode             |                | * decompress.         |
       |                         |                | * NO decode           |
       +----+--------------------+                +----+------------------+
            |                                          |
            +--------------+--------------------------+
                           |
              +------------v---------------+
              | OtapPdata                  | <- CONVERGENCE
              |   context: Context         |
              |   payload: OtlpProtoBytes  | <- Raw serialized OTLP
              +------------+---------------+
                           |
      +------------------v-------------------------------------+   
       |  Shared Infrastructure (gRPC + HTTP)                  |   
       |    * AckRegistry (capacity = grpc + http slots)       |   
       |    * Metrics (requests_started, acks_received, etc.)  |   
       |    * Control Loop (protocol-agnostic ACK/NACK routing)|   
       |    * Precomputed responses                            |   
       +------------------+-------------------------------------+ 
                           |
              +------------v---------------+
              | Pipeline (Bounded MPSC)    |
              +----------------------------+

Example Configuration (Non-breaking for existing gRPC config):

nodes:
    receiver:
      kind: receiver
      plugin_urn: "urn:otel:otlp:receiver"
      config:
        # gRPC settings (flattened at root - existing)
        listening_addr: "127.0.0.1:4317"
        max_concurrent_requests: 100
        wait_for_result: false
        timeout: "30s"

        # HTTP settings (nested - new)
        http:
          listening_addr: "127.0.0.1:4318"
          max_request_body_size: "4MiB". # Wire + decompressed limit
          max_concurrent_requests: 100  # Independent from gRPC
          accept_compressed_requests: true # gzip/deflate/zstd
          wait_for_result: false
          timeout: "30s"

          # Connection keep-alive (optional TCP tuning)
          tcp_nodelay: true
          tcp_keepalive: "45s"
          tcp_keepalive_interval: "15s"
          tcp_keepalive_retries: 5

Key high level design points:

Initial implementation will be protobuf-only (application/x-protobuf). JSON requires deserialization, which breaks zero-copy strategy. If/when we add JSON, it will be an opt-in path with explicit conversion and bounds checks.
Connection keep-alive - hyper enables HTTP/1.1 keep-alive by default (connections reused across requests). TCP-level keep-alive is configurable via tcp_keepalive settings for high-volume scenarios.
Lazy decode (same as gRPC) - Keep OTLP bytes serialized, defer decode to downstream.
Semaphore-based admission control - Unlike gRPC which uses tonic's built-in GlobalConcurrencyLimitLayer middleware, the HTTP receiver has to use raw hyper primitives without a middleware layer. So we will manually check tokio::sync::semaphore.try_acquire_owned() at the start of each request handler, rejecting with 503 if the permit can't be acquired - this prevents memory exhaustion by enforcing the concurrency limit before allocating body buffers.
Manual body collection using http-body-util::Limited - HTTP/1.1 requires collecting the entire request body into a buffer before processing (unlike gRPC's streaming codec which handles framing automatically). We can wrap hyper's Incoming body with Limited::new(body, max_len) to enforce size limits during collection - aborting early with 400 Bad Request if the body exceeds max_request_body_size, preventing memory exhaustion from oversized payloads.
Dual-Limit Enforcement - Enforce max_request_body_size on BOTH wire and decompressed size. This is to prevent bandwidth exhaustion and decompression bombs.
Unified metrics - Single counters for gRPC + HTTP combined. I believe operators care about total receiver load, not per-protocol. We can add protocol label later if required.

OTLP/HTTP Receiver #1144

Description

Activity

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions