Limited OTLP exporter request authentication methods

[majanjua-amzn]

What are you trying to achieve?

We are trying to allow the OTLP exporter to be able to authenticate dynamically to OTLP endpoints that require more than static credentials like API keys in the environment.

What did you expect to see?

We expected to see a mechanism for users of OTel to be able to interact with the HTTP client used in the OTLP exporter and/or modify the HTTP headers as the requests are sent to an OTLP endpoint, however, neither of these mechanisms exists. This capability is missing in all supported OTel instrumentation languages we've looked into (Java, Python, JS, .NET).

Additional context

This discussion was originally triggered by requests to have AWS SigV4 authentication in the OTel Java instrumentation (open-telemetry/opentelemetry-java/issues/7002). In that discussion, we found the capabilities missing in Java are also missing in the other languages, that the issue is not limited to AWS SigV4 authentication, and that dynamic authorization mechanisms for data sent to OTLP endpoints were not supported by OTel instrumentations.

As such, we brought this issue up in the OpenTelemetry Sig: Specifications meeting on 28/01/2025 (Google Doc, meeting recording). In this meeting, we discussed the issue, finding that there is some community interest to address this issue, discussing some potential solutions, and gathering some potential first steps (namely, creating this ticket).

In that meeting, the following related issue previously raised in the golang instrumentation was brought up: open-telemetry/opentelemetry-go/issues/2632. This related issue describes a similar lacking capability in the instrumentation in that golang users are unable to specify the HTTP client used to export data to the OTLP endpoint. We can see in that issue there is interest from the community and other stakeholders like Azure (comment being referenced) to be able to accommodate more complex HTTP request manipulation, especially for the purposes of secure authentication.

One discussed potential solution that is of note for future reference is to introduce callback hooks in the exporters to allow for response-driven actions like re-authentication to the OTLP endpoint, e.g. make request → 401 response → callback hook triggers reauthentication.

[dmathieu]

Discussion on the same subject in the Go repository:

• OAuth2 Authentication for OTLP-HTTP Exporter/Client opentelemetry-go#4536
• Set otel exporter headers value during runtime. opentelemetry-go#5129

[brunobat]

@trask, we don't have official support for AWS SigV4 authentication in Quarkus.
Users would have to connect to aws following the examples in the docs: https://github.com/aws-samples/sigv4a-signing-examples/tree/main/java

[srprash]

Hi @brunobat This is helpful.

However the issue here is that the clients making the call to an OTLP endpoint is abstracted away within the exporters. So it's not possible for OTel users to modify/enhance the behavior of these clients to add a complex auth process like AWS SigV4 where the entire request, including the body, is needed to construct the signature.

[srprash]

Hi folks, to address this feature request I'm proposing the following approach. Would love to get feedbacks to improve the proposal and nail down something concrete that can be implemented in the SDKs.
Tagging some folks that I think may be interested in this, but anyone is welcome to chime in. :)
@trask @jack-berg @lmolkova @majanjua-amzn @mxiamxia

NOTE:

1. The approach below is idiomatic to OpenTelemetry Python but can be generalized to fit other languages semantics.
2. The proposal does not intend to provide specifications about the solution. The intent is to explain the proposal with some examples but the exact specification is yet to be discussed and formalized.

Objective

The objective is to add a callback hook while maintaining compatibility with existing SpanExporter implementation that allows users to provide their authentication mechanisms directly to the SDK for sending telemetry to a service endpoint that requires some sort of authentication such as OAuth or AWS SigV4.

Implementation Approach

1. Introduce an authenticator hook in the SpanExporter:

   • This should be a callable that receives the HTTP request data (headers, body, URL, etc.) and returns a dict of key-value pair(s) of authentication header back to the calling exporter.
     • The idea is similar to the OpenTelemetry Java’s Authentication mechanism via a Supplier implementation but it does not work for complex auth processes like AWS SigV4 where the entire request is needed to compute the authentication header.
     • The request data passed to the authenticator must be immutable to prevent any malicious modification of the request.
   • Users can provide this hook during the exporter builder phase to inject custom logic. The logic will be called when the exported exports each batch of spans to allow for refresh of auth token/credentials.

2. Modify export Method:

   • Before sending the request, call the authenticator hook (if provided) and receive a dict of key-value pairs of headers.
   • The headers will be added to the prepared request and the request will be sent out.

Sample UX for an OpenTelemetry User

from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from otel_aws_sigv4 import sigv4_authenticator  # suppose this is the auth module for AWS SigV4
# Create an OpenTelemetry OTLP exporter with AWS SigV4 signing enabled
exporter = OTLPSpanExporter(
    endpoint="https://xray.us-east-1.amazonaws.com/v1/traces",
    authenticator=sigv4_authenticator
)

# Set up OpenTelemetry with the SigV4-signed exporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.trace import get_tracer_provider

provider = TracerProvider()
provider.add_span_processor(SimpleSpanProcessor(exporter))

# Obtain a tracer and create spans
tracer = get_tracer_provider().get_tracer("otel-example")

with tracer.start_as_current_span("example-span"):
    pass  # Simulating a traced operation

Challenges

A few technical kinks with that I could think of for this approach that we will need to address.

How do we provide a way for the auto-instrumentation users to configure the authenticator?

• We will probably need a new environment variable OTEL_OTLP_EXPORTER_AUTHENTICATOR for this configuration. The value need to be one of the registered authenticators such as oauth2, awssigv4, etc for the SDK to automatically plug-in the specified authenticator.

How can we make the interface signature of the authenticator agnostic to the exporter's http client request type?

• Across languages, the SpanExporter implementations use different http clients to create the export request. For example, Python uses requests while Java uses OkHttp or native http client depending on user's configuration. For the authenticator to be called with the request as parameter, it needs to define a generic type.
• Let's say the new type is called OTLPExportRequest. The SpanExporters across the SDKs must first create this instance with the http method, url, body, etc. and pass it to the authenticator. Then the exporter must use the parts from this OTLPExportRequest along with the headers from the authenticator to construct the final export request.

[austinlparker]

A couple of notes/questions -

1. Is this something that has to be at the SDK level, or could it be implemented solely in the Collector?
2. Wouldn't this need to be for all signals? Wouldn't it make more sense to have this be a part of the OTLP exporter itself, rather than as an extension to the SpanExporter interface?

Thanks.

[jack-berg]

This should be a callable that receives the HTTP request data (headers, body, URL, etc.) and returns a dict of key-value pair(s) of authentication header back to the calling exporter.

In opentelemetry-java, we incrementally write the request body to the output stream. There's no place in the code where we have a full copy of the request bytes along with the other request data you discuss (headers, URL, etc).

[jack-berg]

Is this something that has to be at the SDK level, or could it be implemented solely in the Collector?

Its already is implemented in the collector. https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/sigv4authextension