Span Link Directive in context propagation header to make further spans make span link to parent instead of being part of same trace.

[jha-hitesh]

What am i trying to achieve?

• I have a distributed system involving 100+ micro services, in which there are flows where a single request to the system leads to requests (both sync[api] and async[kafka]) to 10+ micro services, which makes the trace very long.

• This is still okay but the problem multiplies manifold when one of the service sends multiple kafka requests for a single requests it received(eg: triggered a full sync of some dataset), here the trace becomes extremely large.

• I wanted to break the traces into small viewable traces.

What have I tried?

• fortunately Opentelemetry has provided a solution in the form of span links using which we can link multiple traces casually so that they are still relatable.

• But if i try to use that in this case it presents a few problems.

• So assume this is the request FLOW. (Service A -> Service B -> Service C -> Service D)

• now when request reaches Service C , service C emits a lot of kafka messages and each of these are then consumed by Service D which has 20+ spans for each consumed messages, resulting in an extremely large trace.

• Naturally to solve this what i would want is that when Service D consumes the message it should create a separate trace instead of using the service C last span id as parent span id, but at the same time i want to preserve some relation so i would want to add the Service C Last span context as span link.

• Also Service D will have no idea that for which request it needs to create new trace and for which it needs to reuse same trace id as the sole decision depends upon Service C cause if its a multiple kafka message flow Service C would want the Service D consumption flows to be separate trace for a single message flow it could want the Service D spans to be in same trace.

• So ideally Service C needs to pass some directive in the propogation header (possibly in trace flags section of w3c traceparent header) to inform downstream system that due to the large number of messages i am producing this trace is going to be long and the downstream service should instead create new trace with span links to me to keep relationship alive and reduce large trace possibility.

Above is what i am thinking will solve the large trace problem .

• We use automatic instrumentation a lot so naturally even if we want to create a separate trace_id for a request. we can't cause current automatic instrumentation don't have support for creating new traces automatically based on some directive from remote parent span. so we are stuck with extremely large traces which our backed (Grafana Tempo) unable to handle.

Proposal

• If there was a support to pass this directive from caller service into it's propagation context and the similar support in child service to respect this directive it would solve a lot of problem for trace management, it would make traces size configurable.

• This support will also have to be added to all the instrumentation packages of the supported langugages.

• I guess most of Non OTEL based Vendors have this based on my experience with newrelic, There is a possibility that this could be simply an optimization from trace storage backend engine where they are only showing a subset of trace on the UI.

• Regardless, the current implementation of span links can also be tweaked to make traces smaller without depending upon underlying storage backend.

• I am open to suggestions on how to solve this in Opentelemetry Only, not relying on the storage backend( as that can change in future thanks to vendor neutral API, SDK of Opentelemetry)

Additional context.

• Tech Stack Involved in the setup: kubernetes, python, javascript, java, apache kafka, google pubsub, Temporal workers,

At last Apologies if this issue is not well framed, I will take all the feedbacks on the same.
Thank you!

_{Tip: React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}