Internal requests (http.request_uri) do not have traceparent header

[jpsugar-flow]

Is there an existing issue for this?

• I have searched the existing issues

Kong version ($ kong version)

3.7

Current Behavior

We have a custom plugin using resty request_uri and although we do see internal_request spans, they are not connected to the server span that results.

Expected Behavior

Tracing headers are injected into the outgoing request.

Steps To Reproduce

No response

Anything else?

No response

[chobits]

We have a custom plugin using resty request_uri and although we do see internal_request spans
You mean your custom plugin sends an internal request with the URL request_uri?

they are not connected to the server span that results.
And what does this mean, does it mean these two span (main request span and internal request spans) are not associated with each other?

Could you show us more details of these spans like some screen shot for these two spans info.

[jpsugar-flow]

And what does this mean, does it mean these two span (main request span and internal request spans) are not associated with each other?

Sorry, here is a diagram of what we expect to see:

• request [kind=CLIENT] (from calling service)
  • GET / (Kong) [kind=SERVER] (from kong)
    • kong.internal.request [kind=CLIENT] (from kong)
      • GET /auth [kind=SERVER] (from auth service) <-- This one is detached
    • kong.balancer [kind=CLIENT] (from kong)
      • GET /abc/def [kind=SERVER] (from receiving service)

We see everything above except the GET /auth span because the internal request is uninstrumented and does not have trace headers attached to it.

[github-actions]

This issue is marked as stale because it has been open for 14 days with no activity.

[jpsugar-flow]

Ping for not-stale.

[spacewander]

Looks like we need to propagate in

kong/kong/observability/tracing/instrumentation.lua

Lines 195 to 200 in 7d5b21f

	local span = tracer.start_span("kong.internal.request", {
	span_kind = 3, -- client
	attributes = attributes,
	})
	local res, err = request_uri(self, uri, params)

Maybe with this method:

kong/kong/observability/tracing/propagation/init.lua

Line 196 in 7d5b21f

local function propagate(propagation_conf, get_inject_ctx_cb, ...)

Note that the inject method

kong/kong/observability/tracing/propagation/init.lua

Line 154 in 7d5b21f

local formatted_trace_id, injection_err = injector:inject(inject_ctx_instance)

is designed for proxy:

kong/kong/observability/tracing/propagation/injectors/_base.lua

Lines 163 to 170 in 7d5b21f

	local headers, h_err = self:create_headers(inj_tracing_ctx)
	if not headers then
	return nil, h_err
	end
	for h_name, h_value in pairs(headers) do
	set_header(h_name, h_value)
	end

We need to tweak it for http.request_uri.

@jpsugar-flow
PR is welcome!

[jpsugar-flow]

PR is welcome!

I know, but I didn't even know where to start. Thanks @spacewander for your notes, I'll give it another try.

[jpsugar-flow]

Hm, I don't have much experience with lua. It's clear that _INJECTOR:inject needs to be generalized to other ways to set headers on a request, but I don't know how to do that. Can lua take a function as an argument?

[spacewander]

Can lua take a function as an argument?

Yes

[github-actions]

This issue is marked as stale because it has been open for 14 days with no activity.

[jpsugar-flow]

I think this one is beyond my capacity right now. :(