Help needed: proxy-middleware possibly interfering with Datadog trace context

[jorge-gomez-baraibar]

Hello everyone 👋

This might not be an issue with this repo directly, but we’re hoping someone here might have insights.

We’re working on a large Express-based multi-tenant project that uses http-proxy-middleware. Recently, we’ve noticed some strange behavior in our Datadog APM traces.

We see traces with tens of thousands of spans that are completely unrelated to each other. These spans should belong to separate traces, but for some reason, they’re getting grouped into one.

Here's an example of what this looks like in Datadog (10k spans in a single trace):

After reviewing our Datadog integration thoroughly and comparing it with other projects, we’re fairly confident the issue is not on the Datadog side. Our suspicion is that it’s related to how we configure the proxy middleware.

Our proxy configuration:

Here’s how we use createProxyMiddleware for each tenant:

     {
       target: proxyConfig.target,
       changeOrigin: true,
       selfHandleResponse: true,
       pathRewrite: proxyReqPathResolverFactory(proxyConfig.target, logger, hostname),
       timeout: getProxyTimeout(proxyConfig.timeout),
       proxyReq: proxyReqDecorator(hostname, logger),
       proxyRes: proxyResDecorator(proxyConfig, configByHost, serverEnvironment, logger),
    };

This is called for each proxy we need for each tenant we have.

Things we've tried:

• Removing the selfHandleResponseand set it to false to avoid side effects in the proxyRes function (where we add tags to the DD spans). This did not work at all.
• Make sure we are not initializing DD twice.
• Making sure that we are setting tags in spans that are active.

Things that the AI is telling us are happening:

• DD is loosing trace context because of async functions inside of the requests (not sure if this makes a lot of sense, but we tried manually ending responses and traces to no avail)
• DD is initialized twice (we fixed that)

We are using the last major version of this library (^3.0.0).

Has anyone here encountered something similar or have suggestions on how to debug this better? We're open to any ideas.

Also, if this isn't the right place for this discussion, please let me know where better to post it.

Thanks in advance, and have a great day!