[BUG] IllegalArgumentException: value for key [X-Request-Id] already present breaks requests in OpenSearch 3.5.0 #20601
Description
Describe the bug
After upgrading to OpenSearch 3.5.0, the cluster starts and forms normally. However, once requests are sent to the cluster, transport exceptions occur and requests fail.
The following exception appears in the logs:
java.lang.IllegalArgumentException: value for key [X-Request-Id] already present
This appears to be related to the X-Request-Id header propagation introduced in PR #19798. In certain internal flows, the header is added more than once, causing ThreadContext.putSingleHeader() to throw an exception.
This results in failed transport requests and broken client operations.
Symptoms / Impact
- Cluster forms successfully
- Nodes join normally
- As soon as requests hit the cluster:
- Exceptions are logged
- Transport requests fail
- Client requests return errors
Related component
Cluster Manager
To Reproduce
-
Deploy OpenSearch
3.5.0using the official Docker image:opensearchproject/opensearch:3.5.0 -
Run a multi-node cluster (e.g. official Helm chart / Kubernetes StatefulSet).
-
Send requests such as:
GET /_cluster/health
GET /_cat/indices
POST /my-index/_search -
Observe request failures and transport exceptions in logs.
Expected behavior
Duplicate header propagation with identical values should not cause an unhandled exception.
Internal transport communication must not fail because X-Request-Id is already present.
Additional Details
Example Stacktrace
Caused by: java.lang.IllegalArgumentException: value for key [X-Request-Id] already present
at org.opensearch.common.util.concurrent.ThreadContext$ThreadContextStruct.putSingleHeader(...)
at org.opensearch.common.util.concurrent.ThreadContext.putHeader(...)
at org.opensearch.security.transport.SecurityInterceptor.sendRequestDecorate(...)
at org.opensearch.transport.TransportService.sendRequestAsync(...)
...
Workaround
A temporary workaround is to use a patched build that tolerates duplicate insertion if the value is identical.
Example safe logic:
T oldValue = newHeaders.putIfAbsent(key, value);
if (oldValue != null && !oldValue.equals(value)) {
throw new IllegalArgumentException(
"different value for key [" + key + "] already present"
);
}After applying this patch, requests succeed normally.
Environment
- OpenSearch version: 3.5.0
- Deployment method: Official Helm chart (
opensearch/opensearch) - Kubernetes cluster
- No ingress or proxy header injection required to reproduce
- Issue appears to be internal to transport handling
Additional Context
Related change:
PR #19798 — Add X-Request-Id header to uniquely identify a search request
This appears to be a regression introduced in OpenSearch 3.5.0.
Thanks for looking into this!