AWS EKS uses an OIDC provider to assume an IAM Role, but I am encountering a 403 error when trying to connect to OpenSearch.

[ericshiu]

Hi, I’m currently running Vector in a KES pod and have provided a service account to the pod. I’m confident that the service account in the pod can assume the IAM role provided by the OIDC provider. Here’s the evidence:

root@vector-xxx-xxxxxx-wkjvn:/# aws sts get-caller-identity
{
    "UserId": "xxxxx:botocore-session-xxxxxx",
    "Account": "278722xxxxx",
    "Arn": "arn:aws:sts::2787xxxxx:assumed-role/irsa-role-xxxxxg/botocore-session-1xxxxx"
}

However, no matter what I do, I can’t get the OpenSearch permissions to work, and I keep receiving a 403 error.

2024-12-18T11:14:53.351382Z INFO vector::app: Loading configs. paths=["/etc/vector"]
2024-12-18T11:14:53.371771Z INFO vector::topology::running: Running healthchecks.
2024-12-18T11:14:53.371831Z INFO vector: Vector has started. debug="false" version="0.43.1" arch="x86_64" revision="e30bf1f 2024-12-10 16:14:47.175528383"
2024-12-18T11:14:53.371838Z INFO vector::app: API is disabled, enable by setting `api.enabled` to `true` and use commands like `vector top`.
2024-12-18T11:14:53.371839Z INFO vector::topology::builder: Healthcheck disabled.
2024-12-18T11:14:59.203183Z ERROR sink{component_kind="sink" component_id=sink_ilm-test-stg-log-forward component_type=elasticsearch}:request{request_id=1}: vector::sinks::elasticsearch::service: Response contained errors. error_code="http_response_403" response=Response { status: 403, version: HTTP/1.1, headers: {"date": "Wed, 18 Dec 2024 11:14:59 GMT", "content-type": "application/json", "access-control-allow-origin": "*", "connection": "keep-alive"}, body: b"{\"Message\":\"User: anonymous is not authorized to perform: es:ESHttpPost because no resource-based policy allows the es:ESHttpPost action\"}" }
2024-12-18T11:14:59.203223Z ERROR sink{component_kind="sink" component_id=sink_ilm-test-stg-log-forward component_type=elasticsearch}:request{request_id=1}: vector::sinks::util::retries: Not retriable; dropping the request. reason="client-side error, 403 Forbidden: {\"Message\":\"User: anonymous is not authorized to perform: es:ESHttpPost because no resource-based policy allows the es:ESHttpPost action\"}" internal_log_rate_limit=true
2024-12-18T11:14:59.203236Z ERROR sink{component_kind="sink" component_id=sink_ilm-test-stg-log-forward component_type=elasticsearch}:request{request_id=1}: vector_common::internal_event::service: Service call failed. No retries or retries exhausted. error=None request_id=1 error_type="request_failed" stage="sending" internal_log_rate_limit=true
2024-12-18T11:14:59.203246Z ERROR sink{component_kind="sink" component_id=sink_ilm-test-stg-log-forward component_type=elasticsearch}:request{request_id=1}: vector_common::internal_event::component_events_dropped: Events dropped intentional=false count=1 reason="Service call failed. No retries or retries exhausted." internal_log_rate_limit=true

[sinks.sink_{{ .name }}]
    type = "elasticsearch"
    api_version= "v8"
    inputs = ["my_transform"]
    healthcheck = false
    acknowledgements.enabled= true
    bulk.index = "vector-{{ $.Release.Name }}-{{ .name }}-%Y-%m"
    endpoints = [
          xxxxxxxx.xxxx.xxxx
    ]

I tried setting the environment variable VECTOR_LOG="aws_config=debug", but the pod fails to display logs properly. What should I do next to troubleshoot this issue?

https://github.com/vectordotdev/vector/blob/0a2dc2bafa6e56218797a0c238118ed58fd94113/website/cue/reference/components/aws.cue

is not work?

3. In Web IdentityToken credentials from the environment or container (including EKS). These credentials will automatically refresh when expired.

[jszwedko]

I tried setting the environment variable VECTOR_LOG="aws_config=debug", but the pod fails to display logs properly. What should I do next to troubleshoot this issue?

What do you mean by "fails to display logs properly"? That would have been my suggestion to see what role it was authenticating with.

In case it is helpful, the issue here appears to be the authenticated role lacks permissions to do es:ESHttpPost and not that Vector wasn't able to authenticate.