Stackdriver output fails to start in Google Cloud Run #11323
Description
Bug Report
Describe the bug
When running in Cloud Run, fluent-bit's stackdriver output fails to start, getting an HTTP 403 error from the Metadata Server
To Reproduce
- Example log message if applicable:
DEFAULT 2025-12-30T12:09:25.115193Z [2025/12/30 12:09:25.113626552] [ info] [output:stackdriver:stackdriver.0] metadata_server set to http://169.254.169.254
DEFAULT 2025-12-30T12:09:25.115198Z [2025/12/30 12:09:25.113642722] [ warn] [output:stackdriver:stackdriver.0] client_email is not defined, using a default one
DEFAULT 2025-12-30T12:09:25.115202Z [2025/12/30 12:09:25.113648439] [ warn] [output:stackdriver:stackdriver.0] private_key is not defined, fetching it from metadata server
DEFAULT 2025-12-30T12:09:25.116137Z [2025/12/30 12:09:25.114542548] [ warn] [output:stackdriver:stackdriver.0] error
DEFAULT 2025-12-30T12:09:25.116141Z : Unable to validate host: 169.254.169.254:80
DEFAULT 2025-12-30T12:09:25.116146Z [2025/12/30 12:09:25.114575980] [error] [output:stackdriver:stackdriver.0] can't fetch token from the metadata server
DEFAULT 2025-12-30T12:09:25.116149Z [2025/12/30 12:09:25.114583491] [ warn] [output:stackdriver:stackdriver.0] token retrieval failed
DEFAULT 2025-12-30T12:09:25.116550Z [2025/12/30 12:09:25.114997245] [ warn] [output:stackdriver:stackdriver.0] error
DEFAULT 2025-12-30T12:09:25.116554Z : Unable to validate host: 169.254.169.254:80
DEFAULT 2025-12-30T12:09:25.116559Z [2025/12/30 12:09:25.115025923] [error] [output:stackdriver:stackdriver.0] can't fetch project id from the metadata server
DEFAULT 2025-12-30T12:09:25.116563Z [2025/12/30 12:09:25.115038115] [error] [output] failed to initialize 'stackdriver' plugin
- Steps to reproduce the problem:
Run a Cloud Run Service with a fluent-bit container with the following config
[OUTPUT]
Name stackdriver
Match *
Expected behavior
stackdriver output plugin should be able to read the email and token from the Metadata server
Your Environment
-
Version used:
v4.2.2 / v4.2.3 -
Environment name and version (e.g. Kubernetes? What version?):
Google Cloud Run -
Operating System and version:
Distroless version from docker.io/fluent/fluent-bit:latest
Root Cause
Some experimenting shows the root cause is http client automatically adding the default port to the metadata server url, causing the Cloud Run Metadata Server to reject the request.