serviceMonitor not sending correct password from secret using basicAuth

[rjackson-fr]

I have defined a serviceMonitor for an application where the scrape endpoint requires basic authentication. It constantly fails with a 401 error. I have confirmed the username and password values in my secret are correct and that the serviceaccount can get the secret. I can shell into the metric pod and use curl curl http://10.244.1.xxx:8080/metrics/prometheus --user prometheus:prometheus and successfully retrieve the metric data

I deployed a similar test deployment/service/serviceMonitor using image mendhak/http-https-echo in order to see the raw value of the authorization header, and it appears that it is sending the text <secret> as the password:

cHJvbWV0aGV1czo8c2VjcmV0Pg== = prometheus:<secret>

{ "path": "/test", "headers": { "host": "10.244.2.xxx:8080", "user-agent": "custom-collector-distro/0.99.0", "accept": "application/openmetrics-text;version=1.0.0;q=0.5,application/openmetrics-text;version=0.0.1;q=0.4,text/plain;version=0.0.4;q=0.3,*/*;q=0.2", "accept-encoding": "gzip", "authorization": "Basic cHJvbWV0aGV1czo8c2VjcmV0Pg==", "x-prometheus-scrape-timeout-seconds": "10" }, "method": "GET", "body": "", "fresh": false, "hostname": "10.244.2.xxx", "ip": "::ffff:10.244.0.xxx", "ips": ], "protocol": "http", "query": {}, "subdomains": ], "xhr": false, "os": { "hostname": "nginx-deployment-xxxxxxxx-xxxx" }, "connection": {} }

[github-actions]

This issue is stale because it has been open 7 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[rjackson-fr]

FYI: I have confirmed this theory by changing the password on the applications to <secret> and the serviceMonitor collector is now working

[github-actions]

This issue is stale because it has been open 7 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions]

This issue was closed because it has been stalled for 12 days with no activity.

[rjackson-fr]

Can this be re-opened? It is not fixed

[vishiy]

hi @rjackson-fr - just reopened it to track and fix this issue .

[rjackson-fr]

Thank you!

[rjackson-fr]

I see that this unit test is expecting the static string in a test case, but not sure where the actual config is parsed at runtime.

https://github.com/Azure/prometheus-collector/blob/main/otelcollector/otel-allocator/server/server_test.go#L538

[rjackson-fr]

This looks like it could just be the placeholder string in the API is not getting replaced correctly?

https://prometheus.io/docs/prometheus/latest/configuration/configuration/#endpoint

https://prometheus.io/docs/prometheus/latest/configuration/configuration/#http_config

https://prometheus.io/docs/prometheus/latest/configuration/configuration/#configuration-file

[github-actions]

This issue is stale because it has been open 7 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions]

This issue is stale because it has been open 7 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[rjackson-fr]

bump

[gracewehner]

Hi @rjackson-fr, thanks for your issue. For the password to be sent from the otel-allocator to the ama-metrics replica pods, it needs to be sent over HTTPS. We have plans to implement this in the future. The workaround is to use the ama-metrics-prometheus-config configmap with a Prometheus scrape config instead of a service monitor for now.

Here's some of the docs to do so:

• Basic auth with CRs
• Basic auth with Prom Config
• Prom config configmap sample
• Prom config configmap creation

[rjackson-fr]

@gracewehner thanks - that is interesting. I already have a workaround mentioned above, so I will stick with the service monitor for now.

Is that a design choice based on sending the password over non-TLS connection between the monitoring components? Ultimately the password will also be sent to the scraping endpoint on the monitored service pod, and these are often non-TLS behind the ingress anyway. Since the ama-metrics pods are also kubernetes, does it make more sense to have them fetch the secret directly rather than have it fetched and passed from the otel-allocator?

-Rob

[github-actions]

This issue is stale because it has been open 7 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions]

This issue is stale because it has been open 7 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions]

This issue was closed because it has been stalled for 12 days with no activity.