Add proxy and user_agent labels to user login counter metric

[vapopov]

In this PR, added a user agent to the client tools request header. During login, it is passed through the proxy to the auth service. Additionally, information about the public proxy address is included so it can be recorded in metrics.

The user_login_total counter now has two labels: proxy and user_agent.

• The proxy label allows us to track which regions are used by customers during login.
• The user_agent label helps identify the client tools version in use, to detect any affected versions in future, and monitor the progress of client tools managed updates.

Example:

# HELP user_login_total Number of times there was a user login
# TYPE user_login_total counter
user_login_total{proxy="proxy1:8443",user_agent="api/19.0.0-dev"} 1
user_login_total{proxy="proxy1:8443",user_agent="web/19.0.0-dev"} 2
user_login_total{proxy="proxy2:8443",user_agent="api/18.2.5"} 1
user_login_total{proxy="proxy2:8443",user_agent="api/19.0.0-dev"} 2
user_login_total{proxy="proxy2:8443",user_agent="web/19.0.0-dev"} 3

Related:

• https://github.com/gravitational/cloud/issues/14225
• Add default headers for the client roundtrip#32

[tigrato]

How can we ensure that an attacker doesn't break cloud monitoring by spamming tons of agents and causing Prometheus to fail? Are we going to drop them in cloud?

[rosstimothy]

Agree with Tiago, this seems like it could be a very high cardinality metric. cc @evanfreed.

[rosstimothy]

I see a tag for this but not a GH release. Do you plan on creating a new release?

[vapopov]

You are right, I will draft release for this version

[sclevine]

I think the concern about high cardinality on the user agent is worth addressing for scalability reasons, even if authenticated users could likely already cause trouble by connecting many agent versions (which we already record).

Could we parse the user agent for Teleport major version, and only accept reasonable values?

[vapopov]

How can we ensure that an attacker doesn't break cloud monitoring by spamming tons of agents and causing Prometheus to fail? Are we going to drop them in cloud?

@tigrato in order to do that, you need a user from the cluster and must complete the login process. We would probably need to reset counters periodically as well