Expose postings cardinality stats from prometheus tsdb.Head

[damnever]

Is your feature request related to a problem? Please describe.
Users may want to do some improvements based on the cardinality stats.

Describe the solution you'd like
Add a new API to the distributor/query-frontend, which accepts a label name argument as PostingsCardinalityStats does, the distributor divides aggregated stats by replication factor.

The API should support both HTML and JSON responses (grafana-json-datasource may be useful).

[yeya24]

I love this feature and I think it could be very useful.
Did you propose to expose the TSDB stats API https://prometheus.io/docs/prometheus/latest/querying/api/#tsdb-stats? Or another API to expose cardinality stats?

[damnever]

My initial thought is to expose the TSDB stats API directly, however, the API has a limit of 10 records.

[yeya24]

So you are satisfying with the API except the number of records it returns?
If that's the only limitation, probably we could start a discussion upstream to return more records.

[damnever]

Yes, I think the API is good enough for me.

[damnever]

The cache logic of the API seems a little buggy, so I think we may need to reinvent it.

[damnever]

some thoughts about this:

1. expose prometheus.Head.Postings so that we can do some customization
2. use the existing api to select series then calculate those topk stats

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.

[yeya24]

There is no 10 limit upstream anymore and prometheus/prometheus#12389 there is a proposal to add matchers as well.

It is still a useful feature to have. But some design might be required. e.g. How can we get tenant level cardinality information on ingesters using this API? Maybe calling series API under the hood is easier