Feature Request: add per-tenant ha-tracker configuration

[mattmendick]

Is your feature request related to a problem? Please describe.
Currently the ha-tracker functionality in the distributors only allows configuring timeouts globally. For tenants that have different scrape intervals or rule evaluation intervals, it would be desirable to have the ability to specify per tenant overrides for these timeouts so that a tenant could have a 1m scrape interval and still be able to use the ha-tracker functionality with longer timeouts, while other tenants would like the default shorter timeouts.

Describe the solution you'd like
Preferably I'd like to be able to configure ha_tracker_update_timeout, ha_tracker_update_timeout_jitter_max, and ha_tracker_failover_timeout in the limits_config or some other way of specifying these values to override the defaults, for individual tenants.

Describe alternatives you've considered
Enforcing a minimum scrape interval is an alternative, but is not desired as tenants may have good reasons for having a longer scrape interval than others. Having the tenant remote_write some amount of dummy data at a shorter interval is also something that was suggested, but it seems that being able to specify different values for ha-tracker timeouts per tenant would be a more straightforward solution, however I realize this may not be as easy as it might sound to me.

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.

[pracucci]

Still valid.

[bboreham]

Preferably I'd like to be able to configure ha_tracker_update_timeout, [...] in the limits_config

Nitpick: AFAIK that structure is called "overrides" because it doesn't just apply to limits.

I don't see why this would be complex, perhaps a bit tedious to edit all the required files.