Docs: Where is the setting Grafana Mimir ingester in Grafana Mimir distributor?

[yangzw1024]

I'm use helm to deploy grafana mimir, when deployed, I'm cannot find any settings of ingester in the distributor, where is it ?
I' m think the doc is too simple, Should the doc provide a demo configuration with the microservices mode?

[osg-grafana]

cc @krajorama

[wdonne]

The distributor complains about not finding enough healthy ingester instances, but the ones it lists as unhealthy are IP-addresses that don't belong to ingesters, but to other pods instead. Probably the ingester hostname must be set, but that is not explained anywhere.

[krajorama]

The distributor will discover ingesters using the ingester hash ring (see https://grafana.com/docs/mimir/latest/references/architecture/components/distributor/#sharding-and-replication and the further linked docs) so there is no need to set the hostname for ingesters.

By default memberlist is used to handle the hash ring and that's seeded from here

mimir/operations/helm/charts/mimir-distributed/values.yaml

Line 355 in 95791e4

- dns+{{ include "mimir.fullname" . }}-gossip-ring.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain }}:{{ include "mimir.memberlistBindPort" . }}

. Which is the dns for a kube service that should keep track of the right ip addresses: https://github.com/grafana/mimir/blob/main/operations/helm/charts/mimir-distributed/templates/gossip-ring/gossip-ring-svc.yaml .

I would guess that the DNS resolution has some issue on your set up, but that's just a guess without seeing the config or knowing more about the env.

The memberlist ring has a feature to be able to verify if a peer (IP address) belongs to the same ring, see cluster_label https://grafana.com/docs/mimir/latest/configure/configuration-parameters/#memberlist . But it's not enabled by default in our helm chart I think.

cc @narqo

[wdonne]

I took a while, but it resolved itself in the end.