Skip to content

Commit 8f35889

Browse files
SungJin1212SungJin1212
committed
Add zone awareness support for ingester
Signed-off-by: SungJin1212 <tjdwls1201@gmail.com>
1 parent 2a2d1d5 commit 8f35889

17 files changed

Lines changed: 647 additions & 5 deletions

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22

33
## master / unreleased
44

5+
* [FEATURE] Support zone awareness on Ingester. #632
56
* [DEPENDENCY] update kiwigrid/k8s-sidecar docker tag to v2.11.2 #663
67

78
## 3.3.8 / 2026-08-17

‎README.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -382,6 +382,9 @@ Kubernetes: `^1.19.0-0`
382382
| ingester.&ZeroWidthSpace;terminationGracePeriodSeconds | int | `240` | |
383383
| ingester.&ZeroWidthSpace;tolerations | list | `[]` | |
384384
| ingester.&ZeroWidthSpace;topologySpreadConstraints | list | `[]` | |
385+
| ingester.&ZeroWidthSpace;zoneAwareness.&ZeroWidthSpace;enabled | bool | `false` | If true, ingesters are deployed as one StatefulSet (or Deployment) per entry in `zones`, each started with `-ingester.availability-zone=<zone name>`, and ingesters, distributors, queriers and rulers get `-distributor.zone-awareness-enabled`. Replicas of a series are spread across zones, so run at least as many zones as the replication factor to survive a zone outage, and the same number of replicas in every zone to keep the load balanced. |
386+
| ingester.&ZeroWidthSpace;zoneAwareness.&ZeroWidthSpace;migration.&ZeroWidthSpace;enabled | bool | `false` | If true, the zone-less ingester workload is kept alongside the zone workloads and `-distributor.zone-awareness-enabled` is set to false on every component. |
387+
| ingester.&ZeroWidthSpace;zoneAwareness.&ZeroWidthSpace;zones | list | `[{"name":"zone-a","nodeSelector":{},"replicas":1},{"name":"zone-b","nodeSelector":{},"replicas":1},{"name":"zone-c","nodeSelector":{},"replicas":1}]` | Zones to deploy ingesters into. `name` is required. `replicas`, `nodeSelector`, `affinity`, `tolerations` and `annotations` fall back to the top-level ingester values when unset; `podLabels` and `podAnnotations` are merged into them. |
385388
| ingress.&ZeroWidthSpace;annotations | object | `{}` | |
386389
| ingress.&ZeroWidthSpace;enabled | bool | `false` | |
387390
| ingress.&ZeroWidthSpace;hosts[0].&ZeroWidthSpace;host | string | `"chart-example.local"` | |

‎ci/test-zone-aware-values.yaml‎

Lines changed: 188 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,188 @@
1+
ingress:
2+
enabled: true
3+
4+
config:
5+
ingester:
6+
lifecycler:
7+
join_after: 30s
8+
final_sleep: 30s
9+
tokens_file_path: "/data/tokens"
10+
ring:
11+
replication_factor: 1
12+
kvstore:
13+
store: "memberlist"
14+
15+
blocks_storage:
16+
backend: "filesystem"
17+
tsdb:
18+
dir: "/data/tsdb"
19+
bucket_store:
20+
sync_dir: "/data/tsdb-sync"
21+
filesystem:
22+
dir: "/data/store"
23+
24+
distributor:
25+
ring:
26+
kvstore:
27+
store: memberlist
28+
29+
ruler:
30+
ring:
31+
kvstore:
32+
store: memberlist
33+
34+
ruler_storage:
35+
backend: "filesystem"
36+
filesystem:
37+
dir: "/data/store"
38+
39+
alertmanager_storage:
40+
backend: "filesystem"
41+
filesystem:
42+
dir: "/data/store"
43+
44+
memberlist:
45+
join_members:
46+
- '{{ include "cortex.fullname" $ }}-memberlist'
47+
48+
ingester:
49+
replicas: 1
50+
statefulSet:
51+
enabled: true
52+
zoneAwareness:
53+
enabled: true
54+
zones:
55+
- name: zone-a
56+
replicas: 1
57+
- name: zone-b
58+
replicas: 1
59+
- name: zone-c
60+
replicas: 1
61+
extraVolumes:
62+
- name: tmp-test
63+
emptyDir: {}
64+
extraVolumeMounts:
65+
- name: tmp-test
66+
mountPath: /tmp-test
67+
compactor:
68+
replicas: 1
69+
extraVolumes:
70+
- name: tmp-test
71+
emptyDir: {}
72+
extraVolumeMounts:
73+
- name: tmp-test
74+
mountPath: /tmp-test
75+
distributor:
76+
replicas: 1
77+
autoscaling:
78+
enabled: true
79+
minReplicas: 1
80+
extraVolumes:
81+
- name: tmp-test
82+
emptyDir: {}
83+
extraVolumeMounts:
84+
- name: tmp-test
85+
mountPath: /tmp-test
86+
querier:
87+
replicas: 1
88+
autoscaling:
89+
enabled: true
90+
extraVolumes:
91+
- name: tmp-test
92+
emptyDir: {}
93+
extraVolumeMounts:
94+
- name: tmp-test
95+
mountPath: /tmp-test
96+
query_frontend:
97+
replicas: 1
98+
extraVolumes:
99+
- name: tmp-test
100+
emptyDir: {}
101+
extraVolumeMounts:
102+
- name: tmp-test
103+
mountPath: /tmp-test
104+
nginx:
105+
replicas: 1
106+
autoscaling:
107+
enabled: true
108+
extraVolumes:
109+
- name: tmp-test
110+
emptyDir: {}
111+
extraVolumeMounts:
112+
- name: tmp-test
113+
mountPath: /tmp-test
114+
config:
115+
httpSnippet: |-
116+
# http snippet
117+
mainSnippet: |-
118+
# main snippet
119+
serverSnippet: |-
120+
# server snippet
121+
runtimeconfigmap:
122+
annotations:
123+
foo: bar
124+
alertmanager:
125+
replicas: 3
126+
statefulSet:
127+
enabled: true
128+
extraVolumes:
129+
- name: tmp-test
130+
emptyDir: {}
131+
extraVolumeMounts:
132+
- name: tmp-test
133+
mountPath: /tmp-test
134+
sidecar:
135+
enabled: true
136+
ruler:
137+
replicas: 1
138+
enabled: true
139+
extraVolumes:
140+
- name: tmp-test
141+
emptyDir: {}
142+
extraVolumeMounts:
143+
- name: tmp-test
144+
mountPath: /tmp-test
145+
sidecar:
146+
enabled: true
147+
store_gateway:
148+
startupProbe: null
149+
replicas: 1
150+
extraVolumes:
151+
- name: tmp-test
152+
emptyDir: {}
153+
extraVolumeMounts:
154+
- name: tmp-test
155+
mountPath: /tmp-test
156+
157+
memcached-frontend:
158+
enabled: true
159+
architecture: standalone
160+
replicaCount: 1
161+
resources:
162+
requests:
163+
cpu: 0
164+
memory: 0
165+
memcached-blocks:
166+
enabled: true
167+
architecture: standalone
168+
replicaCount: 1
169+
resources:
170+
requests:
171+
cpu: 0
172+
memory: 0
173+
memcached-blocks-index:
174+
enabled: true
175+
architecture: standalone
176+
replicaCount: 1
177+
resources:
178+
requests:
179+
cpu: 0
180+
memory: 0
181+
memcached-blocks-metadata:
182+
enabled: true
183+
architecture: standalone
184+
replicaCount: 1
185+
resources:
186+
requests:
187+
cpu: 0
188+
memory: 0

‎templates/distributor/distributor-dep.yaml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,9 @@ spec:
5454
args:
5555
- "-target=distributor"
5656
- "-config.file=/etc/cortex/cortex.yaml"
57+
{{- if .Values.ingester.zoneAwareness.enabled }}
58+
- -distributor.zone-awareness-enabled={{ not .Values.ingester.zoneAwareness.migration.enabled }}
59+
{{- end }}
5760
{{- range $key, $value := .Values.distributor.extraArgs }}
5861
- "-{{ $key }}={{ $value }}"
5962
{{- end }}

‎templates/ingester/ingester-dep.yaml‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
{{- if and (not .Values.ingester.statefulSet.enabled) .Values.ingester.enabled -}}
1+
{{- if and (not .Values.ingester.statefulSet.enabled) .Values.ingester.enabled (or (not .Values.ingester.zoneAwareness.enabled) .Values.ingester.zoneAwareness.migration.enabled) -}}
22
apiVersion: apps/v1
33
kind: Deployment
44
metadata:
@@ -54,6 +54,9 @@ spec:
5454
args:
5555
- "-target=ingester"
5656
- "-config.file=/etc/cortex/cortex.yaml"
57+
{{- if .Values.ingester.zoneAwareness.enabled }}
58+
- "-distributor.zone-awareness-enabled={{ not .Values.ingester.zoneAwareness.migration.enabled }}"
59+
{{- end }}
5760
{{- include "cortex.memcached" . | nindent 12}}
5861
{{- range $key, $value := .Values.ingester.extraArgs }}
5962
- "-{{ $key }}={{ $value }}"

‎templates/ingester/ingester-hpa.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
{{- if and .Values.ingester.enabled .Values.ingester.autoscaling.enabled -}}
1+
{{- if and .Values.ingester.enabled .Values.ingester.autoscaling.enabled (or (not .Values.ingester.zoneAwareness.enabled) .Values.ingester.zoneAwareness.migration.enabled) -}}
22
{{- with .Values.ingester.autoscaling -}}
33
apiVersion: {{ include "cortex.hpaVersion" $ }}
44
kind: HorizontalPodAutoscaler

‎templates/ingester/ingester-poddisruptionbudget.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
{{- if and (gt (int .Values.ingester.replicas) 1) (.Values.ingester.podDisruptionBudget) (.Values.ingester.enabled) }}
1+
{{- if and (gt (int .Values.ingester.replicas) 1) (.Values.ingester.podDisruptionBudget) (.Values.ingester.enabled) (or (not .Values.ingester.zoneAwareness.enabled) .Values.ingester.zoneAwareness.migration.enabled) }}
22
apiVersion: {{ include "cortex.pdbVersion" . }}
33
kind: PodDisruptionBudget
44
metadata:

‎templates/ingester/ingester-statefulset.yaml‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
{{- if and .Values.ingester.statefulSet.enabled .Values.ingester.enabled -}}
1+
{{- if and .Values.ingester.statefulSet.enabled .Values.ingester.enabled (or (not .Values.ingester.zoneAwareness.enabled) .Values.ingester.zoneAwareness.migration.enabled) -}}
22
apiVersion: apps/v1
33
kind: StatefulSet
44
metadata:
@@ -110,6 +110,9 @@ spec:
110110
args:
111111
- "-target=ingester"
112112
- "-config.file=/etc/cortex/cortex.yaml"
113+
{{- if .Values.ingester.zoneAwareness.enabled }}
114+
- "-distributor.zone-awareness-enabled={{ not .Values.ingester.zoneAwareness.migration.enabled }}"
115+
{{- end }}
113116
{{- include "cortex.memcached" . | nindent 12}}
114117
{{- range $key, $value := .Values.ingester.extraArgs }}
115118
- "-{{ $key }}={{ $value }}"

‎templates/ingester/ingester-svc-headless.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
{{- if .Values.ingester.enabled -}}
1+
{{- if and .Values.ingester.enabled (or (not .Values.ingester.zoneAwareness.enabled) .Values.ingester.zoneAwareness.migration.enabled) -}}
22
apiVersion: v1
33
kind: Service
44
metadata:

0 commit comments

Comments
 (0)