Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion charts/radar-kafka/Chart.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ apiVersion: v2
appVersion: 3.9.0
description: "Apache Kafka for RADAR-base using the Strimzi Operator"
name: radar-kafka
version: 0.3.0
version: 0.5.0
keywords:
- kafka
- queue
Expand Down
13 changes: 9 additions & 4 deletions charts/radar-kafka/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
# radar-kafka
[![Artifact HUB](https://img.shields.io/endpoint?url=https://artifacthub.io/badge/repository/radar-kafka)](https://artifacthub.io/packages/helm/radar-base/radar-kafka)

![Version: 0.3.0](https://img.shields.io/badge/Version-0.3.0-informational?style=flat-square) ![AppVersion: 3.9.0](https://img.shields.io/badge/AppVersion-3.9.0-informational?style=flat-square)
![Version: 0.5.0](https://img.shields.io/badge/Version-0.5.0-informational?style=flat-square) ![AppVersion: 3.9.0](https://img.shields.io/badge/AppVersion-3.9.0-informational?style=flat-square)

Apache Kafka for RADAR-base using the Strimzi Operator

Expand Down Expand Up @@ -60,19 +60,24 @@ Consult the [documentation](https://github.com/lsst-sqre/strimzi-registry-operat
| strimzi-registry-operator | object | `{"clusterName":"radar","operatorNamespace":"default"}` | Values for schema registry operator ref: https://github.com/lsst-sqre/strimzi-registry-operator |
| strimzi-registry-operator.clusterName | string | `"radar"` | Keep in sync with 'fullnameOverride' |
| strimzi-registry-operator.operatorNamespace | string | `"default"` | Keep in sync with namespace used by deployment |
| kafka | object | `{"cruiseControl":{"addRebalanceTemplate":true,"enabled":false,"goals":[],"skipHardGoalCheck":true},"insyncReplicas":2,"metadataVersion":"3.9-IV0","partitions":9,"podSecurityContext":{},"replicas":3,"replicationFactor":3,"resources":{"requests":{"cpu":"100m","memory":"1Gi"}},"securityContext":{},"storage":{"size":"10Gi"}}` | Values for Kafka cluster deployed by Strimzi kafka operator |
| kafka | object | `{"cruiseControl":{"addRebalanceTemplate":true,"enabled":false,"goals":[],"javaOptions":{"Xms":"128m","Xmx":"256m"},"skipHardGoalCheck":true},"insyncReplicas":2,"javaOptions":{"Xms":"512m","Xmx":"1G"},"metadataVersion":"3.9-IV0","nodePool":{"jvmOptions":{"Xms":"512m","Xmx":"1G"},"resources":{"limits":{"cpu":1,"memory":"2Gi"},"requests":{"cpu":"100m","memory":"1Gi"}}},"partitions":9,"podSecurityContext":{},"replicas":3,"replicationFactor":3,"resources":{"limits":{"cpu":1,"memory":"2Gi"},"requests":{"cpu":"100m","memory":"1Gi"}},"securityContext":{},"storage":{"size":"10Gi"},"topicOperator":{"jvmOptions":{"Xms":"64m","Xmx":"128m"}},"userOperator":{"jvmOptions":{"Xms":"64m","Xmx":"128m"}}}` | Values for Kafka cluster deployed by Strimzi kafka operator |
| kafka.replicas | int | `3` | Number of Kafka brokers |
| kafka.replicationFactor | int | `3` | Number of replicas for Kafka topics |
| kafka.insyncReplicas | int | `2` | Number of in-sync kafka broker replicas |
| kafka.partitions | int | `9` | Number of topic data partitions. Rule of thumb: 3 times the number of brokers. Headroom is used for future upscale of brokers. ref: https://learn.conduktor.io/kafka/kafka-topics-choosing-the-replication-factor-and-partitions-count/ |
| kafka.metadataVersion | string | `"3.9-IV0"` | Metadata API version. Keep in sync with appVersion |
| kafka.storage | object | `{"size":"10Gi"}` | Storage size for Kafka pods |
| kafka.resources | object | `{"requests":{"cpu":"100m","memory":"1Gi"}}` | Resource requests for Kafka pods |
| kafka.cruiseControl | object | `{"addRebalanceTemplate":true,"enabled":false,"goals":[],"skipHardGoalCheck":true}` | Config for the Cruise Control rebalancer (https://github.com/linkedin/cruise-control) |
| kafka.resources.requests | object | `{"cpu":"100m","memory":"1Gi"}` | Resource requests for Kafka pods |
| kafka.resources.limits | object | `{"cpu":1,"memory":"2Gi"}` | Resource limits for Kafka pods |
| kafka.javaOptions | object | `{"Xms":"512m","Xmx":"1G"}` | JVM options for Kafka pods |
| kafka.nodePool.resources.requests | object | `{"cpu":"100m","memory":"1Gi"}` | Resource requests for Kafka pods |
| kafka.nodePool.resources.limits | object | `{"cpu":1,"memory":"2Gi"}` | Resource limits for Kafka pods |
| kafka.cruiseControl | object | `{"addRebalanceTemplate":true,"enabled":false,"goals":[],"javaOptions":{"Xms":"128m","Xmx":"256m"},"skipHardGoalCheck":true}` | Config for the Cruise Control rebalancer (https://github.com/linkedin/cruise-control) |
| kafka.cruiseControl.enabled | bool | `false` | Deploy the Cruise Control rebalancer operator. |
| kafka.cruiseControl.addRebalanceTemplate | bool | `true` | Add a KafkaRebalance template CRD. Note that this does not automatically rebalance the cluster. For rebalancing, you need to create a KafkaRebalance resource using the template. |
| kafka.cruiseControl.goals | list | `[]` | Goals for the Cruise Control rebalancer. If left empty, the default goals are used (see: https://github.com/linkedin/cruise-control#goals). |
| kafka.cruiseControl.skipHardGoalCheck | bool | `true` | Skip hard goal check by Cruise Control rebalancer. |
| kafka.cruiseControl.javaOptions | object | `{"Xms":"128m","Xmx":"256m"}` | JVM options for Cruise Control pods |
| kafka.podSecurityContext | object | `{}` | Security Context for Kafka pods ref: https://strimzi.io/docs/operators/latest/deploying#assembly-security-providers-str |
| kafka.securityContext | object | `{}` | Security Context for Kafka containers ref: https://strimzi.io/docs/operators/latest/deploying#assembly-security-providers-str |
| schema-registry | object | check `values.yaml` | Values for schema registry deployed by strimzi-registry-operator ref: https://github.com/lsst-sqre/strimzi-registry-operator |
24 changes: 22 additions & 2 deletions charts/radar-kafka/templates/kafka.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,13 @@ spec:
default.replication.factor: {{ $topicReplicationFactor }}
min.insync.replicas: {{ $insyncReplicas }}
num.partitions: {{ $topicPartitions }}
{{- if .Values.kafka.javaOptions }}
jvmOptions: {{ .Values.kafka.javaOptions | toYaml | nindent 6 }}
{{- end }}
{{- if .Values.kafka.resources }}
resources:
{{ .Values.kafka.resources | toYaml | indent 4 | trim }}
{{- end }}
listeners:
- name: plain
port: 9092
Expand Down Expand Up @@ -71,10 +78,23 @@ spec:
- mode: remove-brokers
template:
name: {{ template "common.names.fullname" . }}-rebalance-template
{{- if .values.kafka.cruiseControl.javaOptions }}
jvmOptions: {{ .values.kafka.cruiseControl.javaOptions | toYaml | nindent 6 }}
{{- end }}
{{- end }}
kafkaExporter:
{{- toYaml .Values.metrics.kafkaExporter | nindent 4 }}
{{- end }}
entityOperator:
topicOperator: {}
userOperator: {}
topicOperator:
{{- if .Values.kafka.topicOperator.jvmOptions }}
jvmOptions: {{ .Values.kafka.topicOperator.jvmOptions | toYaml | nindent 8 }}
{{- else }}
{}
{{- end }}
userOperator:
{{- if .Values.kafka.userOperator.jvmOptions }}
jvmOptions: {{ .Values.kafka.userOperator.jvmOptions | toYaml | nindent 8 }}
{{- else }}
{}
{{- end }}
7 changes: 5 additions & 2 deletions charts/radar-kafka/templates/nodepool.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,12 @@ spec:
roles:
- controller
- broker
{{- if .Values.kafka.resources }}
{{- if .Values.kafka.nodePool.javaOptions }}
jvmOptions: {{ .Values.kafka.nodePool.javaOptions | toYaml | nindent 6 }}
{{- end }}
{{- if .Values.kafka.nodePool.resources }}
resources:
{{ .Values.kafka.resources | toYaml | indent 4 | trim }}
{{ .Values.kafka.nodePool.resources | toYaml | indent 4 | trim }}
{{- end }}
storage:
type: jbod
Expand Down
39 changes: 38 additions & 1 deletion charts/radar-kafka/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -67,11 +67,44 @@ kafka:
storage:
size: 10Gi

# -- Resource requests for Kafka pods
resources:
# -- Resource requests for Kafka pods
requests:
cpu: 100m
memory: 1Gi
# -- Resource limits for Kafka pods
limits:
cpu: 1
memory: 2Gi

# -- JVM options for Kafka pods
javaOptions:
Xms: 512m
Xmx: 1G

nodePool:
jvmOptions:
Xms: 512m
Xmx: 1G
resources:
# -- Resource requests for Kafka pods
requests:
cpu: 100m
memory: 1Gi
# -- Resource limits for Kafka pods
limits:
cpu: 1
memory: 2Gi

userOperator:
jvmOptions:
Xms: 64m
Xmx: 128m

topicOperator:
jvmOptions:
Xms: 64m
Xmx: 128m

# -- Config for the Cruise Control rebalancer (https://github.com/linkedin/cruise-control)
cruiseControl:
Expand All @@ -85,6 +118,10 @@ kafka:
goals: []
# -- Skip hard goal check by Cruise Control rebalancer.
skipHardGoalCheck: true
# -- JVM options for Cruise Control pods
javaOptions:
Xms: 128m
Xmx: 256m

# -- Security Context for Kafka pods
# ref: https://strimzi.io/docs/operators/latest/deploying#assembly-security-providers-str
Expand Down