Looks like this has two components:

1. Clustering coordination (https://nifi.apache.org/nifi-docs/administration-guide.html#kubernetes-clustering)

This just requires us to set nifi.cluster.leader.election.implementation, and give NiFi permission to manage Lease objects:

---
apiVersion: nifi.stackable.tech/v1alpha1
kind: NifiCluster
metadata:
  name: simple-nifi
spec:
  # [snip]
  nodes:
    configOverrides:
      nifi.properties:
        nifi.cluster.leader.election.implementation: KubernetesLeaderElectionManager
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: nifi-extra
rules:
  # https://nifi.apache.org/nifi-docs/administration-guide.html#kubernetes-clustering
  - apiGroups:
      - coordination.k8s.io
    resources:
      - leases
    verbs:
      - create
      - get
      - update
      # undocumented but required
      - patch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: nifi-extra
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: nifi-extra
subjects:
  - kind: ServiceAccount
    name: simple-nifi-serviceaccount

(Though obviously we should provide proper support, this is just illustrative of the scope of change we'd need to make.)

2. Cluster state storage (https://nifi.apache.org/nifi-docs/administration-guide.html#kubernetes-configmap-cluster-state-provider)

This is configured by the XML file Sönke refers to in the OP. In theory the change scope should be similarly small, but this is indeed hard-coded and would require an operator change.

@soenkeliebau will have a look and try it out. @maltesander is up to also helping out

Release note:

The Stackable Operator for NiFi now supports using Kubernetes for cluster coordination and storage, rather than ZooKeeper. (Requires NiFi 2.x, migrating existing installs is currently not supported.)

Docs: https://github.com/stackabletech/nifi-operator/blob/c6e3b2831a650153d5d3f293fb9e3b9f65439557/docs/modules/nifi/pages/usage_guide/clustering.adoc#backend-kubernetes (also reorganized the page to focus on clustering in general rather than ZooKeeper in specific.)

@nightkr in case you didn't already I think this is a good opportunity to update the demos to use the new feature

@sbernauer Depends, I was thinking about at least the getting started but that still ships 1.x for now.

Created stackabletech/demos#221 for that. The demos are already on 2.4.0, IMHO this is the perfect opportunity to test the feature