Pulsar function replicas auto scaling

[zzzming]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

Pulsar function and connector in Kubernetes runtime should be able to auto scale based on resource usage or the input topic backlog size or some other metrics. The auto scale should be horizontal scaling.

Solution

No response

Alternatives

Use the existing Kubernetes HPA. But the state of replicas won't be saved because function stop removes the function statefulset. The parallelism is save in bookkeeper not Kubernetes (upon statefulset removal) in the current implementation.

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!

[hpvd]

maybe this pulsar function in mesh scaling is interesting
streamnative/function-mesh#457

[michaeljmarshall]

Thanks for the suggestion @hpvd. I think this issue is requesting an improvement to the kubernetes runtime that ships with pulsar.

[hpvd]

jep. You are right and sorry the issue was not even the right target to link (it's a preview how function scaling in function mesh should/will work in flexible way in the near future -> using keda)

-> Wanted to point to a solution how functions could be auto-scaled today (with additional tool: function-mesh)

With Kubernetes Horizontal Pod Autoscaler (HPA), Function Mesh supports automatically scaling the number of Pods (Pulsar instances) that are required to run Pulsar functions, sources, and sinks.

Function Mesh auto-scales the number of Pods based on the CPU usage, memory usage, and custom metrics.

source:
https://functionmesh.io/docs/scaling

[github-actions]

The issue had no activity for 30 days, mark with Stale label.