Can Spring Kafka Base on record header which consume from Src Kafka then distribute message to different target kafka cluster

[xzhan]

Hi @garyrussell I read you reply on this thread https://stackoverflow.com/questions/53843328/filter-messages-before-deserialization-based-on-headers

here is my project problem

We have upstream Kafka clusters build on AWS region, which generate message with record heard ().

We have down stream Kafka clusters build on our Internal clusters.

Now we want to build on one smart routing service which consumer message from upstream Kafka then routing (produce) message to target Kafka cluster only.

take example : Upstream Kafka generate two message

one message with record header , key: DC value: SJC
second message with record header , key DC value: DFW

our smart routing service consumer two message then base on header to send message to down stream Kafka one by one , one go to down stream service SJC , another send to down stream service DFW

so base on that can spring kafka support that ? can you guide me how to design well on this case , especially how to write code more flexible with different record header rule

[xzhan]

@garyrussell you always help me , can you help me on this problem and how Spring kafka support it with perfect model

[artembilan]

I would say Spring Integration is the best way to go for this type of solutions: https://docs.spring.io/spring-integration/docs/current/reference/html/kafka.html#kafka
And for routing you just simply can use a Router endpoint to decide to which sub-flow to proceed: https://docs.spring.io/spring-integration/docs/current/reference/html/message-routing.html#messaging-routing-chapter.

Closing this as not a issue for the project.
This is more like a discussion: https://github.com/spring-projects/spring-kafka/discussions or StackOverflow question: https://stackoverflow.com

[xzhan]

@artembilan you are right , that's more like spring-kafka discussion. Thanks for you info about the document ,I will read it and try it . if not work for my project ,I will try to init discussion in Spring -Kafka Group - cc @garyrussell