Duplicated messages with spring.kafka.streams.properties.processing.guarantee: exactly_once_v2

[oleksii-suprun]

Hi All.

Recently I noticed suspicious behavior in my Kafka Streams application. My application reads messages from multiple input topics, extracts some fields from the input message and pushes results to a single output topic. My application configuration is the following:

spring:
  application:
    name: my-app-kstreams
  main:
    banner-mode: off
  kafka:
    bootstrap-servers:
      - XXX.XXX.XXX.XXX:9092
    consumer:
      properties:
        spring.json.trusted.packages: com.example.*
    producer:
      properties:
        spring.json.add.type.headers: false
    streams:
      application-id: ${spring.application.name}
      properties:
        topology.optimization: all
        processing.guarantee: exactly_once_v2
        compression.type: gzip
        auto.offset.reset: latest
        default:
          key.serde: org.apache.kafka.common.serialization.Serdes$StringSerde
          value.serde: org.apache.kafka.common.serialization.Serdes$StringSerde

But during the QA process, I found that the application produces duplicates in the output topic. I can see the same message with a different offset in the Kafka topic despite the exactly_once_v2 configuration. I've also verified input topics and have not found any duplicates there.

Does anyone have any advice on what is wrong?

[oleksii-suprun]

I forgot not to mention, that I verified that my application uses isolation.level = read_committed. I can see it in the application log:

2023-04-25T12:46:34.144+02:00  INFO 52569 --- [           main] o.a.k.clients.consumer.ConsumerConfig    : ConsumerConfig values: 
	...
	isolation.level = read_committed
        ...

[artembilan]

Doesn't look like this has anything to do with Spring (despite the config an Spring Boot env).
Please, consider to ask your question on StackOverflow for broader community or even better in Apache Kafka mailing list.
Spring for Apache Kafka does not do anything specific with Kafka Streams, so your duplication concern is unlikely comes from Spring by itself.

[oleksii-suprun]

Thanks, @artembilan. Yes, I realize that this is not explicitly related to Spring. This question aimed to ensure my configuration is correct from the Spring perspective. Unfortunately, it was hard to understand from the documentation how to properly provide properties for the streaming app.