Question > What are the guarantees in term of consistency between the PostgreSQL event store and the Kafka topic when using the Axon Kafka producer?

[ghilainm]

I have a question for which I couldn't find a clear answer in the documentation:
What are the guarantees in term of consistency between the PostgreSQL event store and the Kafka topic when using the Axon Kafka producer?

I would say that as they occur in the same Unit Of Work it should be atomic. However, I am pretty sure it isn't and when some race conditions occur an event could be publish in PostgreSQL and not in Kafka and the other way around. If this is true, what is the recommended approach to synchronise them? Because I don't see how you can rely on something which is not (eventually) consistent.

Could you please clarify the topic in the documentation?

[ghilainm]

Any news on this topic? I am quite interested.

[smcvb]

@nklmish might you be able to shed some light on this situation?

[nklmish]

Have a look here while sending an event to Kafka, current unit of work is taken into consideration providing we have transactional kafka producer.

[ghilainm]

For me it doesn't guarantee anything. Reading the code just shows that indeed it is in the unit of work, but they are corner cases where you will be able to commit the Kafka transaction and not the Axon event store transaction. This will lead to message in Kafka topic which don't exist in Axon even store.

[nklmish]

For me it doesn't guarantee anything.

I am not sure what sort of guarantees are we referring here, please note that Kafka and Postgre transaction are two distinct transactions, not a single (XA) transaction. Kafka does not support XA transactions

What we are doing is first executing the current unit of work and after its completed we are publishing the event to Kafka. In case event publication fails we are signaling it by throwing EventPublicationFailedException

@abuijze for visibility.

[ghilainm]

I am totally fine with what you said. I am referring to consistency guarantees between the two stores (Kafka and PostgreSQL).

So now consider the case where you can commit to Kafka and then you can not commit to PostgreSQL, there is an inconsistency between the two stores. It is clear that you cannot have immediate consistency but I would like to have eventual consistency when using your Kafka producer.

[nklmish]

So now consider the case where you can commit to Kafka and then you can not commit to PostgreSQL, there is an inconsistency between the two stores

That's why we are executing Kafka work only after the current unit of work is committed, i.e. uow.afterCommit.

If you are referring to two different apps (let say A and B) where App-A is writing transactionally to Kafka, and App-B is consuming from Kafka & committing to its own storage engine; In such a scenario one of the thing to make sure at a design level for App-B is idempotency while consuming events.

Could you please clarify the topic in the documentation?

We will work on improving the documentation and adding relevant examples. Thank you for creating the issue!

[ghilainm]

That's why we are executing Kafka work only after the current unit of work is committed, i.e. uow.afterCommit.

@nklmish But then you may end up with events not committed at all to Kafka and loosing events.

If you are referring to two different apps (let say A and B) where App-A is writing transactionally to Kafka, and App-B is consuming from Kafka & committing to its own storage engine; In such a scenario one of the thing to make sure at a design level for App-B is idempotency while consuming events.

I am creating a read model on my aggregate and I use Kafka to forward events to my read model. I therefore need to have eventual consistency or at least once guarantee of delivery which seems not to be the case with your current design.

[ghilainm]

@nklmish I was wondering if implementing the KafkaPublisher as a TrackingEventProcessor would not be a better solution. It would allow to have at least one guarantee of delivery if the Kafka transaction is committed before the one managing the TrackingToken.

What do you think?

[abuijze]

Hi @ghilainm,
that is indeed a fair option. However, in Axon, we want to keep the Publisher and Processor responsibility strictly separated. Doing so allows users to pick their own guarantees, by configuring it in a way that fits them. Forcing this to be a TrackingProcessor will add to latency and also limit the publication to Kafka to the speed of a sequential write and read. That may not be acceptable for others.

[ghilainm]

@abuijze No problem. But I think that in this case the documentation must be improved in order to explain such 'limitation' in the guarantees offered by the Kafka publisher that you provide. It should also maybe be explained what is the recommended approach depending on the guarantees needed.