Improvements to avoid out-of-order events / lost events in sos-restful

[stefanocke]

Hello Oliver,
at first, thanks a lot for the comprehensive example and for your efforts to bring more support for DDD into Spring (Data).
I have a suggestion for some minor improvements of sos-restful regarding event timestamps: Currently the timestamp is set in the constructor for the event. In case a transaction takes a bit longer than another one, it might happen that an event with an "earlier" timestamp is seen later by other transactions. That is, we get out-of-order events. Even worse, if a consumer polls between the commit of the first and the second transaction, it might miss the out-of-order event at all, since it has already stored the later timestamp in its database and will never query for "earlier" events again.
To solve this problem, we would need to store the "timestamp of the transaction commit" in the event. However... there is no such thing in JPA (or even in SQL) at all.
But I think, with some simple changes, we can get very close to that:

• PersistingEventListener could be @TransactionalEventListener with phase "BEFORE_COMMIT", to get as close as possible to the commit
• PersistingEventListener should perform a flush before persisting the event(s) to make sure most of the SQL work is done before. Again this is to get as close as possible to the commit.
• Instead of a timestamp, a database sequence should be used. JVM time can be different on each node and even database time might not be reliable (see comment below)
• The flush in combination with @TransactionalEventListener will make sure that Hibernate does not rearrange the SQL Statements in a way where the sequence number is retrieved before the insert statement(s) for the aggregate are executed. Instead it will be retrieved as close to the commit as possible. Since the aggreagte is locked in the database by the insert statement, there is no danger that some "faster" transaction will commit an event (with a higher sequence number) for the same aggregate meanwhile.

[mszarlinski]

I think events ordering and clock synchronisation in distributed systems is far harder topic than just generating timestamp on database node. For example, database can be replicated or sharded onto multiple nodes as well. There are already available solutions like vector clocks, but isn't it out of a scope of Spring Data?

[stefanocke]

I am aware that I don't achieve something like 'global ordering' by the proposed improvements. But when using the sequence number instead of timestamp and by getting the sequence number after the insert statement for the aggregate (enforced by flush), I think it is at least guaranteed that the events for one aggregate are in the right order.

[stefanocke]

By the way, we seem to agree that sos-restful has some flaws regarding event timestamps. So what is the solution you propose to make it work for 'real life'?

[stefanocke]

I changed the description and title to make clear it is not my intention to get reliable timestamps.

[mszarlinski]

We should discuss what kind of consistency guarantees you need in the first place. Spring Data is an abstraction over diverse types of storage, including NoSQL databases as well. We should decide on application characteristic first, then come up with a solution.

[stefanocke]

IMHO, sos-restful is an approach that is based on the transactional guarantees of a relational database. To be more specific: It relies on persisting the aggregate and the domain event within the same transaction. If the database does not support this, the sos-restful approach cannot be used.
For more details on this approach see also "Implementing Domain-Driven Design" by Vaughn Vernon. And there especially chapter 8, "Messaging Infrastructure Consistency".

To make clear: I am not talking about Spring Data at all (as you seem to assume in your comments). All I wrote is specific to the sos-restful example where JPA and a relational DB are used.