From 18b8373dcea1d88d46a554ecfbb73c9efebf72b0 Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Thu, 19 Jun 2025 16:52:07 +0100 Subject: [PATCH 1/2] DOC-5438 added info about at-least-once delivery and checkpointing to architecture page --- .../redis-data-integration/architecture.md | 25 ++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-) diff --git a/content/integrate/redis-data-integration/architecture.md b/content/integrate/redis-data-integration/architecture.md index 84e50c32ec..65a96af9d7 100644 --- a/content/integrate/redis-data-integration/architecture.md +++ b/content/integrate/redis-data-integration/architecture.md @@ -42,7 +42,8 @@ in sequence: currently uses an open source collector called [Debezium](https://debezium.io/) for this step. -1. The collector records the captured changes using Redis streams +1. The collector records the captured changes using +[Redis streams]({{< relref "/develop/data-types/streams" >}}) in the RDI database. 1. A *stream processor* reads data from the streams and applies @@ -68,6 +69,28 @@ RDI automatically enters a second phase called *change streaming*, where changes in the data are captured as they happen. Changes are usually added to the target within a few seconds after capture. +## At-least-once delivery guarantee + +RDI guarantees *at-least-once delivery* to the target. This means that +a given change will never be lost, but it might be added to the target +more than once. Apart from a slight performance overhead, adding a +change multiple times is harmless because the multiple writes +are [*idempotent*](https://en.wikipedia.org/wiki/Idempotence) (that is +to say that all writes after the first one make no change to the +overall state). + +## Checkpointing + +RDI uses Redis streams to store the sequence of change events +captured from the source. The events are then retrieved in order +from the streams, processed, and written to the target. The stream +processor uses a *checkpoint* mechanism to keep track of the last +event in the sequence that it has successfully processed and stored. If the processor fails +for any reason, it can restart from the last checkpoint and +re-process any events that might not have been written to the target. +This ensures that all changes are eventually recorded, even in the +face of failures. + ## Backpressure mechanism Sometimes, data records can get added to the streams faster than RDI can From 9eb437984bea449723a16296cbabc010940e9347 Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Thu, 19 Jun 2025 17:02:15 +0100 Subject: [PATCH 2/2] DOC-5348 fixed section header level --- content/integrate/redis-data-integration/architecture.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/integrate/redis-data-integration/architecture.md b/content/integrate/redis-data-integration/architecture.md index 65a96af9d7..a9e9596aed 100644 --- a/content/integrate/redis-data-integration/architecture.md +++ b/content/integrate/redis-data-integration/architecture.md @@ -108,7 +108,7 @@ an error, just an informative message to note that RDI has applied the backpressure mechanism. {{}} -### Supported sources +## Supported sources RDI supports the following database sources using [Debezium Server](https://debezium.io/documentation/reference/stable/operations/debezium-server.html) connectors: