From e8c8f431ce322e261ecdffd7f4b166d6098a04a8 Mon Sep 17 00:00:00 2001 From: Patrik Nordwall Date: Mon, 4 Dec 2023 16:00:33 +0100 Subject: [PATCH] doc: Reference docs for edge Replicated Event Sourcing --- .../src/main/paradox/feature-summary.md | 1 + .../src/main/paradox/feature-summary.md | 48 ++- .../src/main/paradox/images/edge-res.drawio | 352 ++++++++++++++++++ .../src/main/paradox/images/edge-res.svg | 3 + akka-edge-docs/src/main/paradox/overview.md | 2 + .../EdgeReplicationIntegrationSpec.scala | 1 - .../src/main/resources/reference.conf | 3 + .../scaladsl/ReplicationSettings.scala | 2 +- ...rpc-replicated-event-sourcing-transport.md | 40 +- 9 files changed, 431 insertions(+), 21 deletions(-) create mode 100644 akka-edge-docs/src/main/paradox/images/edge-res.drawio create mode 100644 akka-edge-docs/src/main/paradox/images/edge-res.svg diff --git a/akka-distributed-cluster-docs/src/main/paradox/feature-summary.md b/akka-distributed-cluster-docs/src/main/paradox/feature-summary.md index 5fc475444..e7010eb02 100644 --- a/akka-distributed-cluster-docs/src/main/paradox/feature-summary.md +++ b/akka-distributed-cluster-docs/src/main/paradox/feature-summary.md @@ -59,6 +59,7 @@ Replicated Event Sourcing gives: * redundancy to tolerate failures in one location and still be operational * serve requests from a location near the user to provide better responsiveness +* allow updates to an entity from several locations * balance the load over many servers The replicas of the entities are running in separate Akka Clusters for the reasons described in diff --git a/akka-edge-docs/src/main/paradox/feature-summary.md b/akka-edge-docs/src/main/paradox/feature-summary.md index a097bc76b..f8b1b0ee7 100644 --- a/akka-edge-docs/src/main/paradox/feature-summary.md +++ b/akka-edge-docs/src/main/paradox/feature-summary.md @@ -1,6 +1,9 @@ # Feature Summary -The main feature of Akka Edge is Projections over gRPC - asynchronous brokerless service-to-service communication. +Akka Edge has two main features: + +1. Projections over gRPC - asynchronous brokerless service-to-service communication +1. Replicated Event Sourcing over gRPC - active-active entities ## Projections over gRPC @@ -111,16 +114,39 @@ H2 database should not be used when the service is an Akka Cluster with more tha * @extref[Reference documentation of Akka Projection gRPC](akka-projection:grpc.html) * @extref[Reference documentation of Akka Projection gRPC with producer push](akka-projection:grpc-producer-push.html) -## Replicated Event Sourcing is not for Edge +## Replicated Event Sourcing over gRPC + +You would use Replicated Event Sourcing over gRPC for entities that can be updated in more than one geographical +location, such as edge Point-of-Presence (PoP) and different cloud regions. This makes it possible to implement +patterns such as active-active and hot standby. + +![Diagram showing services using Replicated Event Sourcing over gRPC between cloud and edge services](images/edge-res.svg) + +Replicated Event Sourcing gives: + +* redundancy to tolerate failures in one location and still be operational +* serve requests from a location near the user to provide better responsiveness +* allow updates to an entity from several locations +* balance the load over many servers + +The replicas of the entities are running in separate Akka Clusters in the cloud and edge. +A reliable event replication transport over gRPC is used between the Akka Clusters. The replica entities belong +to the same logical Microservice, i.e. same [Bounded Context](https://martinfowler.com/bliki/BoundedContext.html) +in Domain-Driven Design (DDD) terminology. -@extref[Replicated Event Sourcing over gRPC](akka-distributed-cluster:feature-summary.html#replicated-event-sourcing-over-grpc) -is a useful feature in Akka Distributed Cluster, but it is not recommended for edge use cases. The reasons why it is currently -not supported for Akka Edge are: +Note that the connection is established from the edge service. For this you need to setup @extref[Akka Replicated Event Sourcing gRPC with edge topology](akka-projection:grpc-replicated-event-sourcing-transport.html#edge-topology). -* It requires gRPC connectivity in both directions between the replicas. -* The overhead of CRDT metadata may become too large when there are many 100s of replicas, or if the replicas dynamically change over time. +Filters can be used to define that a subset of the entities should be replicated to certain locations. +The filters can be changed at runtime. + +@@@ note +Events are stored in a database for each replica. There is no direct database access between a replica and +the database of another replica, which means different databases, and even different database products, can +be used for the replicas. For example, Postgres in the Cloud and H2 at the edge. +@@@ + +### Learn more -That said, if you can overcome these restrictions it can be a good fit also for edge use cases. You might have -a network topology that allows establishing connections in both directions (e.g. VPN solution) and you might not have -that many edge services. The latter can also be mitigated by strict filters so that not all entities are replicated -everywhere. +* FIXME link to guide +* @extref[Reference documentation of Akka Replicated Event Sourcing](akka:typed/replicated-eventsourcing.html) +* @extref[Reference documentation of Akka Replicated Event Sourcing over gRPC](akka-projection:grpc-replicated-event-sourcing-transport.html) diff --git a/akka-edge-docs/src/main/paradox/images/edge-res.drawio b/akka-edge-docs/src/main/paradox/images/edge-res.drawio new file mode 100644 index 000000000..69dd2e17e --- /dev/null +++ b/akka-edge-docs/src/main/paradox/images/edge-res.drawio @@ -0,0 +1,352 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/akka-edge-docs/src/main/paradox/images/edge-res.svg b/akka-edge-docs/src/main/paradox/images/edge-res.svg new file mode 100644 index 000000000..a7f845953 --- /dev/null +++ b/akka-edge-docs/src/main/paradox/images/edge-res.svg @@ -0,0 +1,3 @@ + + +
Entity A-1
Entity A-1
Entity A-1
Entity A-1
Entity A-2
Entity A-2
gRPC
gRPC
Entity A-3
Entity A-3
Cloud Service
Cloud Service
Edge Service
Edge Service
event journal
event journal
event journal
event journal
Entity A-1
Entity A-1
Entity A-3
Entity A-3
Edge Service
Edge Service
event journal
event journal
Entity A-1
Entity A-1
Entity A-2
Entity A-2
Edge Service
Edge Service
event journal
event journal
Entity A-4
Entity A-4
gRPC
gRPC
Cloud Service
Cloud Service
Entity A-4
Entity A-4
Entity A-1
Entity A-1
Entity A-2
Entity A-2
Entity A-3
Entity A-3
event journal
event journal
Entity A-4
Entity A-4Text is not SVG - cannot display \ No newline at end of file diff --git a/akka-edge-docs/src/main/paradox/overview.md b/akka-edge-docs/src/main/paradox/overview.md index d62f450a6..81723f416 100644 --- a/akka-edge-docs/src/main/paradox/overview.md +++ b/akka-edge-docs/src/main/paradox/overview.md @@ -28,6 +28,8 @@ active-active entities or one-way event replication as provided by @extref[Akka The edge services connect to the cloud services and can use event replication in both directions to communicate with cloud services. An edge service can be a single node or form a small Akka Cluster. +The cloud and edge services may use active-active entities or one-way event replication between cloud and edge. + The edge service can be fully autonomous and continue working when there are network disruptions, or if the edge service chooses to not always be connected. It will catch up on pending events when the communication is established again. diff --git a/akka-projection-grpc-tests/src/it/scala/akka/projection/grpc/replication/EdgeReplicationIntegrationSpec.scala b/akka-projection-grpc-tests/src/it/scala/akka/projection/grpc/replication/EdgeReplicationIntegrationSpec.scala index 1575efb58..ae9cc98b7 100644 --- a/akka-projection-grpc-tests/src/it/scala/akka/projection/grpc/replication/EdgeReplicationIntegrationSpec.scala +++ b/akka-projection-grpc-tests/src/it/scala/akka/projection/grpc/replication/EdgeReplicationIntegrationSpec.scala @@ -199,7 +199,6 @@ class EdgeReplicationIntegrationSpec(testContainerConf: TestContainerConf) 10.seconds, 8, R2dbcReplication()) - .withEdgeReplication(true) } selfReplicaId match { diff --git a/akka-projection-grpc/src/main/resources/reference.conf b/akka-projection-grpc/src/main/resources/reference.conf index ebfd627fd..30b296691 100644 --- a/akka-projection-grpc/src/main/resources/reference.conf +++ b/akka-projection-grpc/src/main/resources/reference.conf @@ -57,6 +57,9 @@ akka.projection.grpc { replication { + # Allow edge replicas to connect and replicate updates + accept-edge-replication = off + # Replicated event sourcing from edge sends each event over sharding, in case that delivery # fails or times out, retry this number of times, with an increasing backoff conntrolled by # the min and max backoff settings diff --git a/akka-projection-grpc/src/main/scala/akka/projection/grpc/replication/scaladsl/ReplicationSettings.scala b/akka-projection-grpc/src/main/scala/akka/projection/grpc/replication/scaladsl/ReplicationSettings.scala index 4b329005c..8f2a5fac4 100644 --- a/akka-projection-grpc/src/main/scala/akka/projection/grpc/replication/scaladsl/ReplicationSettings.scala +++ b/akka-projection-grpc/src/main/scala/akka/projection/grpc/replication/scaladsl/ReplicationSettings.scala @@ -147,7 +147,7 @@ object ReplicationSettings { parallelUpdates = config.getInt("parallel-updates"), projectionProvider = replicationProjectionProvider, eventProducerInterceptor = None, - acceptEdgeReplication = false, + acceptEdgeReplication = replicationConfig.getBoolean("accept-edge-replication"), configureEntity = identity, producerFilter = _ => true, initialConsumerFilter = Vector.empty, diff --git a/docs/src/main/paradox/grpc-replicated-event-sourcing-transport.md b/docs/src/main/paradox/grpc-replicated-event-sourcing-transport.md index 54c48e119..911e391de 100644 --- a/docs/src/main/paradox/grpc-replicated-event-sourcing-transport.md +++ b/docs/src/main/paradox/grpc-replicated-event-sourcing-transport.md @@ -130,7 +130,12 @@ and other connection options as when using Akka gRPC directly. For more details It is also possible to set up @apidoc[akka.projection.grpc.replication.*.ReplicationSettings] through APIs only and not rely on the configuration file at all. -### Binding the publisher +### Fully connected topology + +In a network topology where each replica cluster can connect to each other replica cluster the configuration should +list all replicas and gRPC server must be started in each replica. + +#### Binding the publisher Binding the publisher is a manual step to allow arbitrary customization of the Akka HTTP server and combining the endpoint with other HTTP and gRPC routes. @@ -172,7 +177,28 @@ Scala Java : @@snip [ShoppingCartServer.java](/samples/grpc/shopping-cart-service-java/src/main/resources/grpc.conf) { #http2 } -### Serialization of events +### Edge topology + +In some use cases it is not possible to use a @ref[fully connected topology](#fully-connected-topology), for example because of firewalls or NAT in front of each producer. The consumer may also not know about all producers up front. + +This is typical when using @extref:[Replicated Event Sourcing at the edge](akka-edge:feature-summary.html#replicated-event-sourcing-over-grpc). +where the connection can only be established from the edge service to the cloud service. + +For this purpose, Akka Replicated Event Sourcing gRPC has a mode where the replication streams for both consuming +and producing events are initiated by one side. In this way a star topology can be defined, and it's possible +to combine with replicas that are fully connected. + +You would still define how to connect to other replicas as described above, but it's only needed on the edge side, and +it would typically only define one or a few cloud replicas that it will connect to. A gRPC server is not needed on the +edge side, because there are no incoming connections. + +On the edge side you start with `Replication.grpcEdgeReplication`. + +On the cloud side you would start with `Replication.grpcReplication` as described above, but with the addition +`withEdgeReplication(true)` in the @apidoc[ReplicationSettings] or enable `akka.projection.grpc.replication.accept-edge-replication` +configuration. + +## Serialization of events The events are serialized for being passed over the wire using the same Akka serializer as configured for serializing the events for storage. @@ -192,7 +218,7 @@ By default, events from all Replicated Event Sourced entities are replicated. The same kind of filters as described in @ref:[Akka Projection gRPC Filters](grpc.md#filters) can be used for Replicated Event Sourcing. -The producer defined filter: +The producer filter is defined with `withProducerFilter` or `withProducerFilterTopicExpression` in @apidoc[ReplicationSettings]: Scala : @@snip [ShoppingCart.scala](/samples/replicated/shopping-cart-service-scala/src/main/scala/shopping/cart/ShoppingCart.scala) { #init-producerFilter } @@ -200,18 +226,16 @@ Scala Java : @@snip [ShoppingCart.java](/samples/replicated/shopping-cart-service-java/src/main/java/shopping/cart/ShoppingCart.java) { #init-producerFilter } -Consumer defined filters are updated as described in @ref:[Akka Projection gRPC Consumer defined filter](grpc.md#consumer-defined-filter) +The initial consumer filter is defined with `withInitialConsumerFilter` in @apidoc[ReplicationSettings]. +Consumer defined filters can be updated in runtime as described in @ref:[Akka Projection gRPC Consumer defined filter](grpc.md#consumer-defined-filter) One thing to note is that `streamId` is always the same as the `entityType` when using Replicated Event Sourcing. The entity id based filter criteria must include the replica id as suffix to the entity id, with `|` separator. -Replicated Event Sourcing is bidirectional replication, and therefore you would typically have to define the same -filters on both sides. That is not handled automatically. - ## Sample projects -Source code and build files for complete sample projects can be found in the @extref:[Akka Distributed Cluster Guide](akka-distributed-cluster:guide/3-active-active.html). +Source code and build files for complete sample projects can be found in the @extref:[Akka Distributed Cluster Guide](akka-distributed-cluster:guide/3-active-active.html) and @extref:[Akka Edge Guide](akka-edge:guide.html). ## Security