Skip to content
This repository has been archived by the owner on Jun 10, 2024. It is now read-only.

Document a dynamically-scalable HA deployment #150

Open
cmgrote opened this issue May 21, 2021 · 0 comments
Open

Document a dynamically-scalable HA deployment #150

cmgrote opened this issue May 21, 2021 · 0 comments
Assignees
Labels
enhancement New feature or request

Comments

@cmgrote
Copy link
Member

cmgrote commented May 21, 2021

  • New connector OMAG pods that can come online (or be dropped) at any time
  • Some quorum mechanism across the pods so that one of the pods can be elected to periodically create an index checkpoint and store in some out-of-cluster location (e.g. S3)
  • Initial index store of each new OMAG pod taken from the latest such external checkpoint (see: https://opencrux.com/reference/21.04-1.16.0/checkpointing.html)
  • A readiness probe that would ideally only succeed once the pod's local index is up-to-date (not sure this would be feasible, as what would indicate it is up-to-date assuming there is always some activity happening via other pods (?))

This will be reliant on having a configuration mechanism for the OMAG platform itself that does not require configuration and / or startup via REST, as otherwise the readiness probe would have to be successful just to configure and startup the platform -- in which case it would already start receiving other traffic via a load-balancing service, all of which would fail prior to the connector being configured and started up (takes at least 20-30 seconds for an empty system, could be several minutes or longer if also bootstrapping its index). Having several minutes of "random" failures for requests that the load-balancer just happens to send to this bootstrapping pod would be unacceptable -- hence dependency on having a non-REST mechanism to start the pods, so readiness probe can indicate that the pod is truly ready to start receiving and (correctly) responding to requests.

@cmgrote cmgrote added the enhancement New feature or request label May 21, 2021
@cmgrote cmgrote self-assigned this May 21, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant