[RORDEV-1861] Failover mode and cluster health check added for remote audit clusters

[mateuszkp96]

- type: new
  components: [es]
  text: "Audit: a remote audit cluster can now run in `failover` mode, and ROR checks the cluster before it starts auditing."

Summary

This is an addition to RORDEV-2190. It gives the remote audit cluster two things: a failover mode, and a connectivity check at startup.

Before this change, a remote audit cluster had only one mode: round-robin. ROR sent audit events to all nodes in turn. If a node was down, the ES RestClient marked it dead and kept it dead with a growing backoff. During that time ROR could drop audit events. ROR also accepted any cluster settings without a check. A typo in a node URL, or nodes from two different clusters, was only found later in the logs.

Failover mode

In failover mode ROR uses one client per node. It sends every audit event to the first node. It moves to the next node only when the current node fails.

• A failure that can be retried (an IOException, or a 502/503/504 response) opens the circuit of that node. ROR then tries the next node.
• A fatal failure (any other response error) stops the request. ROR does not try the other nodes, because the other nodes will answer the same.
• An open circuit keeps a node out of the selection for 1000 * 1.5^(failures-1) milliseconds, up to 30 minutes. This is the same upper limit as the ES RestClient uses.
• A successful request closes the circuit of that node.
• When all circuits are open, ROR tries the node whose circuit opens again first. This keeps the client usable instead of dropping the event.

round-robin stays the default. It is unchanged.

Connectivity check

When ROR creates an audit output for a remote cluster, it now sends GET / to every configured node, in parallel, with three retries. The results decide the outcome:

Result	Outcome
No node answers	Settings are rejected
Nodes answer, but with different cluster_uuid	Settings are rejected
Some nodes answer, all from the same cluster	ROR logs a warning and starts
All nodes answer, all from the same cluster	ROR starts

Two different clusters in one audit output is always an error. One audit output can write to one cluster only.

The new ignore_es_connectivity_problems flag lets you accept an unreachable cluster:

readonlyrest:
  audit:
    enabled: true
    outputs:
      - type: index
        cluster:
          nodes: [ "http://audit1:9200", "http://audit2:9200" ]
          mode: failover                          # or round-robin (default)
          ignore_es_connectivity_problems: false  # default

The flag covers a connectivity problem only. It does not cover a wrong cluster. Note also that a data_stream output must read or create the data stream when it starts, so that output still fails when no node answers.

Tests

• FailoverClientTests — 15 unit tests for the failover client: node selection, the two failure classes, circuit timing, the 30-minute limit, and the all-circuits-open case.
• ReadonlyRestStartingTests — startup with mixed clusters, with all nodes down, and with ignore_es_connectivity_problems on. Three WireMock stubs give two nodes of one cluster and one node of another.
• AuditingConfigTests — decoding of mode: failover and of ignore_es_connectivity_problems.
• RemoteClusterAuditingToolsSuite — rewritten. It now tests both modes against Toxiproxy, and the settings reload with the nodes down. Each test forces a fresh engine, so no circuit state moves between tests.

Scope

• Core: the failover client, the connectivity check, the settings decoder, and the audit output creation.
• All 34 es*x modules: the same four files in each — the failover client build-up, the retryable-response test, and the protected creator methods.

For the reviewer

• IndexBasedAuditOutputServiceCreator.index and DataStreamBasedAuditOutputServiceCreator.dataStream are now protected. Callers use createIndexService / createDataStreamService, which run the check first. This makes it impossible to build an output that skips the check.
• The connectivity check runs on every settings reload, so a reload now costs one HTTP round trip per audit node.
• The docs need a new page section for mode: failover and ignore_es_connectivity_problems. The error message already points to docs.readonlyrest.com/elasticsearch/audit#custom-audit-cluster.

Summary by CodeRabbit

• New Features

  • Added support for remote audit clusters with round-robin and failover behavior.
  • Added a connectivity check for remote audit setups, with an option to continue when problems are ignored.
  • Improved audit output handling for both index-based and data-stream-based configurations.

• Bug Fixes

  • Remote audit connections now handle partial node outages more gracefully.
  • Retry behavior was improved for temporary Elasticsearch gateway errors.