feat: fall back to contact points on control reconnect (DRIVER-201)

Default fallback-to-original-contact-points to true: once a control
reconnection round has exhausted the live nodes, the retained,
still-unresolved contact points are appended, so Netty resolves their
names again and a cluster that moved to new addresses is found (#215).
Every metadata node, the control node included, otherwise holds an
address resolved once. Monitors that re-resolve on their own opt out
through TopologyMonitor.reresolvesNodeAddresses(), unless the live-node
plan is empty. HeartbeatIT turns the fallback off, since it counts every
init OPTIONS as a heartbeat; MockResolverIT gains the recovery case.

Fixes #215

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Author: nikagra

core/src/main/java/com/datastax/oss/driver/api/core/config/DefaultDriverOption.java

@@ -701,8 +701,21 @@ public enum DefaultDriverOption implements DriverOption {
   CONTROL_CONNECTION_AGREEMENT_WARN("advanced.control-connection.schema-agreement.warn-on-failure"),
 
   /**
-   * Whether to forcibly add original contact points held by MetadataManager to the reconnection
-   * plan, in case there is no live nodes available according to LBP. Experimental.
+   * Whether to append the original contact points held by MetadataManager to the control
+   * connection's reconnection plan, after the live nodes reported by the load balancing policy.
+   * Defaults to {@code true}.
+   *
+   * <p>This is the driver's DNS re-resolution path. A metadata node holds an address resolved once
+   * and never re-resolved, the node the control connection reached included; a contact point given
+   * as a hostname is kept unresolved (the default) and looked up again through Netty's resolver on
+   * each connect, so once the live nodes are exhausted the contact points find a cluster that moved
+   * to new addresses, as soon as the JVM's DNS cache ({@code networkaddress.cache.ttl}) has
+   * expired. A resolved contact point ({@code advanced.resolve-contact-points = true}, or a
+   * programmatic resolved address) is appended as it is and never re-resolved.
+   *
+   * <p>Skipped when the topology monitor re-resolves node addresses on its own ({@code
+   * TopologyMonitor#reresolvesNodeAddresses()}: the Cloud SNI proxy, and client routes with full
+   * route coverage), unless the live-node plan is empty.
    *
    * <p>Value-type: boolean
    */

core/src/main/java/com/datastax/oss/driver/api/core/config/OptionsMap.java

@@ -369,7 +369,7 @@ protected static void fillWithDriverDefaults(OptionsMap map) {
     map.put(TypedDriverOption.CONTROL_CONNECTION_AGREEMENT_INTERVAL, Duration.ofMillis(200));
     map.put(TypedDriverOption.CONTROL_CONNECTION_AGREEMENT_TIMEOUT, Duration.ofSeconds(10));
     map.put(TypedDriverOption.CONTROL_CONNECTION_AGREEMENT_WARN, true);
-    map.put(TypedDriverOption.CONTROL_CONNECTION_RECONNECT_CONTACT_POINTS, false);
+    map.put(TypedDriverOption.CONTROL_CONNECTION_RECONNECT_CONTACT_POINTS, true);
     map.put(TypedDriverOption.PREPARE_ON_ALL_NODES, true);
     map.put(TypedDriverOption.REPREPARE_ENABLED, true);
     map.put(TypedDriverOption.REPREPARE_CHECK_SYSTEM_TABLE, false);

core/src/main/java/com/datastax/oss/driver/api/core/config/TypedDriverOption.java

@@ -600,7 +600,15 @@ public String toString() {
   public static final TypedDriverOption<Boolean> CONTROL_CONNECTION_AGREEMENT_WARN =
       new TypedDriverOption<>(
           DefaultDriverOption.CONTROL_CONNECTION_AGREEMENT_WARN, GenericType.BOOLEAN);
-  /** Whether to forcibly try original contacts if no live nodes are available */
+  /**
+   * Whether to append the original contact points to the control connection's reconnection plan,
+   * after the live nodes reported by the load balancing policy (defaults to {@code true}).
+   *
+   * <p>A contact point given as a hostname is kept unresolved and looked up again on each connect,
+   * so this is how the driver picks up changed DNS records once the live nodes are exhausted. It is
+   * skipped for a topology monitor that re-resolves node addresses on its own, unless the live-node
+   * plan is empty.
+   */
   public static final TypedDriverOption<Boolean> CONTROL_CONNECTION_RECONNECT_CONTACT_POINTS =
       new TypedDriverOption<>(
           DefaultDriverOption.CONTROL_CONNECTION_RECONNECT_CONTACT_POINTS, GenericType.BOOLEAN);

core/src/main/java/com/datastax/oss/driver/internal/core/control/ControlConnection.java

@@ -61,7 +61,6 @@
 import java.util.List;
 import java.util.Map;
 import java.util.Map.Entry;
-import java.util.Objects;
 import java.util.Queue;
 import java.util.WeakHashMap;
 import java.util.concurrent.CompletableFuture;
@@ -694,12 +693,16 @@ private boolean isControlNode(Node eventNode) {
           && eventNode.getHostId().equals(state.current.getHostId())) {
         return true;
       }
-      if (state.current == null
-          && state.pending != null
-          && Objects.equals(eventNode.getEndPoint(), state.pending.getEndPoint())) {
-        return true;
-      }
-      return false;
+      // Reference identity, not endpoint equality: the reconnection plan now appends the
+      // unresolved contact-point nodes (fallback-to-original-contact-points defaults to true), so a
+      // mixed resolved/unresolved pair is routine here, and DefaultEndPoint.equals resolves the
+      // unresolved side of one -- a blocking lookup on this admin executor, during the very DNS
+      // outage the fallback exists for. Nodes carry identity semantics (no equals override, and
+      // lastNodeState/lastNodeDistance already key on the instance) and connect() stores the plan's
+      // own node as pending, so an event about that node carries the same object. A mixed pair that
+      // is not the same object is a contact point we have not identified yet; the resolve step is
+      // about to replace it with the registered node, after which the hostId branch above answers.
+      return state.current == null && state.pending != null && eventNode == state.pending;
     }
 
     private void onDistanceEvent(DistanceEvent event) {

core/src/main/java/com/datastax/oss/driver/internal/core/metadata/ClientRoutesTopologyMonitor.java

@@ -21,6 +21,7 @@
 import com.datastax.oss.driver.api.core.config.ClientRoutesConfig;
 import com.datastax.oss.driver.api.core.config.DefaultDriverOption;
 import com.datastax.oss.driver.api.core.metadata.EndPoint;
+import com.datastax.oss.driver.api.core.metadata.Node;
 import com.datastax.oss.driver.internal.core.adminrequest.AdminRequestHandler;
 import com.datastax.oss.driver.internal.core.adminrequest.AdminResult;
 import com.datastax.oss.driver.internal.core.adminrequest.AdminRow;
@@ -480,6 +481,27 @@ protected EndPoint buildNodeEndPoint(
     return new ClientRoutesEndPoint(this, hostId, broadcastInetAddress, fallback);
   }
 
+  @Override
+  public boolean reresolvesNodeAddresses() {
+    // A ClientRoutesEndPoint looks its route hostname up on every connect, but only while a route
+    // exists for that host id; without one it falls back to a static, already-resolved address. So
+    // this answers true only while every known node has a live route, and "every" has to mean at
+    // least one: an empty node set (before the first refresh, or after every node was removed) must
+    // not suppress the contact-point fallback at the one moment it is the only way back. A closed
+    // monitor re-resolves nothing at all, however complete its cache still looks.
+    //
+    // A route only reaches DNS if the address column it came from holds a name rather than an IP
+    // literal, and nothing validates that today (scylladb/java-driver#1064). An all-literal
+    // deployment therefore answers true here and suppresses the contact-point fallback; tightening
+    // this waits on that column's contract.
+    if (closed) {
+      return false;
+    }
+    // getNodes() is already keyed by host id, so this is exactly "every known node has a route".
+    Map<UUID, Node> nodes = context.getMetadataManager().getMetadata().getNodes();
+    return !nodes.isEmpty() && resolvedRoutesCache.get().keySet().containsAll(nodes.keySet());
+  }
+
   /**
    * Builds the CQL query to fetch client routes.
    *

core/src/main/java/com/datastax/oss/driver/internal/core/metadata/CloudTopologyMonitor.java

@@ -44,4 +44,12 @@ protected EndPoint buildNodeEndPoint(
     UUID hostId = Objects.requireNonNull(row.getUuid("host_id"));
     return new SniEndPoint(cloudProxyAddress, hostId.toString());
   }
+
+  @Override
+  public boolean reresolvesNodeAddresses() {
+    // Every node is reached through the SNI proxy, whose hostname SniEndPoint#resolve() looks up on
+    // every call, so addresses stay current on their own: appending the contact points as a DNS
+    // fallback would add nothing, and could resurrect nodes this monitor has removed.
+    return true;
+  }
 }

core/src/main/java/com/datastax/oss/driver/internal/core/metadata/LoadBalancingPolicyWrapper.java

@@ -193,11 +193,19 @@ public Queue<Node> newControlReconnectionQueryPlan() {
     // is an immutable QueryPlan (add()/addAll() throw), so concatenate rather than mutate. The
     // nodes retained by MetadataManager are appended, not fresh copies: their identity stays stable
     // across reconnection rounds, and no throwaway node is minted per round.
+    //
+    // A monitor that re-resolves node addresses itself (the Cloud SNI proxy, client routes with
+    // full coverage) keeps them fresh without this fallback, and appending raw contact points could
+    // resurrect nodes it has removed -- unless the live-node plan is empty, in which case there is
+    // nothing else to try. isEmpty() on a policy's plan is size() == 0 through AbstractCollection;
+    // QueryPlan's contract names this call, since it makes "size() never throws" load-bearing.
     if (state == State.RUNNING
         && context
             .getConfig()
             .getDefaultProfile()
-            .getBoolean(DefaultDriverOption.CONTROL_CONNECTION_RECONNECT_CONTACT_POINTS)) {
+            .getBoolean(DefaultDriverOption.CONTROL_CONNECTION_RECONNECT_CONTACT_POINTS)
+        && (!context.getTopologyMonitor().reresolvesNodeAddresses()
+            || regularQueryPlan.isEmpty())) {
       Object[] contactNodes = context.getMetadataManager().getContactPoints().toArray();
       ArrayUtils.shuffleHead(contactNodes, contactNodes.length);
       return new CompositeQueryPlan(regularQueryPlan, new SimpleQueryPlan(contactNodes));

core/src/main/java/com/datastax/oss/driver/internal/core/metadata/MetadataManager.java

@@ -188,6 +188,11 @@ public boolean wasImplicitContactPoint() {
    * they are never added to metadata and never exposed to user-facing APIs (events, {@link
    * com.datastax.oss.driver.api.core.metadata.Metadata#getNodes()}, or {@link
    * com.datastax.oss.driver.api.core.metadata.NodeStateListener} callbacks).
+   *
+   * <p>The metadata node stores {@code nodeInfo.getEndPoint()} as-is and never re-resolves it. A
+   * contact-point hostname is re-read only through the control connection's reconnection fallback
+   * ({@code advanced.control-connection.reconnection.fallback-to-original-contact-points}), which
+   * re-offers the retained, still-unresolved contact points to Netty's resolver on each connect.
    */
   public CompletionStage<Node> registerNode(NodeInfo nodeInfo) {
     Preconditions.checkNotNull(nodeInfo.getHostId(), "Cannot register node without hostId");

core/src/main/java/com/datastax/oss/driver/internal/core/metadata/TopologyMonitor.java

@@ -141,4 +141,24 @@ public interface TopologyMonitor extends AsyncAutoCloseable {
    * {@link DefaultTopologyMonitor}) should override this method.
    */
   default void resetColumnCaches() {}
+
+  /**
+   * Whether this monitor re-resolves node addresses on every connection attempt (for example by
+   * handing out a proxy hostname to be looked up at connect time), rather than registering an
+   * address resolved once.
+   *
+   * <p>When {@code true}, the control connection's reconnection plan does not append the original
+   * contact points as a DNS re-resolution fallback (see {@code
+   * advanced.control-connection.reconnection.fallback-to-original-contact-points}) unless the
+   * live-node plan is empty: the monitor keeps addresses fresh on its own, and appending raw
+   * contact points could resurrect nodes it has removed.
+   *
+   * <p>The default is {@code false}, which is right for {@link DefaultTopologyMonitor}: peers hold
+   * a resolved address from the peers table, and the node the control connection reached is
+   * registered under the address it reached, so neither re-reads DNS. Proxy-based monitors override
+   * this.
+   */
+  default boolean reresolvesNodeAddresses() {
+    return false;
+  }
 }

core/src/main/java/com/datastax/oss/driver/internal/core/util/collection/QueryPlan.java

@@ -39,8 +39,10 @@
  * methods throw.
  *
  * <p>Both {@link #size()} and {@link #iterator()} are supported and never throw, even if called
- * concurrently. These methods are implemented for reporting purposes only, the driver itself does
- * not use them.
+ * concurrently. They exist mainly for reporting, and the request path does not use them; the one
+ * driver caller is {@code LoadBalancingPolicyWrapper#newControlReconnectionQueryPlan}, which asks
+ * {@code isEmpty()} (that is, {@code size() == 0}) of the plan a policy returned, so a custom
+ * implementation that throws from {@code size()} breaks control-connection reconnection.
  *
  * <p>All built-in {@link QueryPlan} implementations can be safely reused for custom load balancing
  * policies; if you plan to do so, study the source code of {@link