Skip to content

Nexuses can duel when updating the RackNetworkConfig in the bootstore #10855

Description

@jgallagher

The sync_switch_configuration background task is responsible for ensuring that the contents of the replicated bootstore contain an up-to-date SystemNetworkingConfig, which itself is comprised of a RackNetworkConfig (containing all the information required for upstream connectivity, including static routes / BGP / BFD configs) and a BlueprintExternalNetworkingConfig (containing all the information required to set up NAT entries for critical control plane services like boundary NTP). The high-level process here is "load the latest config from CRDB, then check whether it's newer than what's in the bootstore, and if so, update the bootstore". However, the RackNetworkConfig portion of this config has no way to know what "newer" means. If two Nexus instances are executing this background task at the same time as an operator changes the network config, this ordering is possible:

  1. Nexus A loads the current network config from CRDB (call this config C1)
  2. Operator makes a change, producing config C2
  3. Nexus B loads the current network config from CRDB (now C2)
  4. Nexus B checks the bootstore, finds C1, and updates the bootstore to hold C2
  5. Nexus A checks the bootstore, finds C2, and updates the bootstore to hold C1

This will correct itself the next time the bg task runs, so is eventually consistent, but we shouldn't allow the visible "flapping" produced by step 5: Nexus A should have a way of knowing that C2 is newer than the C1 it loaded and therefore it shouldn't update the bootstore.

When I initially added the BlueprintExternalNetworkingConfig it had the same problem, and as part of addressing it (#10320, which also describes this issue in less detail), we added a generation number from the blueprint to the bootstore config, specifically so Nexus can check that generation against the generation of the blueprint it loaded, and skip updating the bootstore if it finds itself in the blueprint version of step 5 above:

let is_blueprint_out_of_date = if desired_gen < current_gen {
warn!(
log, "our loaded blueprint generation is out of date";
"bootstore-gen" => current_gen,
"our-blueprint-gen" => desired_gen,
);
DesiredConfigOutOfDate::Yes
} else {
DesiredConfigOutOfDate::No
};

The network config apparatus as a whole needs something akin to this.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions