The sync_switch_configuration background task is responsible for ensuring that the contents of the replicated bootstore contain an up-to-date SystemNetworkingConfig, which itself is comprised of a RackNetworkConfig (containing all the information required for upstream connectivity, including static routes / BGP / BFD configs) and a BlueprintExternalNetworkingConfig (containing all the information required to set up NAT entries for critical control plane services like boundary NTP). The high-level process here is "load the latest config from CRDB, then check whether it's newer than what's in the bootstore, and if so, update the bootstore". However, the RackNetworkConfig portion of this config has no way to know what "newer" means. If two Nexus instances are executing this background task at the same time as an operator changes the network config, this ordering is possible:
- Nexus A loads the current network config from CRDB (call this config C1)
- Operator makes a change, producing config C2
- Nexus B loads the current network config from CRDB (now C2)
- Nexus B checks the bootstore, finds C1, and updates the bootstore to hold C2
- Nexus A checks the bootstore, finds C2, and updates the bootstore to hold C1
This will correct itself the next time the bg task runs, so is eventually consistent, but we shouldn't allow the visible "flapping" produced by step 5: Nexus A should have a way of knowing that C2 is newer than the C1 it loaded and therefore it shouldn't update the bootstore.
When I initially added the BlueprintExternalNetworkingConfig it had the same problem, and as part of addressing it (#10320, which also describes this issue in less detail), we added a generation number from the blueprint to the bootstore config, specifically so Nexus can check that generation against the generation of the blueprint it loaded, and skip updating the bootstore if it finds itself in the blueprint version of step 5 above:
|
let is_blueprint_out_of_date = if desired_gen < current_gen { |
|
warn!( |
|
log, "our loaded blueprint generation is out of date"; |
|
"bootstore-gen" => current_gen, |
|
"our-blueprint-gen" => desired_gen, |
|
); |
|
DesiredConfigOutOfDate::Yes |
|
} else { |
|
DesiredConfigOutOfDate::No |
|
}; |
The network config apparatus as a whole needs something akin to this.
The
sync_switch_configurationbackground task is responsible for ensuring that the contents of the replicated bootstore contain an up-to-dateSystemNetworkingConfig, which itself is comprised of aRackNetworkConfig(containing all the information required for upstream connectivity, including static routes / BGP / BFD configs) and aBlueprintExternalNetworkingConfig(containing all the information required to set up NAT entries for critical control plane services like boundary NTP). The high-level process here is "load the latest config from CRDB, then check whether it's newer than what's in the bootstore, and if so, update the bootstore". However, theRackNetworkConfigportion of this config has no way to know what "newer" means. If two Nexus instances are executing this background task at the same time as an operator changes the network config, this ordering is possible:This will correct itself the next time the bg task runs, so is eventually consistent, but we shouldn't allow the visible "flapping" produced by step 5: Nexus A should have a way of knowing that C2 is newer than the C1 it loaded and therefore it shouldn't update the bootstore.
When I initially added the
BlueprintExternalNetworkingConfigit had the same problem, and as part of addressing it (#10320, which also describes this issue in less detail), we added a generation number from the blueprint to the bootstore config, specifically so Nexus can check that generation against the generation of the blueprint it loaded, and skip updating the bootstore if it finds itself in the blueprint version of step 5 above:omicron/nexus/src/app/background/tasks/sync_switch_configuration.rs
Lines 2123 to 2132 in 8f03979
The network config apparatus as a whole needs something akin to this.