[sled-agent] Integrate config-reconciler #8064

jgallagher · 2025-04-29T17:35:10Z

This PR integrates the new sled-agent-config-reconciler crate with sled-agent. It will not currently pass tests due to the reconciler not being completely implemented, but I'd like to get any feedback on this integration work itself (particularly as it pertains to the API of sled-agent-config-reconciler). See the description of #8063 for more context.

There are a couple serious warts with this PR:

The inventory system has not been updated with all the details we need to report for the reconciler. This is a bigger chunk of work because it involves a database migration and touches various bits of Nexus, so I'll do that in a separate PR.
This integration removes most uses of the StorageManager (because its functionality is being absorbed into sled-agent-config-reconciler); however, the storage manager also has a rich set of test support. This PR leaves a couple sled-agent submodules using that test support (support-bundle/storage and zone-bundle). In the long run I think it'd be better to rework these (if there are no remaining production uses of StorageManager), but for now I think this is... okay? Feedback welcome.

This is somewhat extracted from #8064, but can be landed independently and will make some of the followup sled-agent-config-reconciler PRs a little cleaner. Fixes #7774.

smklein · 2025-04-29T20:09:52Z

dev-tools/omdb/src/bin/omdb/sled_agent.rs

@@ -34,14 +34,6 @@ enum SledAgentCommands {
    #[clap(subcommand)]
    Zones(ZoneCommands),

-    /// print information about zpools


Are you expecting that inventory will supplant this info? Or are you planning on replacing this access to the sled agent later?

I was expecting that inventory would supplant this. (I think maybe it already has, in practice? I definitely only look at inventory when I'm curious about zpools; I don't think I've ever used these omdb subcommands.)

smklein · 2025-04-29T20:11:43Z

nexus/reconfigurator/planning/src/system.rs

+        // TODO this inventory needs to be more robust for the reconciler;
+        // this is nontrivial so coming in a subsequent PR. For now just


super-nit, link to PR, so this comment doesn't get lost?

PR isn't open yet, so hard to link. 😅 However, I put a stub in RSS on this branch where we need the updated inventory:

omicron/sled-agent/src/rack_setup/service.rs

Lines 426 to 434 in abd7542

// Wait until the config reconciler on the target sled has successfully

// reconciled the config at `generation`.

async fn wait_for_config_reconciliation_on_sled(

&self,

_sled_address: SocketAddrV6,

_generation: Generation,

) -> Result<(), SetupServiceError> {

unimplemented!("needs updated inventory")

}

which guarantees this PR can't pass the helios/deploy check until the inventory work is done.

smklein · 2025-04-29T20:47:36Z

sled-agent/src/services.rs

-}
-
-#[cfg(all(test, target_os = "illumos"))]
-mod illumos_tests {


We're deleting a lot of tests here - do we really not care about these anymore, or do we have coverage elsewehre?

Little of column a, little of column b:

All the methods these tests are exercising have themselves been deleted, so we're forced to remove these tests too.

The functionality we've removed here will live in the config-reconciler now, but the tests won't port over 1-to-1 because the mechanics are changing (e.g., ledgering is largely independent from starting zones now, which is very different from how ensure_all_omicron_zones_persistent() worked). The reconciler will have its own tests for the new mechanics that cover this same functionality.

andrewjstone · 2025-04-30T17:45:48Z

sled-agent/src/artifact_store.rs

+                // is valid against the current ledgered sled config.
+                if let Some(config_reconciler) = config_reconciler {
+                    config_reconciler
+                        .validate_artifact_config(new_config.clone())


I'm concerned about deadlocks here as pointed out in #8063

andrewjstone · 2025-04-30T17:58:55Z

sled-agent/src/long_running_tasks.rs

+    } else {
+        TimeSyncConfig::Normal
+    };
+    let mut config_reconciler = ConfigReconcilerHandle::new(


Did I miss where we call spawn_reconciliation_task, or is it not here yet?

It can't happen this early in the process; it happens during SledAgent::new() (after we've gotten our config from RSS or the ledger):

omicron/sled-agent/src/sled_agent.rs

Lines 600 to 610 in abd7542

// Start reconciling against our ledgered sled config.

config_reconciler.spawn_reconciliation_task(

etherstub_vnic,

ReconcilerFacilities {

service_manager: services.clone(),

metrics_queue: metrics_manager.request_queue(),

zone_bundler: long_running_task_handles.zone_bundler.clone(),

},

SledAgentArtifactStoreWrapper(Arc::clone(&artifact_store)),

&parent_log,

);

This is somewhat extracted from #8064, but can be landed independently and will make some of the followup sled-agent-config-reconciler PRs a little cleaner. We don't yet ledger `OmicronSledConfig`s to disk, so we're free to fiddle with the details of its fields without worrying about backwards compatibility. Fixes #7774.

jgallagher requested review from andrewjstone, davepacheco and smklein April 29, 2025 17:35

jgallagher added a commit that referenced this pull request Apr 29, 2025

OmicronSledConfig: Just one Generation

7cd6e49

This is somewhat extracted from #8064, but can be landed independently and will make some of the followup sled-agent-config-reconciler PRs a little cleaner. Fixes #7774.

jgallagher mentioned this pull request Apr 29, 2025

OmicronSledConfig: Just one Generation #8067

Merged

smklein reviewed Apr 29, 2025

View reviewed changes

andrewjstone reviewed Apr 30, 2025

View reviewed changes

jgallagher mentioned this pull request Apr 30, 2025

Introduce sled-agent-config-reconciler skeleton #8063

Merged

jgallagher force-pushed the john/sled-agent-config-reconciler-2 branch from abd7542 to 2574c5c Compare April 30, 2025 19:17

Base automatically changed from john/sled-agent-config-reconciler-1 to main May 1, 2025 12:34

jgallagher added 2 commits May 2, 2025 10:59

[sled-agent] Integrate config-reconciler

8deb806

use new ConfigReconcilerSpawnToken interface

a057195

jgallagher force-pushed the john/sled-agent-config-reconciler-2 branch from 2574c5c to a057195 Compare May 2, 2025 14:59

This was referenced May 15, 2025

[sled-agent-config-reconciler] Flesh out main reconicilation task #8162

Merged

[sled-agent] Don't implicitly create datasets when launching zones #8173

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[sled-agent] Integrate config-reconciler #8064

[sled-agent] Integrate config-reconciler #8064

jgallagher commented Apr 29, 2025

smklein Apr 29, 2025

jgallagher Apr 30, 2025

smklein Apr 29, 2025

jgallagher Apr 30, 2025

smklein Apr 29, 2025

jgallagher Apr 30, 2025

andrewjstone Apr 30, 2025

andrewjstone Apr 30, 2025

jgallagher Apr 30, 2025

		// TODO this inventory needs to be more robust for the reconciler;
		// this is nontrivial so coming in a subsequent PR. For now just

	// Wait until the config reconciler on the target sled has successfully
	// reconciled the config at `generation`.
	async fn wait_for_config_reconciliation_on_sled(
	&self,
	_sled_address: SocketAddrV6,
	_generation: Generation,
	) -> Result<(), SetupServiceError> {
	unimplemented!("needs updated inventory")
	}

	// Start reconciling against our ledgered sled config.
	config_reconciler.spawn_reconciliation_task(
	etherstub_vnic,
	ReconcilerFacilities {
	service_manager: services.clone(),
	metrics_queue: metrics_manager.request_queue(),
	zone_bundler: long_running_task_handles.zone_bundler.clone(),
	},
	SledAgentArtifactStoreWrapper(Arc::clone(&artifact_store)),
	&parent_log,
	);

[sled-agent] Integrate config-reconciler #8064

Are you sure you want to change the base?

[sled-agent] Integrate config-reconciler #8064

Conversation

jgallagher commented Apr 29, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment