[bootstrap-agent] Refactor for clarity (#3827)

The main goals here are, in roughly decreasing order:

1. Make the startup flow of bootstrap agent more obvious/clear
2. Simplify the handoff from bootstrap agent to sled agent
3. Use fewer tokio `Mutex`es
4. In general, use less shared memory objects and more message passing

The two big changes I made to work towards those:

1. Removed the `bootstrap::Agent` type entirely. The bootstrap server
now spawns a tokio task that handles messages and holds the sled agent
state in a local, non-mutex variable that it is free to mutate as
needed.
2. Removed `HardwareMonitor`. This was the source of a lot of confusion
for me personally, and also made the handoff to sled-agent quite
complex. Now the bootstrap agent is solely responsible for monitoring
for hardware changes. (This is the place where I am most likely to have
introduced issues, hence needing to test this more.)

There are many smaller changes, including a few things that now happen
in a slightly different order (but I believe to be correct either way);
for example, prior to this PR, on a cold boot the sled-agent dropshot
server would start before the bootstrap agent dropshot server. Now, the
bootstrap server always starts first.

Author: jgallagher

Cargo.lock

@@ -73,6 +73,7 @@ tofino.workspace = true
 tokio = { workspace = true, features = [ "full" ] }
 tokio-tungstenite.workspace = true
 toml.workspace = true
+usdt.workspace = true
 uuid.workspace = true
 zeroize.workspace = true
 zone.workspace = true

sled-agent/Cargo.toml

@@ -8,13 +8,10 @@ use camino::Utf8PathBuf;
 use clap::{Parser, Subcommand};
 use omicron_common::cmd::fatal;
 use omicron_common::cmd::CmdError;
+use omicron_sled_agent::bootstrap::server as bootstrap_server;
 use omicron_sled_agent::bootstrap::RssAccessError;
-use omicron_sled_agent::bootstrap::{
-    config::Config as BootstrapConfig, server as bootstrap_server,
-};
 use omicron_sled_agent::rack_setup::config::SetupServiceConfig as RssConfig;
 use omicron_sled_agent::{config::Config as SledConfig, server as sled_server};
-use uuid::Uuid;
 
 #[derive(Subcommand, Debug)]
 enum OpenapiFlavor {
@@ -92,33 +89,16 @@ async fn do_run() -> Result<(), CmdError> {
                 None
             };
 
-            // Derive the bootstrap addresses from the data link's MAC address.
-            let link = config
-                .get_link()
-                .map_err(|e| CmdError::Failure(e.to_string()))?;
-
-            // Configure and run the Bootstrap server.
-            let bootstrap_config = BootstrapConfig {
-                id: Uuid::new_v4(),
-                link,
-                log: config.log.clone(),
-                updates: config.updates.clone(),
-            };
-
-            // TODO: It's a little silly to pass the config this way - namely,
-            // that we construct the bootstrap config from `config`, but then
-            // pass it separately just so the sled agent can ingest it later on.
-            let server =
-                bootstrap_server::Server::start(bootstrap_config, config)
-                    .await
-                    .map_err(CmdError::Failure)?;
+            let server = bootstrap_server::Server::start(config)
+                .await
+                .map_err(|err| CmdError::Failure(format!("{err:#}")))?;
 
             // If requested, automatically supply the RSS configuration.
             //
             // This should remain equivalent to the HTTP request which can
             // be invoked by Wicket.
             if let Some(rss_config) = rss_config {
-                match server.agent().start_rack_initialize(rss_config) {
+                match server.start_rack_initialize(rss_config) {
                     // If the rack has already been initialized, we shouldn't
                     // abandon the server.
                     Ok(_) | Err(RssAccessError::AlreadyInitialized) => {}