test failed in CI: test_disk_move_between_instances #9203
Description
This test failed on a CI run on #9197:
https://github.com/oxidecomputer/omicron/pull/9197/checks?check_run_id=52494170053
Log showing the specific test failure:
https://buildomat.eng.oxide.computer/wg/0/details/01K7877FD921YHG7MRNBWWFJ9F/IwF5mrQGUnDV623PGprZasBFHEUUt6nIuqNJxN3bEHyP7idk/01K7877SEBB5ZT52166EKXPQY9#S7698
Excerpt from the log showing the failure:
FAIL [ 360.193s] omicron-nexus::test_all integration_tests::disks::test_disk_move_between_instances
stdout ───
running 1 test
sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(6573047), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(6573047), kind: System }, version: Some(RouterVersion { router_id: a8a83e2b-1598-4353-b0b5-4bdfbcf0cc3a, version: 3 }), routes: {ResolvedVpcRoute { dest: V6(Ipv6Net { addr: fd21:6126:483a::, width: 64 }), target: VpcSubnet(V6(Ipv6Net { addr: fd21:6126:483a::, width: 64 })) }, ResolvedVpcRoute { dest: V6(Ipv6Net { addr: ::, width: 0 }), target: InternetGateway(Instance(e1ae1569-b19a-40e5-af8f-a3e5a1f1382d)) }, ResolvedVpcRoute { dest: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), target: VpcSubnet(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, ResolvedVpcRoute { dest: V4(Ipv4Net { addr: 0.0.0.0, width: 0 }), target: InternetGateway(Instance(e1ae1569-b19a-40e5-af8f-a3e5a1f1382d)) }} }
sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(6573047), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(6573047), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(6573047), kind: System }, version: Some(RouterVersion { router_id: a8a83e2b-1598-4353-b0b5-4bdfbcf0cc3a, version: 4 }), routes: {ResolvedVpcRoute { dest: V4(Ipv4Net { addr: 0.0.0.0, width: 0 }), target: InternetGateway(Instance(e1ae1569-b19a-40e5-af8f-a3e5a1f1382d)) }, ResolvedVpcRoute { dest: V6(Ipv6Net { addr: ::, width: 0 }), target: InternetGateway(Instance(e1ae1569-b19a-40e5-af8f-a3e5a1f1382d)) }, ResolvedVpcRoute { dest: V4(Ipv4Net { addr: 172.30.0.0, width: 22 }), target: VpcSubnet(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, ResolvedVpcRoute { dest: V6(Ipv6Net { addr: fd21:6126:483a::, width: 64 }), target: VpcSubnet(V6(Ipv6Net { addr: fd21:6126:483a::, width: 64 })) }} }
sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(6573047), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(6573047), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
test integration_tests::disks::test_disk_move_between_instances has been running for over 60 seconds
sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(6573047), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(6573047), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(6573047), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(6573047), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(6573047), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(6573047), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(6573047), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(6573047), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(6573047), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
sled b6d65341-167c-41df-9b5c-41cded99c229 successfully installed routes ResolvedVpcRouteSet { id: RouterId { vni: Vni(6573047), kind: Custom(V4(Ipv4Net { addr: 172.30.0.0, width: 22 })) }, version: None, routes: {} }
test integration_tests::disks::test_disk_move_between_instances ... FAILED
failures:
failures:
integration_tests::disks::test_disk_move_between_instances
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 469 filtered out; finished in 359.77s
stderr ───
log file: /var/tmp/omicron_tmp/test_all-d335379d5fc1140c-test_disk_move_between_instances.23925.0.log
note: configured to log to "/var/tmp/omicron_tmp/test_all-d335379d5fc1140c-test_disk_move_between_instances.23925.0.log"
DB URL: postgresql://root@[::1]:44148/omicron?sslmode=disable
DB address: [::1]:44148
log file: /var/tmp/omicron_tmp/test_all-d335379d5fc1140c-test_disk_move_between_instances.23925.2.log
note: configured to log to "/var/tmp/omicron_tmp/test_all-d335379d5fc1140c-test_disk_move_between_instances.23925.2.log"
thread 'integration_tests::disks::test_disk_move_between_instances' panicked at nexus/tests/integration_tests/instances.rs:7697:19:
instance 9e982e19-f25f-43cb-81f7-17501cc70b91 did not transition to Stopped after 320s: timed out after 321.172888216s
stack backtrace:
0: __rustc::rust_begin_unwind
at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/std/src/panicking.rs:697:5
1: core::panicking::panic_fmt
at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/panicking.rs:75:14
2: {async_fn#0}
at ./tests/integration_tests/instances.rs:7697:19
3: {async_fn#0}
at ./tests/integration_tests/instances.rs:7648:6
4: {async_fn#0}
at ./tests/integration_tests/disks.rs:546:10
5: {async_block#0}
at ./tests/integration_tests/disks.rs:485:1
6: poll<&mut dyn core::future::future::Future<Output=()>>
at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/future/future.rs:124:9
7: poll<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/future/future.rs:124:9
8: {closure#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:742:70
9: with_budget<core::task::poll::Poll<()>, tokio::runtime::scheduler::current_thread::{impl#8}::block_on::{closure#0}::{closure#0}::{closure_env#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>>
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/task/coop/mod.rs:167:5
10: budget<core::task::poll::Poll<()>, tokio::runtime::scheduler::current_thread::{impl#8}::block_on::{closure#0}::{closure#0}::{closure_env#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>>
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/task/coop/mod.rs:133:5
11: {closure#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:742:25
12: tokio::runtime::scheduler::current_thread::Context::enter
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:432:19
13: {closure#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:741:44
14: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:829:68
15: tokio::runtime::context::scoped::Scoped<T>::set
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/context/scoped.rs:40:9
16: tokio::runtime::context::set_scheduler::{{closure}}
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/context.rs:176:38
17: try_with<tokio::runtime::context::Context, tokio::runtime::context::set_scheduler::{closure_env#0}<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>), tokio::runtime::scheduler::current_thread::{impl#8}::enter::{closure_env#0}<tokio::runtime::scheduler::current_thread::{impl#8}::block_on::{closure_env#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>, core::option::Option<()>>>, (alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>)>
at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/std/src/thread/local.rs:315:12
18: std::thread::local::LocalKey<T>::with
at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/std/src/thread/local.rs:279:20
19: tokio::runtime::context::set_scheduler
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/context.rs:176:17
20: tokio::runtime::scheduler::current_thread::CoreGuard::enter
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:829:27
21: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:729:24
22: {closure#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:200:33
23: tokio::runtime::context::runtime::enter_runtime
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/context/runtime.rs:65:16
24: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/scheduler/current_thread/mod.rs:188:9
25: tokio::runtime::runtime::Runtime::block_on_inner
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/runtime.rs:356:52
26: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>
at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.47.0/src/runtime/runtime.rs:330:18
27: test_disk_move_between_instances
at ./tests/integration_tests/disks.rs:485:1
28: test_all::integration_tests::disks::test_disk_move_between_instances::{{closure}}
at ./tests/integration_tests/disks.rs:486:79
29: core::ops::function::FnOnce::call_once
at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/ops/function.rs:250:5
30: core::ops::function::FnOnce::call_once
at /rustc/29483883eed69d5fb4db01964cdf2af4d86e9cb2/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
WARN: dropped CockroachInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
WARN: temporary directory leaked: "/var/tmp/omicron_tmp/.tmpwCWt7k"
If you would like to access the database for debugging, run the following:
# Run the database
cargo xtask db-dev run --no-populate --store-dir "/var/tmp/omicron_tmp/.tmpwCWt7k/data"
# Access the database. Note the port may change if you run multiple databases.
cockroach sql --host=localhost:32221 --insecure
WARN: dropped ClickHouse process without cleaning it up first (there may still be a child process running (PID 23939) and a temporary directory leaked, /var/tmp/omicron_tmp/test_all-d335379d5fc1140c-test_disk_move_between_instances.23925.1-clickhouse-ub6Rg4)
failed to clean up ClickHouse data dir:
- /var/tmp/omicron_tmp/test_all-d335379d5fc1140c-test_disk_move_between_instances.23925.1-clickhouse-ub6Rg4: File exists (os error 17)
WARN: dropped DendriteInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
WARN: dendrite temporary directory leaked: /var/tmp/omicron_tmp/.tmpRPHu72
WARN: dropped MgdInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
WARN: mgd temporary directory leaked: /var/tmp/omicron_tmp/.tmpoefmyY