ZFS import on boot causes VERIFY3S panic at dsl_dataset_get_remap_deadlist_object() #17876
Description
System information
| Type | Version/Name |
|---|---|
| Distribution Name | NixOS |
| Distribution Version | 25.11 (Xantusia) (Nixpkgs commit: 6a08e6bb) |
| Kernel Version | 6.12.55 #1-NixOS SMP PREEMPT_DYNAMIC |
| Architecture | x86_64 GNU/Linux |
| OpenZFS Version | zfs-2.3.4-1 |
Describe the problem you're observing
ZFS mounting is causing a kernel panic on boot:
Oct 27 23:48:02 akira kernel: VERIFY3S(err, ==, ENOENT) failed (52 == 2)
Oct 27 23:48:02 akira kernel: PANIC at dsl_dataset.c:4912:dsl_dataset_get_remap_deadlist_object()
Oct 27 23:48:02 akira kernel: Showing stack for process 1041
Oct 27 23:48:02 akira kernel: CPU: 2 UID: 0 PID: 1041 Comm: zpool Tainted: P O 6.12.55 #1-NixOS
Oct 27 23:48:02 akira kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE
Oct 27 23:48:02 akira kernel: Hardware name: HP HP t640 Thin Client/8523, BIOS M43 v01.06 04/22/2020
Oct 27 23:48:02 akira kernel: Call Trace:
Oct 27 23:48:02 akira kernel: <TASK>
Oct 27 23:48:02 akira kernel: dump_stack_lvl+0x5d/0x80
Oct 27 23:48:02 akira kernel: spl_panic+0xf4/0x10b [spl]
Oct 27 23:48:02 akira kernel: ? srso_return_thunk+0x5/0x5f
Oct 27 23:48:02 akira kernel: ? zap_lookup+0x57/0x120 [zfs]
Oct 27 23:48:02 akira kernel: dsl_dataset_get_remap_deadlist_object+0xd5/0xe0 [zfs]
Oct 27 23:48:02 akira kernel: dsl_dataset_hold_obj+0x872/0xa70 [zfs]
Oct 27 23:48:02 akira kernel: traverse_pool+0x13b/0x1e0 [zfs]
Oct 27 23:48:02 akira kernel: ? __pfx_spa_load_verify_cb+0x10/0x10 [zfs]
Oct 27 23:48:02 akira kernel: spa_load+0xe52/0x19c0 [zfs]
Oct 27 23:48:02 akira kernel: spa_load_best+0x54/0x2c0 [zfs]
Oct 27 23:48:02 akira kernel: spa_import+0x22d/0x6d0 [zfs]
Oct 27 23:48:02 akira kernel: zfs_ioc_pool_import+0x14c/0x170 [zfs]
Oct 27 23:48:02 akira kernel: zfsdev_ioctl_common+0x7be/0x980 [zfs]
Oct 27 23:48:02 akira kernel: ? srso_return_thunk+0x5/0x5f
Oct 27 23:48:02 akira kernel: ? srso_return_thunk+0x5/0x5f
Oct 27 23:48:02 akira kernel: zfsdev_ioctl+0x53/0xe0 [zfs]
Oct 27 23:48:02 akira kernel: __x64_sys_ioctl+0x94/0xd0
Oct 27 23:48:02 akira kernel: do_syscall_64+0xb7/0x200
Oct 27 23:48:02 akira kernel: entry_SYSCALL_64_after_hwframe+0x77/0x7f
Oct 27 23:48:02 akira kernel: RIP: 0033:0x7f4fa791e11d
Oct 27 23:48:02 akira kernel: Code: 04 25 28 00 00 00 48 89 45 c8 31 c0 48 8d 45 10 c7 45 b0 10 00 00 00 48 89 45 b8 48 8d 45 d0 48 89 45 c0 b8 10 00 00 00 0f 05 <89> c2 3d 00 f0 ff ff 77 22 48 8b 45 c8 64 48 2b 04 25 28 00 00 00
Oct 27 23:48:02 akira kernel: RSP: 002b:00007ffd9a6bb4a0 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
Oct 27 23:48:02 akira kernel: RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f4fa791e11d
Oct 27 23:48:02 akira kernel: RDX: 00007ffd9a6bc660 RSI: 0000000000005a02 RDI: 0000000000000003
Oct 27 23:48:02 akira kernel: RBP: 00007ffd9a6bb4f0 R08: 0000000000000000 R09: 0000000000000000
Oct 27 23:48:02 akira kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 000055ed7e1884e0
Oct 27 23:48:02 akira kernel: R13: 00007ffd9a6bc660 R14: 000055ed7e18f0f0 R15: 000055ed7e1ce1e0
Oct 27 23:48:02 akira kernel: </TASK>
systemd status for the mounting service:
● zfs-import-zroot.service - Import ZFS pool "zroot"
Loaded: loaded (/etc/systemd/system/zfs-import-zroot.service; enabled; preset: ignored)
Active: activating (start) since Mon 2025-10-27 23:48:00 PDT; 7min ago
Job: 450
Invocation: c5c67f70a9ea40f88af0793696d46402
Main PID: 832 (zfs-import-zroo)
IP: 0B in, 0B out
IO: 33.5M read, 0B written
Tasks: 2 (limit: 6857)
Memory: 4.6M (peak: 8.1M)
CPU: 200ms
CGroup: /system.slice/zfs-import-zroot.service
├─ 832 /nix/store/ciarnmsx8lvsrmdbjddpmx0pqjrm8imb-bash-5.3p3/bin/bash /nix/store/23g9dhl672l8swc52gwi3rmimbgyw80n-unit-script-zfs-import-zroot-start/bin/zfs-import-zroot-start
└─1041 /nix/store/kzbg6pynrmja6p6fl25j9f6cw1rg75zk-zfs-user-2.3.4/sbin/zpool import -d /dev/disk/by-id -N zroot
Describe how to reproduce the problem
I don't have any reproducing steps here. This just happened out of nowhere after a reboot (maybe during a drive scrub?). Looking at the systemd output, it's freezing at zpool import.
Since this unit is hanging and it's a core dependency of my booting system, the only way that I was able to boot it was by adding the systemd.mask=zfs-mount.service kernel parameter via bootloader.