System information
| Type |
Version/Name |
| Distribution Name |
Debian |
| Distribution Version |
13 |
| Kernel Version |
6.12.101+deb13-amd64 |
| Architecture |
x86_64 |
| OpenZFS Version |
zfs-2.4.3-2~bpo13+1 / zfs-kmod-2.4.3-2~bpo13+1 |
Describe the problem you're observing
At ~18.9 hours of uptime, a ZFS write-completion thread (z_wr_int_0) hit the assertion VERIFY3S(remove_reference(hdr, hdr), >, 0) failed (0 > 0), PANIC at arc.c:7026 in arc_write_done(). The thread died via spl_panic().
~71 seconds later, an unrelated userspace process (du walking a directory on the pool) took a NULL pointer dereference (address 0x60) in arc_buf_access() via zfs_readdir → dnode_hold_impl → dbuf_hold_impl. RBX (the hdr pointer) was NULL. I believe this second oops is fallout from the first panic leaving the ARC in an inconsistent state rather than an independent bug, but I'm including both traces.
Pool is a single NVMe vdev (Samsung PM9A3 15.36TB, MZQL215THBLA), no L2ARC, no SLOG. Child datasets are encrypted via raw hex key file. The pool was created under an older release and has not been zpool upgraded; some feature flags are inactive.
A full scrub after reboot completed with 0 errors, so on-disk state appears unaffected — this looks like in-memory ARC refcount corruption.
No sign of ECC errors.
~19 hours ago had the same problem, that's why the uptime is ~19 hours.
Describe how to reproduce the problem
Not reliably reproducible. Occurred once after ~19h uptime under normal load (backup server).
Include any warning/errors/backtraces from the system logs
Aug 19 11:58:37 server kernel: PANIC at arc.c:7026:arc_write_done()
Aug 19 11:58:37 server kernel: Showing stack for process 2225
Aug 19 11:58:37 server kernel: CPU: 70 UID: 0 PID: 2225 Comm: z_wr_int_0 Tainted: P OE 6.12.101+deb13-amd64 #1 Debian 6.12.101-1
Aug 19 11:58:37 server kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Aug 19 11:58:37 server kernel: Call Trace:
Aug 19 11:58:37 server kernel: <TASK>
Aug 19 11:58:37 server kernel: dump_stack_lvl+0x5d/0x80
Aug 19 11:58:37 server kernel: spl_panic+0xf4/0x10b [spl]
Aug 19 11:58:37 server kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Aug 19 11:58:37 server kernel: ? spl_kmem_cache_free+0x133/0x1e0 [spl]
Aug 19 11:58:37 server kernel: arc_write_done+0x43f/0x530 [zfs]
Aug 19 11:58:37 server kernel: zio_done+0x537/0x1050 [zfs]
Aug 19 11:58:37 server kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Aug 19 11:58:37 server kernel: zio_execute+0x8f/0x130 [zfs]
Aug 19 11:58:37 server kernel: taskq_thread+0x352/0x6f0 [spl]
Aug 19 11:58:37 server kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Aug 19 11:58:37 server kernel: ? __pfx_default_wake_function+0x10/0x10
Aug 19 11:58:37 server kernel: ? __pfx_zio_execute+0x10/0x10 [zfs]
Aug 19 11:58:37 server kernel: ? __pfx_taskq_thread+0x10/0x10 [spl]
Aug 19 11:58:37 server kernel: kthread+0xcf/0x100
Aug 19 11:58:37 server kernel: ? __pfx_kthread+0x10/0x10
Aug 19 11:58:37 server kernel: ret_from_fork+0x31/0x50
Aug 19 11:58:37 server kernel: ? __pfx_kthread+0x10/0x10
Aug 19 11:58:37 server kernel: ret_from_fork_asm+0x1a/0x30
Aug 19 11:58:37 server kernel: </TASK>
Aug 19 11:59:48 server kernel: BUG: kernel NULL pointer dereference, address: 0000000000000060
Aug 19 11:59:48 server kernel: #PF: supervisor read access in kernel mode
Aug 19 11:59:48 server kernel: #PF: error_code(0x0000) - not-present page
Aug 19 11:59:48 server kernel: PGD 17d6404067 P4D 0
Aug 19 11:59:48 server kernel: Oops: Oops: 0000 [#1] PREEMPT SMP NOPTI
Aug 19 11:59:48 server kernel: CPU: 6 UID: 0 PID: 346029 Comm: du Tainted: P OE 6.12.101+deb13-amd64 #1 Debian 6.12.101-1
Aug 19 11:59:48 server kernel: Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Aug 19 11:59:48 server kernel: RIP: 0010:arc_buf_access+0x14/0x190 [zfs]
Aug 19 11:59:48 server kernel: Code: e8 a1 46 52 d0 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66 0f 1f 00 0f 1f 44 00 00 41 56 41 55 41 54 55 53 48 8b 1f <48> 81 7b 60 c0 65 42 c1 0f 84 09 01 00 00 48 8b 33 48 85 f6 75 0b
Aug 19 11:59:48 server kernel: RSP: 0018:ff7d7babc265fb70 EFLAGS: 00010282
Aug 19 11:59:48 server kernel: RAX: ff39a1f2cc31ce60 RBX: 0000000000000000 RCX: 0000000000004e69
Aug 19 11:59:48 server kernel: RDX: 0000000000004e69 RSI: ff39a1f07463a000 RDI: ff39a1f42518d240
Aug 19 11:59:48 server kernel: RBP: 0000000000000000 R08: 9ae16a3b2f90408f R09: 9ae16a3b2f90404f
Aug 19 11:59:48 server kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Aug 19 11:59:48 server kernel: R13: ff39a20874f48750 R14: ff7d7babc265fc10 R15: 0000000000000000
Aug 19 11:59:48 server kernel: FS: 00007f19752c2740(0000) GS:ff39a21d8eb00000(0000) knlGS:0000000000000000
Aug 19 11:59:48 server kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 19 11:59:48 server kernel: CR2: 0000000000000060 CR3: 000000037d34a002 CR4: 0000000000f71ef0
Aug 19 11:59:48 server kernel: PKRU: 55555554
Aug 19 11:59:48 server kernel: Call Trace:
Aug 19 11:59:48 server kernel: <TASK>
Aug 19 11:59:48 server kernel: dbuf_hold_impl+0x9f/0x750 [zfs]
Aug 19 11:59:48 server kernel: dbuf_hold+0x31/0x60 [zfs]
Aug 19 11:59:48 server kernel: dnode_hold_impl+0x100/0x1310 [zfs]
Aug 19 11:59:48 server kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Aug 19 11:59:48 server kernel: ? zfs_statvfs+0x1d6/0x5a0 [zfs]
Aug 19 11:59:48 server kernel: zap_lockdir+0x59/0x110 [zfs]
Aug 19 11:59:48 server kernel: zap_cursor_retrieve+0x188/0x370 [zfs]
Aug 19 11:59:48 server kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Aug 19 11:59:48 server kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Aug 19 11:59:48 server kernel: ? filldir64+0x40/0x190
Aug 19 11:59:48 server kernel: zfs_readdir+0x219/0x4b0 [zfs]
Aug 19 11:59:48 server kernel: zpl_iterate+0x53/0x80 [zfs]
Aug 19 11:59:48 server kernel: iterate_dir+0x10e/0x200
Aug 19 11:59:48 server kernel: __x64_sys_getdents64+0x86/0x130
Aug 19 11:59:48 server kernel: ? __pfx_filldir64+0x10/0x10
Aug 19 11:59:48 server kernel: do_syscall_64+0x87/0x1b0
Aug 19 11:59:48 server kernel: ? do_user_addr_fault+0x36c/0x620
Aug 19 11:59:48 server kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Aug 19 11:59:48 server kernel: ? arch_exit_to_user_mode_prepare.isra.0+0x16/0xa0
Aug 19 11:59:48 server kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
Aug 19 11:59:48 server kernel: RIP: 0033:0x7f19753a18e3
Aug 19 11:59:48 server kernel: Code: 8b 05 31 85 10 00 64 c7 00 16 00 00 00 31 c0 eb 9e e8 11 03 04 00 90 b8 ff ff ff 7f 48 39 c2 48 0f 47 d0 b8 d9 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 05 c3 0f 1f 40 00 48 8b 15 f9 84 10 00 f7 d8
Aug 19 11:59:48 server kernel: RSP: 002b:00007ffc8f50de98 EFLAGS: 00000293 ORIG_RAX: 00000000000000d9
Aug 19 11:59:48 server kernel: RAX: ffffffffffffffda RBX: 0000562a63505df0 RCX: 00007f19753a18e3
Aug 19 11:59:48 server kernel: RDX: 0000000000008000 RSI: 0000562a63505e20 RDI: 0000000000000003
Aug 19 11:59:48 server kernel: RBP: 0000562a63505e20 R08: 00000000ffffff9c R09: 0000000000000001
Aug 19 11:59:48 server kernel: R10: 0000000000000000 R11: 0000000000000293 R12: 0000562a63505df4
Aug 19 11:59:48 server kernel: R13: ffffffffffffffa0 R14: 0000000000000000 R15: 0000000000000000
Aug 19 11:59:48 server kernel: </TASK>
Aug 19 11:59:48 server kernel: Modules linked in: xt_connlimit nf_conncount xt_connmark xt_mark macvlan veth nf_conntrack_netlink xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE bridge stp llc xfrm_user xfrm_algo xt_set ip_set nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_addrtype nft_compat wireguard libchacha20poly1305 chacha_x86_64 poly1305_x86_64 curve25519_x86_64 libcurve25519_generic libchacha ip6_udp_tunnel udp_tunnel overlay tun nf_tables libcrc32c bonding tls binfmt_misc nls_ascii nls_cp437 vfat fat ipmi_ssif amd_atl intel_rapl_msr intel_rapl_common amd64_edac edac_mce_amd rndis_host cdc_ether usbnet kvm_amd mii joydev evdev kvm irdma irqbypass zfs(POE) crct10dif_pclmul ghash_clmulni_intel sha512_ssse3 i40e sha256_ssse3 sha1_ssse3 aesni_intel gf128mul crypto_simd ib_uverbs ast cryptd drm_shmem_helper dax_hmem rapl drm_kms_helper ib_core cxl_acpi acpi_cpufreq ccp pcspkr i2c_algo_bit acpi_ipmi spd5118 k10temp ipmi_si ipmi_devintf ipmi_msghandler spl(OE) button hid_generic usbhid hid sch_fq drm tcp_bbr
Aug 19 11:59:48 server kernel: nvme_fabrics nvme_keyring efi_pstore configfs nfnetlink efivarfs ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 crc32c_generic xhci_pci dm_mod xhci_hcd ahci libahci ice libata sp5100_tco nvme crc32_pclmul watchdog usbcore crc32c_intel scsi_mod nvme_core gnss i2c_piix4 libie scsi_common nvme_auth usb_common i2c_smbus
Aug 19 11:59:48 server kernel: CR2: 0000000000000060
Aug 19 11:59:48 server kernel: ---[ end trace 0000000000000000 ]---
System information
Describe the problem you're observing
At ~18.9 hours of uptime, a ZFS write-completion thread (z_wr_int_0) hit the assertion VERIFY3S(remove_reference(hdr, hdr), >, 0) failed (0 > 0), PANIC at arc.c:7026 in arc_write_done(). The thread died via spl_panic().
~71 seconds later, an unrelated userspace process (du walking a directory on the pool) took a NULL pointer dereference (address 0x60) in arc_buf_access() via zfs_readdir → dnode_hold_impl → dbuf_hold_impl. RBX (the hdr pointer) was NULL. I believe this second oops is fallout from the first panic leaving the ARC in an inconsistent state rather than an independent bug, but I'm including both traces.
Pool is a single NVMe vdev (Samsung PM9A3 15.36TB, MZQL215THBLA), no L2ARC, no SLOG. Child datasets are encrypted via raw hex key file. The pool was created under an older release and has not been zpool upgraded; some feature flags are inactive.
A full scrub after reboot completed with 0 errors, so on-disk state appears unaffected — this looks like in-memory ARC refcount corruption.
No sign of ECC errors.
~19 hours ago had the same problem, that's why the uptime is ~19 hours.
Describe how to reproduce the problem
Not reliably reproducible. Occurred once after ~19h uptime under normal load (backup server).
Include any warning/errors/backtraces from the system logs