System information
|
|
| Distribution |
TrueNAS SCALE 25.10.6 (Goldeye); previously 25.04.2.6 (Fangtooth) |
| Kernel |
6.12.99-production+truenas (event 1 was on 6.12.15-production+truenas) |
| OpenZFS |
2.3.4-1 (event 1 occurred on 2.3.0-1) |
| Architecture |
x86_64, Intel Core Ultra 9 285HX (24 threads) |
| Hardware |
Minisforum MS-02-Ultra, BIOS 1.03 (2025-10-22) |
| RAM |
128 GB DDR5-5600 (4 × 32 GB Micron), non-ECC |
| Pool |
tank: 2 × 2-way mirror, 4 × NVMe (2 × Samsung 990 PRO 4TB, 2 × WD_BLACK SN850X 4TB) |
| Compression |
lz4 on all datasets |
| Notable tunables |
THP = never; nvme_core.multipath=N; no ZFS module parameter overrides |
Summary
Four kernel faults over eleven days, all in ZFS code paths, on two different OpenZFS
versions and two different kernels. Two are fatal faults in spa_taskq_dispatch() on the
write path; one is an unrecoverable infinite loop in the LZ4 decompressor on the read path;
the fourth is a fatal fault during ARC buffer teardown (dbuf_evict →
arc_buf_destroy → spl_slab_reclaim → vfree), where the mem_cgroup pointer derived
from a page being freed held the value 0x80. In every case a field that should hold a
small integer, count, or valid pointer instead holds garbage — twice demonstrably the
same two struct fields.
All four were captured over netconsole. Nothing reached disk, because a panicking or
soft-locked kernel cannot sync.
For completeness: this machine also had four earlier unclean resets (2026-07-25 to
2026-08-08) with no log trail at all, before netconsole was in place. We do not claim
they are this bug — they are uncharacterised, and at least one earlier crash series on this
box had a different, separately root-caused mechanism. Only the three captures below are
offered as evidence.
Event 1 — 2026-08-13, OpenZFS 2.3.0-1, kernel 6.12.15 — page fault, write path
Uptime at fault: 432347 s (5d 0h 06m).
BUG: unable to handle page fault for address: 00000000c2c51028
#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
PGD 0 P4D 0
Oops: Oops: 0000 [#1] PREEMPT SMP NOPTI
CPU: 15 UID: 0 PID: 439082 Comm: z_wr_int_0 Tainted: P OE 6.12.15-production+truenas #1
Hardware name: Micro Computer (HK) Tech Limited MS-02-Ultra/ARWSA, BIOS 1.03 10/22/2025
RIP: 0010:spa_taskq_dispatch+0x94/0xb0 [zfs]
Code: 00 4c 89 e6 5d c1 e1 1b 41 5c 41 5d 41 5e e9 c3 3a e0 ff 4c 8b b3 80 01 00 00 e8 47
76 ff ff 8b 8b 78 01 00 00 31 d2 48 f7 f1 <49> 8b 3c d6 eb be 48 8b 83 80 ...
RAX: 0000000000020a7b RBX: ffff98c082a394e0 RCX: 00000000c0aa3570
RDX: 0000000020435bcb RSI: 0000000000000053 RDI: ffffffffbe0303e0
RBP: ffff98de5d74df00 R08: ffff98de5d74df00 R09: 0000000000000000
R10: ffff98c019c153f0 R11: 0000000000550011 R12: ffffffffc0b1a1d0
R13: 0000000000000000 R14: ffffffffc0aa31d0 R15: ffff98de5d74b440
CR2: 00000000c2c51028
Call Trace:
<TASK>
zio_done+0x6e0/0xd90 [zfs]
zio_execute+0x92/0xe0 [zfs]
taskq_thread+0x253/0x4f0 [spl]
kthread+0xcf/0x100
ret_from_fork+0x31/0x50
ret_from_fork_asm+0x1a/0x30
</TASK>
Kernel panic - not syncing: Fatal exception
Event 2 — 2026-08-16, OpenZFS 2.3.4-1, kernel 6.12.99 — divide error, write path
Uptime at fault: 259426 s (3d 0h 04m). Same function, after upgrading 2.3.0 → 2.3.4.
Oops: divide error: 0000 [#1] PREEMPT SMP NOPTI
CPU: 11 UID: 0 PID: 2629239 Comm: z_wr_int_3 Tainted: P OE 6.12.99-production+truenas #1
RIP: 0010:spa_taskq_dispatch+0x91/0xb0 [zfs]
Code: 38 04 00 00 4c 89 e6 5d c1 e1 1b 41 5c 41 5d 41 5e e9 23 5f 9f ff 4c 8b b3 80 01 00
00 e8 d7 78 ff ff 8b 8b 78 01 00 00 31 d2 <48> f7 f1 49 8b 3c d6 eb be 48 ...
RAX: 0000ebf27b907816 RBX: ffff8b8de33c3ba0 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 00000000000000ee RDI: ffffffff97630400
RBP: ffff8b8fb37ee880 R08: ffff8b8fb37ee880 R09: 0000000000000001
R10: 0000000000000003 R11: 00000000000000ee R12: ffffffffc09e8700
R13: 0000000000000001 R14: 0000000000000000 R15: ffff8b8fb37ee880
Call Trace:
<TASK>
zio_vdev_io_start+0xdb/0x290 [zfs]
zio_execute+0x92/0xe0 [zfs]
vdev_queue_io_done+0x9a/0x160 [zfs]
zio_vdev_io_done+0x110/0x1c0 [zfs]
zio_execute+0x92/0xe0 [zfs]
taskq_thread+0x253/0x4f0 [spl]
kthread+0xcf/0x100
ret_from_fork+0x31/0x50
ret_from_fork_asm+0x1a/0x30
</TASK>
Kernel panic - not syncing: Fatal exception
Analysis of events 1 and 2 — the same two struct fields, wrong both times
Decoding the Code: bytes, the faulting sequence is:
mov r14, [rbx+0x180] ; taskq array pointer (believed stqs_taskq)
mov ecx, [rbx+0x178] ; taskq count (believed stqs_count)
xor edx, edx
div rcx ; <-- event 2 faults here (#DE), RCX = 0
mov rdi, [r14+rdx*8] ; <-- event 1 faults here (#PF), bad index
- Event 1 survived the divide and faulted on the index.
RCX = 0xc0aa3570, which is
the low half of a module pointer: R14 = 0xffffffffc0aa31d0, only 0x3a0 away. A
pointer-shaped value sitting in an integer count field.
- Event 2 faulted on the divide itself:
RCX = 0 — and R14 = 0 as well.
⇒ Same struct, same field pair, garbage in both, across two OpenZFS versions and two
kernels, in the write path both times.
Our hypothesis, offered as a lead rather than a finding: a taskq lifecycle race rather
than stray corruption. A count of 0 together with a NULL pointer is the all-zeroes
shape of a struct that has been freed, zeroed, or never populated — not the shape of random
corruption. OpenZFS 2.2+ added dynamically-sized ZIO taskqs; a dispatch for a
(type, priority) whose taskq set is torn down or not yet populated would produce exactly
event 2, and event 1 looks like the same struct caught mid-teardown.
We have not confirmed that +0x178/+0x180 are stqs_count/stqs_taskq in the
shipped zfs.ko — someone with the matching build should verify that before giving our
reading any weight.
Event 3 — 2026-08-19, OpenZFS 2.3.4-1, kernel 6.12.99 — infinite loop, read path
Uptime at fault: 259623 s (3d 0h 07m). Not a panic — one CPU looped forever and the
machine had to be power-cycled after 40 minutes.
watchdog: BUG: soft lockup - CPU#7 stuck for 157s! [IoThread:3375]
CPU: 7 UID: 0 PID: 3375 Comm: IoThread Tainted: P OEL 6.12.99-production+truenas #1
Hardware name: Micro Computer (HK) Tech Limited MS-02-Ultra/ARWSA, BIOS 1.03 10/22/2025
RIP: 0010:LZ4_uncompress_unknownOutputSize+0x5be/0x860 [zfs]
Code: 59 02 00 00 48 83 fa 0f 0f 86 6c 02 00 00 48 8b 55 00 48 83 c3 20 48 83 c5 20 48 89
53 e0 48 8b 55 e8 48 89 53 e8 48 8b 55 f0 <48> 89 53 f0 48 8b 55 f8 48 89 53 f8 48
39 c3 72 d3 e9 b7 fa ff ff
RAX: ffffd0b1b9fff183 RBX: ffffd0b1b9fff00e RCX: ffffd0b1b9ffef6e
RDX: 61622f6e69622f3d RSI: ffffd0b1ba019000 RDI: 0000000000001bda
RBP: ffffd0b1b9ffd434 R08: ffffd0b1b9ff9000 R09: 0000000000000215
R10: ffffd0b1ba018fc0 R11: 000000000000000f R12: ffffd0b1b9fff183
R13: ffff8edf3d523a08 R14: ffffd0b1b9ff9000 R15: ffff8edf3d525630
Call Trace:
<TASK>
zfs_lz4_decompress_buf.constprop.0+0x2b/0x40 [zfs]
zfs_lz4_decompress+0x4d/0x90 [zfs]
? abd_init_struct+0x40/0x50 [zfs]
arc_buf_fill+0x26f/0x4c0 [zfs]
arc_read+0x1502/0x1770 [zfs]
? __pfx_dbuf_read_done+0x10/0x10 [zfs]
dbuf_read_impl.constprop.0+0x27f/0x380 [zfs]
dbuf_read+0x2af/0x5d0 [zfs]
dmu_buf_hold_array_by_dnode+0x118/0x5e0 [zfs]
dmu_read_uio_dnode+0x74/0x150 [zfs]
zfs_read+0x25e/0x530 [zfs]
zpl_iter_read+0xb7/0x120 [zfs]
vfs_read+0x244/0x370
__x64_sys_pread64+0x98/0xd0
do_syscall_64+0x87/0x1a0
entry_SYSCALL_64_after_hwframe+0x76/0x7e
</TASK>
Analysis of event 3
- The lockup was re-reported 11 times over 36 minutes (157 s → 2191 s stuck) and the
register dump is byte-identical in every report — same RIP offset, same RAX,
RBX, RCX. RBX is the loop's advancing cursor (cmp rbx,rax; jb) and it never
advanced. This is a genuinely non-terminating loop, not slow progress.
RDX holds 0x61622f6e69622f3d, i.e. the ASCII bytes =/bin/ab — plain text occupying
a register the decompressor is treating as data.- Userspace caller:
middlewared performing a 4096-byte pread64. The machine hosts no
VMs or containers (virt.instance.query → []); this was ordinary middleware file I/O,
not a heavy workload.
- Collateral:
txg_sync on both pools and a systemd task blocked for 120–483 s; every
process touching the pool hung, including new SSH logins. Writes later resumed; the
looping thread never did, and it could not be signalled — a graceful reboot issued from
the web UI never started, and the box had to be hard power-cycled.
The point of interest: ZFS verifies the block checksum before handing the buffer to
arc_buf_fill() for decompression. So either a buffer that passed its checksum decompressed
as garbage, or arc_buf_fill() was handed the wrong buffer or length. Separately,
LZ4_uncompress_unknownOutputSize() appears able to loop indefinitely on malformed input
rather than returning an error — arguably a hardening bug in its own right, whatever
delivered the bad input.
Event 4 — 2026-08-24, OpenZFS 2.3.4-1, kernel 6.12.99 — NULL-adjacent deref, ARC eviction/free path
Uptime at fault: 33976 s (9h 26m). Fault at 06:36:57 local; auto-reboot 10 s later.
BUG: kernel NULL pointer dereference, address: 00000000000005c8
#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
PGD 0 P4D 0
Oops: Oops: 0000 [#1] PREEMPT SMP NOPTI
CPU: 22 UID: 0 PID: 506 Comm: dbuf_evict Tainted: P OE 6.12.99-production+truenas #1
Hardware name: Micro Computer (HK) Tech Limited MS-02-Ultra/ARWSA, BIOS 1.03 10/22/2025
RIP: 0010:__mod_memcg_state+0x27/0x130
Code: 90 90 90 66 0f 1f 00 0f 1f 44 00 00 0f 1f 44 00 00 48 63 c6 55 0f b6 80 40 ae 26 8a
53 3c ff 0f 84 e3 00 00 00 89 d3 48 89 fd <48> 8b 97 48 05 00 00 48 63 cb 65 48 01
4c c2 18 83 fe 25 0f 84 bb
RSP: 0018:ffffceb64140bd78 EFLAGS: 00010013
RAX: 0000000000000022 RBX: 00000000ffffffff RCX: 0000000000000000
RDX: 00000000ffffffff RSI: 0000000000000032 RDI: 0000000000000080
RBP: 0000000000000080 R08: 0000000000000000 R09: 0000000000000147
R10: 00000000000007e0 R11: 00000000002a0029 R12: fffff0e0efe5f4c0
R13: ffff8b9c632a0490 R14: ffff8b9c632a04a8 R15: ffffffffc02d1340
CR2: 00000000000005c8 CR3: 0000001cbaf4a004 CR4: 0000000000f72ef0
Call Trace:
<TASK>
mod_memcg_state.constprop.0+0x1c/0x30
vfree.part.0+0xa7/0x240
spl_slab_reclaim+0xf3/0x140 [spl]
arc_buf_destroy_impl+0x41/0x140 [zfs]
arc_buf_destroy+0x69/0xe0 [zfs]
dbuf_destroy+0x2e/0x2e0 [zfs]
dbuf_evict_one+0x11f/0x160 [zfs]
dbuf_evict_thread+0xa2/0x100 [zfs]
thread_generic_wrapper+0x5a/0x70 [spl]
kthread+0xcf/0x100
ret_from_fork+0x31/0x50
ret_from_fork_asm+0x1a/0x30
</TASK>
Kernel panic - not syncing: Fatal exception
Analysis of event 4
- The faulting instruction is
mov rdx, [rdi+0x548] with RDI = 0x80: the
mem_cgroup pointer passed to __mod_memcg_state() held the value 0x80
(CR2 = 0x80 + 0x548 = 0x5c8). RSI = 0x32 (the stat item) and RDX = 0xffffffff
(a −1 delta) look normal; only the memcg pointer is garbage.
- The memcg pointer on this path derives from the page being
vfree()d by
spl_slab_reclaim() during ARC dbuf eviction — i.e. the page's memcg_data (or the
vm area's accounting) held a small flag-like integer where a pointer belongs.
R12 = fffff0e0efe5f4c0 is consistent with a struct page in the vmemmap region.
- 0x80 is not a plausible bitflip of any valid kernel pointer (they differ in dozens of
bits), matching the pattern of events 1–3: logically wrong values, not decayed ones.
- This is the first of the four on the free/teardown path; the thread is ZFS's own
dbuf-cache eviction thread.
- Workload context, offered without causal claim: for the ~12 minutes before the fault, a
remote backup client had been sequentially reading one dataset over an NFSv4 read-only
mount at a modest ~11 MiB/s. Sustained reads do churn the dbuf cache (whence
dbuf_evict activity), but the same nightly job performs the same reads without
incident on other days.
- Unlike events 1–3, this fault arrived only 9h 26m after boot — the box was on a
nightly-reboot regime by this date, which the fault therefore defeats.
What is common to all four
- All four are a length, count, or pointer field holding a value it cannot legitimately
hold — and in every case the garbage is logically shaped (a pointer half in a count,
an all-zeroes struct, ASCII text as data, a flag-like small int as a pointer), not
single-bit decay.
- They span the write path, the read path, and the eviction/free path, and two
OpenZFS versions (2.3.0, 2.3.4).
- Events 1–3 arrived a whole number of days after boot, +0–7 minutes (5d, 3d, 3d). We
noted that as an odd empirical pattern; event 4 (9h 26m) breaks it, so we no longer
offer it as predictive — recorded only for completeness.
- No pool damage:
zpool status reports ONLINE, 0 0 0, "No known data errors" after
each event, and scrubs repair 0 B.
What we have ruled out
- Transparent huge pages — an earlier, separately root-caused crash series on this same
box was THP colliding with ARC. THP has read [never] throughout, verified after every
upgrade, with compact_stall and compact_daemon_wake both 0.
- Power — the machine is fed through a metered smart plug logging every 6–10 s. Across
one of these events voltage never left 241.6–245.3 V, and power rose (the boot
signature) rather than dipping. Two identical machines on the same UPS output have zero
unclean resets over the same period. Sub-second transients remain formally unexcluded.
- Disks — SMART clean on all four NVMe devices; zero lifetime minutes above warning
temperature; no nvme error has ever appeared on the netconsole wire; both legs of a
mirror are never independently at fault.
- Thermal — CPU ~56 °C, NVMe 50–53 °C, NIC 56 °C under sustained load, against limits of
95/105 °C.
- Memory leak or exhaustion — three days of continuous recorder history covering the run
that ended in event 3: ARC steady within a ~0.5 GB band (121.4 → 119.1 GB), total memory
usage flat at 94–96%. ARC is the elastic consumer here, so a leak elsewhere would have
forced ARC eviction; it did not.
Known limitation, stated up front
The RAM is non-ECC (Error Correction Type: None), so we cannot categorically exclude a
memory fault. What argues against it: memtest has been run repeatedly across this machine's
history and has always passed; and events 1 and 2 put garbage into the same two struct
fields of the same function on two different kernels and two different module builds — a
pattern random bitflips do not reproduce. We would rather state this plainly than have it
surface later.
Reproduction
No reliable reproducer. The faults occur under ordinary mixed NFS/SMB load, and there is no
known trigger we can invoke on demand. The boot-relative cadence seen in events 1–3 did not
survive event 4, so we currently have no predictive signal at all.
What we can supply on request
- Full unabridged netconsole captures for all four events, including complete module lists.
- The exact
zfs.ko and kernel build in use, for stqs_count / stqs_taskq offset
verification.
zpool history -il, pool and dataset properties, and scrub results before and after each
event.- Forensics from a separate single-block corruption on this pool, dated by birth txg via
zdb -ddddd, where both mirror legs held identical bad data from birth — i.e. the buffer
went bad in memory between checksum and mirror split. We are not claiming it is the
same bug; it is offered in case it proves relevant.
System information
6.12.99-production+truenas(event 1 was on6.12.15-production+truenas)tank: 2 × 2-way mirror, 4 × NVMe (2 × Samsung 990 PRO 4TB, 2 × WD_BLACK SN850X 4TB)lz4on all datasetsnever;nvme_core.multipath=N; no ZFS module parameter overridesSummary
Four kernel faults over eleven days, all in ZFS code paths, on two different OpenZFS
versions and two different kernels. Two are fatal faults in
spa_taskq_dispatch()on thewrite path; one is an unrecoverable infinite loop in the LZ4 decompressor on the read path;
the fourth is a fatal fault during ARC buffer teardown (
dbuf_evict→arc_buf_destroy→spl_slab_reclaim→vfree), where themem_cgrouppointer derivedfrom a page being freed held the value
0x80. In every case a field that should hold asmall integer, count, or valid pointer instead holds garbage — twice demonstrably the
same two struct fields.
All four were captured over netconsole. Nothing reached disk, because a panicking or
soft-locked kernel cannot sync.
For completeness: this machine also had four earlier unclean resets (2026-07-25 to
2026-08-08) with no log trail at all, before netconsole was in place. We do not claim
they are this bug — they are uncharacterised, and at least one earlier crash series on this
box had a different, separately root-caused mechanism. Only the three captures below are
offered as evidence.
Event 1 — 2026-08-13, OpenZFS 2.3.0-1, kernel 6.12.15 — page fault, write path
Uptime at fault: 432347 s (5d 0h 06m).
Event 2 — 2026-08-16, OpenZFS 2.3.4-1, kernel 6.12.99 — divide error, write path
Uptime at fault: 259426 s (3d 0h 04m). Same function, after upgrading 2.3.0 → 2.3.4.
Analysis of events 1 and 2 — the same two struct fields, wrong both times
Decoding the
Code:bytes, the faulting sequence is:RCX=0xc0aa3570, which isthe low half of a module pointer:
R14=0xffffffffc0aa31d0, only 0x3a0 away. Apointer-shaped value sitting in an integer count field.
RCX= 0 — andR14= 0 as well.⇒ Same struct, same field pair, garbage in both, across two OpenZFS versions and two
kernels, in the write path both times.
Our hypothesis, offered as a lead rather than a finding: a taskq lifecycle race rather
than stray corruption. A count of
0together with aNULLpointer is the all-zeroesshape of a struct that has been freed, zeroed, or never populated — not the shape of random
corruption. OpenZFS 2.2+ added dynamically-sized ZIO taskqs; a dispatch for a
(type, priority)whose taskq set is torn down or not yet populated would produce exactlyevent 2, and event 1 looks like the same struct caught mid-teardown.
We have not confirmed that
+0x178/+0x180arestqs_count/stqs_taskqin theshipped
zfs.ko— someone with the matching build should verify that before giving ourreading any weight.
Event 3 — 2026-08-19, OpenZFS 2.3.4-1, kernel 6.12.99 — infinite loop, read path
Uptime at fault: 259623 s (3d 0h 07m). Not a panic — one CPU looped forever and the
machine had to be power-cycled after 40 minutes.
Analysis of event 3
register dump is byte-identical in every report — same
RIPoffset, sameRAX,RBX,RCX.RBXis the loop's advancing cursor (cmp rbx,rax; jb) and it neveradvanced. This is a genuinely non-terminating loop, not slow progress.
RDXholds0x61622f6e69622f3d, i.e. the ASCII bytes=/bin/ab— plain text occupyinga register the decompressor is treating as data.
middlewaredperforming a 4096-bytepread64. The machine hosts noVMs or containers (
virt.instance.query→[]); this was ordinary middleware file I/O,not a heavy workload.
txg_syncon both pools and asystemdtask blocked for 120–483 s; everyprocess touching the pool hung, including new SSH logins. Writes later resumed; the
looping thread never did, and it could not be signalled — a graceful reboot issued from
the web UI never started, and the box had to be hard power-cycled.
The point of interest: ZFS verifies the block checksum before handing the buffer to
arc_buf_fill()for decompression. So either a buffer that passed its checksum decompressedas garbage, or
arc_buf_fill()was handed the wrong buffer or length. Separately,LZ4_uncompress_unknownOutputSize()appears able to loop indefinitely on malformed inputrather than returning an error — arguably a hardening bug in its own right, whatever
delivered the bad input.
Event 4 — 2026-08-24, OpenZFS 2.3.4-1, kernel 6.12.99 — NULL-adjacent deref, ARC eviction/free path
Uptime at fault: 33976 s (9h 26m). Fault at 06:36:57 local; auto-reboot 10 s later.
Analysis of event 4
mov rdx, [rdi+0x548]withRDI=0x80: themem_cgrouppointer passed to__mod_memcg_state()held the value 0x80(
CR2= 0x80 + 0x548 = 0x5c8).RSI= 0x32 (the stat item) andRDX=0xffffffff(a −1 delta) look normal; only the memcg pointer is garbage.
vfree()d byspl_slab_reclaim()during ARC dbuf eviction — i.e. the page'smemcg_data(or thevm area's accounting) held a small flag-like integer where a pointer belongs.
R12=fffff0e0efe5f4c0is consistent with astruct pagein the vmemmap region.bits), matching the pattern of events 1–3: logically wrong values, not decayed ones.
dbuf-cache eviction thread.
remote backup client had been sequentially reading one dataset over an NFSv4 read-only
mount at a modest ~11 MiB/s. Sustained reads do churn the dbuf cache (whence
dbuf_evictactivity), but the same nightly job performs the same reads withoutincident on other days.
nightly-reboot regime by this date, which the fault therefore defeats.
What is common to all four
hold — and in every case the garbage is logically shaped (a pointer half in a count,
an all-zeroes struct, ASCII text as data, a flag-like small int as a pointer), not
single-bit decay.
OpenZFS versions (2.3.0, 2.3.4).
noted that as an odd empirical pattern; event 4 (9h 26m) breaks it, so we no longer
offer it as predictive — recorded only for completeness.
zpool statusreportsONLINE,0 0 0, "No known data errors" aftereach event, and scrubs repair 0 B.
What we have ruled out
box was THP colliding with ARC. THP has read
[never]throughout, verified after everyupgrade, with
compact_stallandcompact_daemon_wakeboth 0.one of these events voltage never left 241.6–245.3 V, and power rose (the boot
signature) rather than dipping. Two identical machines on the same UPS output have zero
unclean resets over the same period. Sub-second transients remain formally unexcluded.
temperature; no
nvmeerror has ever appeared on the netconsole wire; both legs of amirror are never independently at fault.
95/105 °C.
that ended in event 3: ARC steady within a ~0.5 GB band (121.4 → 119.1 GB), total memory
usage flat at 94–96%. ARC is the elastic consumer here, so a leak elsewhere would have
forced ARC eviction; it did not.
Known limitation, stated up front
The RAM is non-ECC (
Error Correction Type: None), so we cannot categorically exclude amemory fault. What argues against it: memtest has been run repeatedly across this machine's
history and has always passed; and events 1 and 2 put garbage into the same two struct
fields of the same function on two different kernels and two different module builds — a
pattern random bitflips do not reproduce. We would rather state this plainly than have it
surface later.
Reproduction
No reliable reproducer. The faults occur under ordinary mixed NFS/SMB load, and there is no
known trigger we can invoke on demand. The boot-relative cadence seen in events 1–3 did not
survive event 4, so we currently have no predictive signal at all.
What we can supply on request
zfs.koand kernel build in use, forstqs_count/stqs_taskqoffsetverification.
zpool history -il, pool and dataset properties, and scrub results before and after eachevent.
zdb -ddddd, where both mirror legs held identical bad data from birth — i.e. the bufferwent bad in memory between checksum and mirror split. We are not claiming it is the
same bug; it is offered in case it proves relevant.