(AI Disclosure: Claude Opus used to interrogate logs and draft report, however reviewed for accuracy and terseness before submitting. Also submitted to Proxmox bugzilla - https://bugzilla.proxmox.com/show_bug.cgi?id=7924
System information
| Type |
Version/Name |
| Distribution Name |
Proxmox VE |
| Distribution Version |
9.2.10 (Debian-based) |
| Kernel Version |
7.0.14-8-pve (#1 SMP PREEMPT_DYNAMIC, PREEMPT(lazy)) |
| Architecture |
x86_64 (AMD EPYC 4545P, 128 GiB RAM) |
| OpenZFS Version |
zfs-2.4.3-pve1 (kmod) |
Pool: single pool "xserve", 55.9T NVMe, ONLINE, no errors, dedup off, ~16% cap.
Describe the problem you're observing
Host took a fatal kernel oops in the ZFS mmap read path. A process (nifmake, a build tool running as an unprivileged user inside an unprivileged LXC container) page-faulted on a memory-mapped file on ZFS. The oops shows memset called from zfs_fillpage → dmu_read with an underflowed (negative) length:
RDX: ffffffffffff6a00 (memset count, ≈ -0x9600)R14: ffffffffffffe000, R15: 0000000000002000, R13: 0000000000007600- Faulting address
CR2: ffff8bb330a00000, error code 0x0003 (present, write, permissions violation) — i.e. the runaway memset wrote forward through kernel memory until it hit a protected page.
This is consistent with the file being truncated concurrently with the page fault, so that i_size - page_offset in the fillpage path went negative.
Context: two containers on the host were in a sustained memcg OOM-kill storm (the kernel OOM killer was repeatedly killing tasks inside their cgroups). The oops occurred in page-fault context; the host did not recover (panic_on_oops=0) and hung until manually power-cycled ~18 minutes later.
Behaviour seen across kernels 7.0.2→7.0.14 and several zfs point releases. Earlier hangs left no pstore record (journal simply stops mid-activity), which is consistent with the same oops-then-hang mode. However only this occurrence was captured via EFI pstore.
Possibly related historical report: #4705 (zfs_fillpage out-of-bounds, 2016).
Describe how to reproduce the problem
No minimal reproducer yet. Conditions when it fired:
- Unprivileged LXC containers on a ZFS-backed rootfs (Proxmox).
- Heavy memcg OOM pressure in two containers (OOM killer firing repeatedly, killing tasks mid-operation).
- A build workload (
nifmake) reading files via mmap while files are being modified/truncated (build churn + OOM kills interrupting writers).
Presumed race: mmap page fault begins on a page near EOF; the file is truncated before the fillpage length computation; i_size - page_offset underflows; memset/dmu_read runs with a huge length and corrupts kernel memory until faulting.
Include any warning/errors/backtraces from the system logs
<1>[217237.894354] BUG: unable to handle page fault for address: ffff8bb330a00000
<1>[217237.894373] #PF: supervisor write access in kernel mode
<1>[217237.894378] #PF: error_code(0x0003) - permissions violation
<6>[217237.894384] PGD 1d33e01067 P4D 1d33e01067 PUD 100618063 PMD 8000001d30a001a1
<4>[217237.894392] Oops: Oops: 0003 [#1] SMP NOPTI
<4>[217237.894398] CPU: 1 UID: 101000 PID: 443866 Comm: nifmake Tainted: P OE 7.0.14-8-pve #1 PREEMPT(lazy)
<4>[217237.894406] Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
<4>[217237.894412] Hardware name: B650D4U3-2L2Q/BCM/B650D4U3-2L2Q/BCM, BIOS 22.11 02/03/2026
<4>[217237.894418] RIP: 0010:memset+0xb/0x20
<4>[217237.894425] Code: 06 44 8a 1e 44 88 1f c3 cc cc cc cc 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66 90 49 89 f9 40 88 f0 48 89 d1 <f3> aa 4c 89 c8 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 90
<4>[217237.894437] RSP: 0018:ffffcc549fc03a80 EFLAGS: 00010286
<4>[217237.894443] RAX: 0000000000007600 RBX: ffff8ba69652b000 RCX: fffffff365b29000
<4>[217237.894449] RDX: ffffffffffff6a00 RSI: 0000000000000000 RDI: ffff8bb330a00000
<4>[217237.894454] RBP: ffffcc549fc03ae8 R08: 0000000000000000 R09: ffff8ba696532600
<4>[217237.894461] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
<4>[217237.894466] R13: 0000000000007600 R14: ffffffffffffe000 R15: 0000000000002000
<4>[217237.894472] FS: 00007fda1f76fa80(0000) GS:ffff8bb5ad38e000(0000) knlGS:0000000000000000
<4>[217237.894478] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
<4>[217237.894483] CR2: ffff8bb330a00000 CR3: 0000000e58f09000 CR4: 0000000000f50ef0
<4>[217237.894489] PKRU: 55555554
<4>[217237.894492] Call Trace:
<4>[217237.894495] <TASK>
<4>[217237.894501] ? dmu_read_impl+0x72/0x230 [zfs]
<4>[217237.894619] dmu_read+0x61/0xb0 [zfs]
<4>[217237.894706] zfs_fillpage+0x16b/0x210 [zfs]
<4>[217237.894772] zfs_getpage+0xe1/0x360 [zfs]
<4>[217237.894820] zpl_read_folio+0x3e/0x70 [zfs]
<4>[217237.894865] ? __pfx_zpl_read_folio+0x10/0x10 [zfs]
<4>[217237.894908] filemap_read_folio+0x42/0xf0
<4>[217237.894915] filemap_fault+0xa27/0x1300
<4>[217237.894920] __do_fault+0x3d/0x190
<4>[217237.894923] do_fault+0x33a/0x580
<4>[217237.894927] __handle_mm_fault+0x9d5/0x1030
<4>[217237.894932] handle_mm_fault+0x136/0x3a0
<4>[217237.894936] do_user_addr_fault+0x2f8/0x820
<4>[217237.894941] exc_page_fault+0x81/0x1c0
<4>[217237.895070] asm_exc_page_fault+0x27/0x30
<4>[217237.895179] RIP: 0033:0x56e6c7b0c4e0
<4>[217237.895450] RSP: 002b:00007ffd8fefb3a0 EFLAGS: 00010293
<4>[217237.895536] RAX: 00007fda1f971000 RBX: 00007ffd8fefb560 RCX: 0000000000000018
<4>[217237.895623] RDX: 00007fda1f971000 RSI: 00007fda1f9784d4 RDI: 0000000000000000
<4>[217237.895706] RBP: 00007ffd8fefb4e0 R08: 000056e6c7b24e40 R09: 0000000000000000
<4>[217237.895792] R10: 0000000000000055 R11: 00007fda1f6f1040 R12: ffffffffffffff70
<4>[217237.895874] R13: 0000000000000003 R14: 00007ffd8fefb6f0 R15: ffffffffffffff70
<4>[217237.895955] </TASK>
<4>[217237.896033] Modules linked in: dm_snapshot tcp_diag inet_diag nvidia_uvm(OE) nvidia_drm(OE) nvidia_modeset(OE) nvidia(OE) iommufd nft_ct nft_masq nft_chain_nat nf_nat nf_conntrack [...] zfs(PO) spl(O) [...] [last unloaded: vfio]
<4>[217237.897530] CR2: ffff8bb330a00000
<4>[217237.897627] ---[ end trace 0000000000000000 ]---
Note: kernel is tainted P/OE by out-of-tree nvidia modules; they do not appear anywhere in the trace, which is entirely in the mm/ZFS mmap read path.
Immediately preceding the oops (same second), the journal shows a memcg OOM storm in two LXC cgroups (oom_memcg=/lxc/107, /lxc/132), with the OOM killer repeatedly invoked. Full pstore dmesg captures (including the OOM process-table dump) are attached: panic-190.txt, panic-192.txt.
panic-190.txt
panic-192.txt
(AI Disclosure: Claude Opus used to interrogate logs and draft report, however reviewed for accuracy and terseness before submitting. Also submitted to Proxmox bugzilla - https://bugzilla.proxmox.com/show_bug.cgi?id=7924
System information
Pool: single pool "xserve", 55.9T NVMe, ONLINE, no errors, dedup off, ~16% cap.
Describe the problem you're observing
Host took a fatal kernel oops in the ZFS mmap read path. A process (
nifmake, a build tool running as an unprivileged user inside an unprivileged LXC container) page-faulted on a memory-mapped file on ZFS. The oops showsmemsetcalled fromzfs_fillpage → dmu_readwith an underflowed (negative) length:RDX: ffffffffffff6a00(memset count, ≈ -0x9600)R14: ffffffffffffe000,R15: 0000000000002000,R13: 0000000000007600CR2: ffff8bb330a00000, error code 0x0003 (present, write, permissions violation) — i.e. the runaway memset wrote forward through kernel memory until it hit a protected page.This is consistent with the file being truncated concurrently with the page fault, so that
i_size - page_offsetin the fillpage path went negative.Context: two containers on the host were in a sustained memcg OOM-kill storm (the kernel OOM killer was repeatedly killing tasks inside their cgroups). The oops occurred in page-fault context; the host did not recover (panic_on_oops=0) and hung until manually power-cycled ~18 minutes later.
Behaviour seen across kernels 7.0.2→7.0.14 and several zfs point releases. Earlier hangs left no pstore record (journal simply stops mid-activity), which is consistent with the same oops-then-hang mode. However only this occurrence was captured via EFI pstore.
Possibly related historical report: #4705 (zfs_fillpage out-of-bounds, 2016).
Describe how to reproduce the problem
No minimal reproducer yet. Conditions when it fired:
nifmake) reading files via mmap while files are being modified/truncated (build churn + OOM kills interrupting writers).Presumed race: mmap page fault begins on a page near EOF; the file is truncated before the fillpage length computation;
i_size - page_offsetunderflows; memset/dmu_read runs with a huge length and corrupts kernel memory until faulting.Include any warning/errors/backtraces from the system logs
Note: kernel is tainted P/OE by out-of-tree nvidia modules; they do not appear anywhere in the trace, which is entirely in the mm/ZFS mmap read path.
Immediately preceding the oops (same second), the journal shows a memcg OOM storm in two LXC cgroups (
oom_memcg=/lxc/107,/lxc/132), with the OOM killer repeatedly invoked. Full pstore dmesg captures (including the OOM process-table dump) are attached:panic-190.txt,panic-192.txt.panic-190.txt
panic-192.txt