kernel oops on 6.12.56 with zfs 6cfc3dba9c

[tycho]

System information

Type	Version/Name
Distribution Name	Arch Linux
Distribution Version	N/A
Kernel Version	6.12.56
Architecture	x86_64
OpenZFS Version	6cfc3db

Describe the problem you're observing

On boot, at pool import time:

[    3.925788] BUG: unable to handle page fault for address: ffffd4dcc7324000
[    3.925798] #PF: supervisor write access in kernel mode
[    3.925803] #PF: error_code(0x0002) - not-present page
[    3.925807] PGD 100000067 P4D 100000067 PUD 1002a0067 PMD 125d15067 PTE 0
[    3.925817] Oops: Oops: 0002 [#1] PREEMPT SMP NOPTI
[    3.925822] CPU: 6 UID: 0 PID: 588 Comm: zpool Tainted: P           O       6.12.56-1-hsw-lts #1 5a8e91f07c1cf97ff3cf23390d666e2e8182aff6
[    3.925832] Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE
[    3.925836] Hardware name: Puget Systems System Product Name/ROG STRIX X570-E GAMING, BIOS 4021 08/09/2021
[    3.925842] RIP: 0010:memcpy+0x8/0x10
[    3.925848] Code: 2f 27 00 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66 90 48 89 f8 48 89 d1 <f3> a4 e9 0c 2f 27 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90
[    3.925858] RSP: 0018:ffffd4dcc117b8d8 EFLAGS: 00010286
[    3.925863] RAX: ffffd4dcc7324000 RBX: ffffd4dcc732c000 RCX: 0000000000008000
[    3.925867] RDX: 0000000000008000 RSI: ffffd4dcc7a38000 RDI: ffffd4dcc7324000
[    3.925872] RBP: 0000000000008000 R08: 0000000000000030 R09: 0000000000000000
[    3.925876] R10: ffffffffffffffff R11: 000000000000002c R12: 0000000000000067
[    3.925881] R13: ffff8e27a4956200 R14: 0000000000983800 R15: 0000000000338000
[    3.925886] FS:  00007f267243c900(0000) GS:ffff8e462e900000(0000) knlGS:0000000000000000
[    3.925891] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    3.925895] CR2: ffffd4dcc7324000 CR3: 00000001059e6000 CR4: 0000000000750ef0
[    3.925900] PKRU: 55555554
[    3.925903] Call Trace:
[    3.925907]  <TASK>
[    3.925911]  dmu_read_impl+0x115/0x220 [zfs 722a384bd62b387a490473f88d70221ef829dc79]
[    3.926042]  dmu_read+0x54/0xa0 [zfs 722a384bd62b387a490473f88d70221ef829dc79]
[    3.926138]  ? srso_alias_return_thunk+0x5/0xfbef5
[    3.926145]  brt_load+0x105/0x3e0 [zfs 722a384bd62b387a490473f88d70221ef829dc79]
[    3.926220]  spa_load+0xd80/0x1af0 [zfs 722a384bd62b387a490473f88d70221ef829dc79]
[    3.926313]  ? srso_alias_return_thunk+0x5/0xfbef5
[    3.926318]  spa_tryimport+0x164/0x490 [zfs 722a384bd62b387a490473f88d70221ef829dc79]
[    3.926400]  zfs_ioc_pool_tryimport+0x67/0xc0 [zfs 722a384bd62b387a490473f88d70221ef829dc79]
[    3.926477]  zfsdev_ioctl_common+0x6e9/0x940 [zfs 722a384bd62b387a490473f88d70221ef829dc79]
[    3.926549]  ? srso_alias_return_thunk+0x5/0xfbef5
[    3.926554]  zfsdev_ioctl+0x4f/0xd0 [zfs 722a384bd62b387a490473f88d70221ef829dc79]
[    3.926619]  __x64_sys_ioctl+0x96/0xd0
[    3.926625]  do_syscall_64+0x75/0x180
[    3.926630]  ? srso_alias_return_thunk+0x5/0xfbef5
[    3.926634]  ? __alloc_pages_noprof+0x16a/0x330
[    3.926640]  ? srso_alias_return_thunk+0x5/0xfbef5
[    3.926643]  ? __mod_memcg_lruvec_state+0x9b/0x150
[    3.926648]  ? srso_alias_return_thunk+0x5/0xfbef5
[    3.926652]  ? __lruvec_stat_mod_folio+0x84/0xd0
[    3.926655]  ? srso_alias_return_thunk+0x5/0xfbef5
[    3.926660]  ? srso_alias_return_thunk+0x5/0xfbef5
[    3.926663]  ? __folio_mod_stat+0x2d/0x90
[    3.926668]  ? srso_alias_return_thunk+0x5/0xfbef5
[    3.926671]  ? folio_add_new_anon_rmap+0x71/0x180
[    3.926675]  ? srso_alias_return_thunk+0x5/0xfbef5
[    3.926679]  ? __handle_mm_fault+0xe61/0x1ee0
[    3.926684]  ? __x64_sys_close+0x38/0x80
[    3.926689]  ? srso_alias_return_thunk+0x5/0xfbef5
[    3.926693]  ? __count_memcg_events+0x59/0xf0
[    3.926697]  ? srso_alias_return_thunk+0x5/0xfbef5
[    3.926701]  ? handle_mm_fault+0x1c5/0x2c0
[    3.926706]  ? srso_alias_return_thunk+0x5/0xfbef5
[    3.926709]  ? do_user_addr_fault+0x35a/0x610
[    3.926714]  ? srso_alias_return_thunk+0x5/0xfbef5
[    3.926718]  ? srso_alias_return_thunk+0x5/0xfbef5
[    3.926722]  ? irqentry_exit_to_user_mode+0x28/0x190
[    3.926726]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
[    3.926731] RIP: 0033:0x7f2672cd474d
[    3.926734] Code: 04 25 28 00 00 00 48 89 45 c8 31 c0 48 8d 45 10 c7 45 b0 10 00 00 00 48 89 45 b8 48 8d 45 d0 48 89 45 c0 b8 10 00 00 00 0f 05 <89> c2 3d 00 f0 ff ff 77 1a 48 8b 45 c8 64 48 2b 04 25 28 00 00 00
[    3.926742] RSP: 002b:00007ffce96cc570 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[    3.926747] RAX: ffffffffffffffda RBX: 0000555b2b6ba540 RCX: 00007f2672cd474d
[    3.926750] RDX: 00007ffce96cc5f0 RSI: 0000000000005a06 RDI: 0000000000000003
[    3.926754] RBP: 00007ffce96cc5c0 R08: 0000000000000000 R09: 00007ffce96cc434
[    3.926757] R10: 0000000000000022 R11: 0000000000000246 R12: 00007ffce96cfe00
[    3.926761] R13: 0000555b2b6c10c8 R14: 00007ffce96cfe80 R15: 0000555b2b6c1d20
[    3.926767]  </TASK>
[    3.926769] Modules linked in: jc42 regmap_i2c intel_rapl_msr intel_rapl_common ee1004 polyval_clmulni asus_ec_sensors snd_hda_codec_realtek snd_hda_codec_generic polyval_generic snd_hda_scodec_component ghash_clmulni_intel snd_hda_codec_hdmi sha512_ssse3 sha1_ssse3 snd_hda_intel aesni_intel snd_intel_dspcfg eeepc_wmi snd_intel_sdw_acpi crypto_simd asus_wmi btusb sparse_keymap btmtk cryptd btrtl snd_hda_codec btbcm nvidia_drm(PO+) i8042 btintel nvidia_modeset(PO) snd_hwdep platform_profile drm_ttm_helper battery ttm bluetooth kvm_amd snd_hda_core video sp5100_tco ccp wmi_bmof ixgbe(+) r8169 watchdog usbip_host snd_pcm realtek usbip_core mdio_devres snd_timer pl2303 i2c_piix4 mdio snd soundcore k10temp i2c_smbus rfkill usbserial igb libphy i2c_algo_bit dca nvidia_uvm(PO) tpm_crb tpm_tis tpm_tis_core nvidia(PO) zfs(PO) i2c_dev spl(O) edac_mce_amd crypto_user fuse bpf_preload tpm libaescfb ecdh_generic ecc ip_tables sha256_ssse3 usbhid mxm_wmi wmi btrfs blake2b_generic libcrc32c crc32c_intel xor raid6_pq nvme nvme_core
[    3.926846]  nvme_auth
[    3.926877] CR2: ffffd4dcc7324000
[    3.926881] ---[ end trace 0000000000000000 ]---
[    3.926884] RIP: 0010:memcpy+0x8/0x10
[    3.926887] Code: 2f 27 00 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66 90 48 89 f8 48 89 d1 <f3> a4 e9 0c 2f 27 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90
[    3.926895] RSP: 0018:ffffd4dcc117b8d8 EFLAGS: 00010286
[    3.926898] RAX: ffffd4dcc7324000 RBX: ffffd4dcc732c000 RCX: 0000000000008000
[    3.926902] RDX: 0000000000008000 RSI: ffffd4dcc7a38000 RDI: ffffd4dcc7324000
[    3.926905] RBP: 0000000000008000 R08: 0000000000000030 R09: 0000000000000000
[    3.926909] R10: ffffffffffffffff R11: 000000000000002c R12: 0000000000000067
[    3.926913] R13: ffff8e27a4956200 R14: 0000000000983800 R15: 0000000000338000
[    3.926916] FS:  00007f267243c900(0000) GS:ffff8e462e900000(0000) knlGS:0000000000000000
[    3.926921] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    3.926924] CR2: ffffd4dcc7324000 CR3: 00000001059e6000 CR4: 0000000000750ef0
[    3.926928] PKRU: 55555554
[    3.926930] note: zpool[588] exited with irqs disabled

Describe how to reproduce the problem

I'm not 100% certain what the repro is, because I have a second system with a different ZFS pool on it, and that one booted up on the same kernel and ZFS version without complaint. I'm hoping the kernel oops log above gives some hints though.

Include any warning/errors/backtraces from the system logs

Nothing after the oops, because import fails and it won't see any pools after that point.

I previously booted this system without issues on revision 3a55e76, so I'm going to try rolling back to that and see if the system is okay.

[alex-moch]

I can confirm the same issue on Gentoo Linux with Kernel 6.17.7. I tried on multiple systems:

• Gentoo ~amd64/llvm with gentoo-kernel-bin-6.17.7 and zfs-9999: Kernel oops.
  • This is a bare metal workstation on ASUS TUF GAMING Z890-PRO WIFI.
  • Building zfs-9999 with GCC fixes the issue.
  • A custom kernel with ZFS builtin also fails when building with LLVM. GCC not tested.
• Gentoo ~amd64/hardened/selinux boots fine in QEMU/KVM but on Lenovo X1 Carbon I get an oops.
  • GCC compiler.
  • I noticed that I have set -O3 on the notebook. Will test later with -O2.
• Gentoo ~arm64/musl/llvm: No issues on 6.12.47.
  • This is a VM on VMware Fusion.

For the non-Gentoo users: zfs-9999 is the live ebuild which pulls the latest version from the git master.

[alex-moch]

There seems to be a race condition.

I set -O2 on the Lenovo X1 and it booted up with no issues. On the next reboot I got a kernel oops.

I remember that, when testing, during one of the boots I received a kernel NULL pointer dereference instead of unable to handle page fault.

After building zfs-9999 with GCC on the desktop workstation, I have done multiple reboots without issue.

EDIT: I was wrong about GCC. My desktop workstation just failed the latest boot/ZFS import.

[nilssonk]

It indeed seems to be a recently introduced race condition.

Openzfs master built against kernel 6.17.7 on x86_64 with gcc oopses on 80% of boots.

Downgrading to 2.4.0-rc3 seemingly solves the problem.

[nilssonk]

I did some bisecting and it seems like ea125ee is the likely culprit. I've been importing and exporting my pools in a loop for 30 minutes on the preceding 033dbdc and everything is normal.

@alex-moch Perhaps you can try these:

Good

EGIT_OVERRIDE_COMMIT_OPENZFS_ZFS=033dbdc emerge -1 zfs

Bad

EGIT_OVERRIDE_COMMIT_OPENZFS_ZFS=ea125ee emerge -1 zfs

[alex-moch]

I did some bisecting and it seems like ea125ee is the likely culprit.

I can confirm that. I reverted ec268cd and ea125ee in a local branch and my custom kernel 6.17.7 with ZFS built-in boots up fine and imports the ZFS pool.

@alex-moch Perhaps you can try these:

Good

EGIT_OVERRIDE_COMMIT_OPENZFS_ZFS=033dbdc emerge -1 zfs

Bad

EGIT_OVERRIDE_COMMIT_OPENZFS_ZFS=ea125ee emerge -1 zfs

I have also tested this against gentoo-kernel-bin-6.17.7 and the good commit booted and imported the ZFS pool fine while the bad commit immediately oopsed.

Thanks for testing, @nilssonk.

[sirus20x6]

I did some bisecting and it seems like ea125ee is the likely culprit.

I can confirm that. I reverted ec268cd and ea125ee in a local branch and my custom kernel 6.17.7 with ZFS built-in boots up fine and imports the ZFS pool.

@alex-moch Perhaps you can try these:

Good

EGIT_OVERRIDE_COMMIT_OPENZFS_ZFS=033dbdc emerge -1 zfs

Bad

EGIT_OVERRIDE_COMMIT_OPENZFS_ZFS=ea125ee emerge -1 zfs

I have also tested this against gentoo-kernel-bin-6.17.7 and the good commit booted and imported the ZFS pool fine while the bad commit immediately oopsed.

Thanks for testing, @nilssonk.

Confirmed this fixed it for me too!

[IvanVolosyuk]

@amotin FYI

[amotin]

Thanks for the ping. I think #17915 should fix it. #17875 seems not the cause, but merely a trigger.

[alex-moch]

I can confirm that #17915 fixed it for me.

[nilssonk]

Second confirmation that #17915 stops the oopsing.

Now, shouldn't this have been detected by fairly trivial testing of #17875?
How did the fact that something as simple as "importing a pool a few times in a row" breaks things slip by?

I'm not trying to be snarky or anything here, it is just surprising to me.

[alex-moch]

Now, shouldn't this have been detected by fairly trivial testing of #17875?
How did the fact that something as simple as "importing a pool a few times in a row" breaks things slip by?

At least for me, the bug was only triggered on bare metal. All the VMs imported the pools without issue.