Basic NVME support built-in

[vazub]

NVME devices are more and more prevalent, especially in the desktop segment. It would make sense to consider them first-class citizen, without depending on initramfs. This would allow to drop initramfs altogether for even more cases, without growing the kernel size too much.

[ptr1337]

Related discussion:

https://gitlab.archlinux.org/archlinux/packaging/packages/linux/-/issues/126

[vazub]

Related discussion:

https://gitlab.archlinux.org/archlinux/packaging/packages/linux/-/issues/126

I would argue, that the discussion is not specifically applicable to CachyOS case, due to a different target audience defined for the project. For a desktop-first OS, the reported original reasons for building NVME support as a module, are not actually relevant. One is for VM-specific usage, another is for a 3rd-party manufacturer that can't seem to be bothered by using the default Linux driver.

The otherwise pros of providing a generic built-in support for the prevalent type of drives in modern desktop systems would outweigh the mentioned cons, in my opinion.

[ptr1337]

I just added it as context in general for the discussion to consider also their views :)

This was not directly a "no, we do not do it"

[damachine]

Improvements are always a good thing. We all love it and want it!
But :P
Excuse me if I raise a few points of criticism as a long-time Arch user.

• The NVMe controllers that have problems with the standard driver (FS#68021, FS#74271) don't suddenly work better just because the operating system has a different name.
• Possible anyone with a problematic controller is affected by CachyOS just the same.
• An integrated module cannot be disabled.
• Gamers also use VMs (for testing, for older games, etc.).
• Desktop users also have hardware problems.

How significant(~x) would the performance increase be during boot? Is it worth it?

The I/O runtime performance makes no difference.

• The NVMe driver is the same code.
• Whether integrated or as a module – the data transfer is identical.
• Latency, throughput, IOPS: all the same.

I think Arch took the right approach by rejecting this. as long as the points are not solved.
Of course, this is just my opinion.
Peace

[vazub]

@damachine

* An integrated module cannot be disabled.

This is simply a false and misleading statement. Easily done at kernel boot with an added parameter:

initcall_blacklist=nvme_init

Problem solved.

With this, your other argumentation points are simply irrelevant.

[damachine]

@vazub

Thanks for the response! However, I have to respectfully disagree.

initcall_blacklist=nvme_init is not equivalent to proper module blacklisting, as was already discussed in the original Arch issue #126:

The core issue:

With built-in drivers (=y):

• modprobe.blacklist doesn't work at all
• initcall_blacklist=nvme_init is the only option, but:
  • Users reported it doesn't work reliably
  • Requires knowing exact function names
  • And initcall_blacklist is Designed as a debugging tool, not a permanent solution

  https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html
  initcall_blacklist=  [KNL] Do not execute a comma-separated list of
                        initcall functions.  Useful for debugging built-in
                        modules and initcalls.
  

I appreciate CachyOS pushing boundaries for performance, but forcing a built-in driver that breaks standard troubleshooting methods seems like a regression in system flexibility for a very minor gain.

[aidanharris]

Why were these issues not considered a problem for SATA drives which are built-in? Even if you build it in I think you should still be able to unbind the device like this:
https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF#Binding_vfio-pci_via_device_ID

For example:

# cd /sys/module/nvme/drivers/pci:nvme
# ls -al
total 0
drwxr-xr-x  2 root root    0 Mar  2 04:38 .
drwxr-xr-x 28 root root    0 Mar  2 04:38 ..
lrwxrwxrwx  1 root root    0 Mar  2 04:38 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:01:00.0
lrwxrwxrwx  1 root root    0 Mar  2 04:38 0000:04:00.0 -> ../../../../devices/pci0000:00/0000:00:01.2/0000:02:00.0/0000:03:01.0/0000:04:00.0
lrwxrwxrwx  1 root root    0 Mar  2 04:38 0000:0f:00.0 -> ../../../../devices/pci0000:00/0000:00:03.2/0000:0f:00.0
--w-------  1 root root 4096 Mar  2 04:38 bind
lrwxrwxrwx  1 root root    0 Mar  2 04:38 module -> ../../../../module/nvme
--w-------  1 root root 4096 Mar  2 04:38 new_id
--w-------  1 root root 4096 Mar  2 04:42 remove_id
--w-------  1 root root 4096 Mar  2 04:38 uevent
--w-------  1 root root 4096 Mar  2 04:38 unbind
# nvme="0000:01:00.0"
# /bin/echo 0000:01:00.0 > unbind
# modprobe vfio-pci
# nvme_vd="$(cat /sys/bus/pci/devices/$nvme/vendor) $(cat /sys/bus/pci/devices/$nvme/device)"
# echo $nvme_vd
0x144d 0xa80a
# /bin/echo 0000:01:00.0 > unbind
# At this point the device won't show in lsblk anymore
# echo "$nvme_vd" > /sys/bus/pci/drivers/vfio-pci/new_id
# ls -al /sys/bus/pci/drivers/vfio-pci/
total 0
drwxr-xr-x  2 root root    0 Mar  2 05:01 .
drwxr-xr-x 29 root root    0 Mar  2 04:53 ..
lrwxrwxrwx  1 root root    0 Mar  2 05:01 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:01:00.0
--w-------  1 root root 4096 Mar  2 04:56 bind
lrwxrwxrwx  1 root root    0 Mar  2 04:54 module -> ../../../../module/vfio_pci
--w-------  1 root root 4096 Mar  2 05:01 new_id
--w-------  1 root root 4096 Mar  2 04:54 remove_id
--w-------  1 root root 4096 Mar  2 04:54 uevent
--w-------  1 root root 4096 Mar  2 04:54 unbind

This can be done as a hook in the initramfs or with a Systemd unit.

[daedaevibin]

One possible approach would be to build in the core components required
to boot on the vast majority of modern desktop systems. This would allow
CachyOS to natively boot without relying on an initramfs in many cases,
while keeping the kernel size impact relatively small.

For example, building in components such as:

NVME
EXT4
BTRFS
XFS
USB HID
EFI
PCI

These represent the most common storage, filesystem, and input paths
required for a typical desktop boot process.

A compromise already exists for NVMe specifically, since it is effectively
the modern equivalent of SATA storage. Enabling the core NVMe support
directly in the kernel would cover most systems without significantly
reducing flexibility:

CONFIG_NVME_CORE=y
CONFIG_BLK_DEV_NVME=y
CONFIG_NVME_PCI=y

[TheDcoder]

@damachine IMO these criticisms are very superficial, as someone with these niche non-compliant controllers which don't work with Linux wouldn't be using them with the system anyway, and if an OEM wants to ship their own modules, they can just as easily ship their own kernel, it's not something that we should bother ourselves with.

How significant(~x) would the performance increase be during boot? Is it worth it?

It's not just about performance, with the right configuration we can entirely skip a bulky portion of kernel updates and storage required for the kernel to boot, initramfs is dead weight if the kernel is already capable of mounting the root FS directly, it significantly eases the complexities associated with initramfs configuration.

And initcall_blacklist is Designed as a debugging tool, not a permanent solution

I think you're imposing your own interpretation on the documentation, it merely states that is useful for debugging, not that it was specifically designed for just that. It should in theory be the same as never loading the module since it is never initialized.

---

My take on this is that we should've never disabled building this as a built-in in the first place, it breaks feature parity with systems which use SATA, SCSI etc. as they are able to boot themselves without loading an image with the modules. I simply see no good reason to keep the parity broken.

[damachine]

@TheDcoder I think I might have expressed myself a bit poorly earlier. To be clear: I am a fan of this change and I use a built-in setup myself to keep my boot process as lean as possible. However, my argument comes from a place of 'distro-wide responsibility'. While we (as enthusiasts) love the minimalism, we shouldn't ignore the reason why Arch keeps this as a module. If CachyOS is confident in prioritizing this 'enthusiast-first' optimization over the fail-safe flexibility of a module, I'm all for it. I just wanted to make sure we’re making that trade-off consciously. I'm not here to block it, just giving my two cents as a long-time user.

[daedaevibin]

@TheDcoder I think I might have expressed myself a bit poorly earlier. To be clear: I am a fan of this change and I use a built-in setup myself to keep my boot process as lean as possible. However, my argument comes from a place of 'distro-wide responsibility'. While we (as enthusiasts) love the minimalism, we shouldn't ignore the reason why Arch keeps this as a module. If CachyOS is confident in prioritizing this 'enthusiast-first' optimization over the fail-safe flexibility of a module, I'm all for it. I just wanted to make sure we’re making that trade-off consciously.

I will say this, while I see and understand the decision behind your thoughts.

NVMe is modern hardware, and judging by the fact CachyOS is built for modern hardware (Actually, you cannot even install it without meeting a certain standard) it should support at least the Core parts to prevent bloat elsewhere and diminish edge-case issues.

This would be the "logically consistent" approach to match the other things CachyOS has been doing or is currently doing.

Current:

Kernel -> initramfs -> Load Module -> Mount Root.

This is the general arch route, to run on all hardware from a 2008 PC to current modern devices.

It should be, by CachyOS standards:

Kernel -> Mount Root.

---

In my own experience, the driver is rather stable, and would make the system "leaner" overall, which is the primary goal I see consistently. It is also small, so it really is not very much trouble at all.