Skip to content

fix: load kernel modules via host kmod on non-container OS - #37

Merged
BSWANG merged 1 commit into
AliyunContainerService:mainfrom
BSWANG:fix/modprobe-on-host-zstd
Jul 23, 2026
Merged

BSWANG merged 1 commit into
AliyunContainerService:mainfrom
BSWANG:fix/modprobe-on-host-zstd

Conversation

@BSWANG

@BSWANG BSWANG commented Jul 23, 2026

Copy link
Copy Markdown
Member

Problem

On nodes whose host kmod is newer than the agent container's, modprobe erdma fails with:

modprobe: ERROR: could not insert 'erdma': Exec format error

Root cause: module load (modprobe/rmmod for erdma, smc, nvidia-peermem) ran inside the agent container via containerExec. The container's kmod (e.g. Alibaba Linux 8, kmod-25, +XZ +ZLIB only) cannot decompress ZSTD-compressed modules shipped by newer hosts (e.g. Ubuntu 24.04, kmod with +ZSTD). The module files live in the host's /lib/modules (a hostPath mount) and load into the shared host kernel, so they must be decompressed by the host kmod.

Fix

Add a moduleExec() helper that selects the exec context for kernel-module operations:

  • regular OS → hostExec (enters host mount/cgroup namespaces via nsenter, using the host's ZSTD-capable kmod)
  • container-optimized OS (lifsea, already runs with host userspace) → containerExec

Route modprobe erdma, EnsureSMCR (modprobe smc) and nvidia-peermem loading through it in the default, compat and ofed drivers.

This also:

  • aligns the load step with the install step, which already ran on the host via hostExec for non-container OS;
  • collapses the duplicated host/container selection in the OFED driver into moduleExec().

No new privileges are required — hostExec (nsenter -t 1, hostPID, SYS_ADMIN, hostPath /lib/modules) was already used by the install step and the OFED driver on these same nodes.

Test

  • GOOS=linux / GOOS=darwin build, go vet, and go test ./internal/drivers/ all pass.

modprobe/rmmod for erdma, smc and nvidia-peermem were run inside the
agent container via containerExec. The container's kmod (e.g. Alibaba
Linux 8, kmod-25, only +XZ +ZLIB) cannot decompress ZSTD-compressed
modules shipped by newer hosts (e.g. Ubuntu 24.04, kmod +ZSTD), so
loading a host-built .ko.zst failed with:

  modprobe: ERROR: could not insert 'erdma': Exec format error

The module files live in the host's /lib/modules (a hostPath mount) and
load into the shared host kernel, so they must be decompressed by the
host kmod. Route all module operations through a new moduleExec() helper
that uses hostExec on a regular OS and containerExec on a
container-optimized OS (lifsea, which already runs with host userspace).

This also aligns the load step with the install step, which already ran
on the host via hostExec for non-container OS, and collapses the
duplicated host/container selection in the OFED driver into moduleExec().
@BSWANG
BSWANG force-pushed the fix/modprobe-on-host-zstd branch from 8888bc0 to 3b3b7e6 Compare July 23, 2026 16:34
@BSWANG
BSWANG merged commit 72e8f0c into AliyunContainerService:main Jul 23, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant