running MCV without --privileged

Problem:

MCV uses buildah internally to build/push OCI cache images. Buildah traditionally
requires `--privileged` because it needs to mount overlay filesystems and run as root.
This is a security concern in production Kubernetes and CI environments.

What changed in the Containerfile

1. Storage driver: FUSE/overlay → VFS (redhat-et#173)

  driver="vfs"

  FUSE-overlayfs requires either --privileged or CAP_SYS_ADMIN + /dev/fuse access.
  VFS is a naive copy-based storage driver that needs no special kernel capabilities.
  It's slower (copies instead of overlays), but MCV only builds small single-layer
  cache images, so the performance difference is negligible.

2. Non-root user (UID/GID 1000)

  ```
  RUN groupadd -g 1000 appgroup && \
      useradd -u 1000 -g appgroup -m -s /bin/bash appuser
  USER appuser
  ```

  Running as root inside the container is unnecessary and a security risk. The
  fixed UID/GID 1000 also makes volume mount permissions predictable.

  Runtime flags: Podman vs Docker

  Podman:
  `podman run -v /path/to/cache:/tests:Z,U <image> ...`

  - :Z — relabels the volume for SELinux (private to this container)
  - :U — remaps the volume ownership to match the in-container user (UID 1000).
    Podman runs rootless with user namespaces, so the host UID and container UID
    differ. :U bridges that gap so appuser can read/write the mount.

  You only need :Z,U on volumes the container needs to write to (the cache output
  directory). Read-only mounts like model files only need :Z (or :ro,Z).

  Docker:

  ```
  docker run --user $(id -u):$(id -g) \
      --security-opt seccomp=unconfined \
      --security-opt apparmor=unconfined \
      -v /path/to/cache:/tests:Z \
      <image> ...
  ```

  - `--user $(id -u):$(id -g)` — Docker doesn't have Podman's user-namespace
    remapping, so you explicitly run as your host UID/GID to match volume ownership.
    Without this, the container runs as UID 1000 (appuser) which may not own the
    host-side mount.
  - `--security-opt seccomp=unconfined` — buildah makes syscalls (like mount,
    unshare) that Docker's default seccomp profile blocks. Disabling seccomp allows
    these without granting full --privileged.
  - `--security-opt apparmor=unconfined` — on Ubuntu (base container), AppArmor's
    default Docker profile also blocks some of buildah's mount/namespace operations.
    Disabling it is the minimal escalation needed.
  - `:Z` — SELinux relabeling, same as Podman. No :U needed because --user handles
    ownership directly.

  Why the difference

  Podman is rootless-native — it uses user namespaces automatically, so :U remaps
  ownership transparently. Docker doesn't do user-namespace remapping by default,
  so you need --user to align UIDs, and --security-opt to relax seccomp/AppArmor
  enough for buildah's syscalls without going full --privileged.

  What you're NOT granting

  Neither approach uses --privileged. The container does NOT get:
  - Full device access
  - CAP_SYS_ADMIN
  - Host PID/network namespace
  - Write access to /dev, /proc, /sys

  It's a targeted relaxation: let buildah do its mounts and namespace operations,
  nothing more.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Signed-off-by: Billy McFall <22157057+Billy99@users.noreply.github.com>

Author: Billy99

.github/workflows/mcv-build-example-images.yml

@@ -96,11 +96,11 @@ jobs:
             archive="${out_dir}/${tag}.oci"
 
             echo "==> Building ${image} using MCV container from /work/${reldir}"
-            podman run --rm --privileged \
+            podman run --rm \
               -e TAG="${tag}" \
               -e IMAGE="${image}" \
               -e RELDIR="${reldir}" \
-              -v "${GITHUB_WORKSPACE}:/work:rw" \
+              -v "${GITHUB_WORKSPACE}:/work:Z,U" \
               quay.io/gkm/mcv:latest bash -lc '
                 set -euo pipefail
                 /mcv -l info -c -i "$IMAGE" -d "/work/$RELDIR" &&

mcv/README.md

@@ -29,9 +29,57 @@ A Model/GPU kernel cache container packaging utility inspired by
 ### Install dependencies
 
 ```bash
-sudo dnf install gpgme-devel
-sudo dnf install btrfs-progs-devel
+sudo dnf install -y gpgme-devel btrfs-progs-devel
 ```
+OR
+```bash
+sudo apt install -y libgpgme-dev libbtrfs-dev uidmap
+```
+
+On Ubuntu 24.04, *running* `mcv` unprivileged to build cache images (its
+embedded buildah creates a user namespace) requires unprivileged user
+namespaces, which Ubuntu restricts by default via AppArmor. This is only needed
+at runtime — compiling with `make build` does not need it. There are two ways to
+allow it.
+
+**Preferred — scoped AppArmor profile.** Grant the `userns` permission only to
+the `mcv` binary, leaving the global restriction in place for every other
+program. Create `/etc/apparmor.d/mcv` (adjust the path to match your installed
+binary):
+
+```bash
+abi <abi/4.0>,
+include <tunables/global>
+
+profile mcv /home/<user>/go/bin/mcv flags=(unconfined) {
+  userns,
+  include if exists <local/mcv>
+}
+```
+
+Then load it:
+
+```bash
+sudo apparmor_parser -r /etc/apparmor.d/mcv
+```
+
+When running MCV inside a container instead of natively, apply the same `userns`
+permission to the container runtime's profile (e.g. `podman`/`rootlesskit`)
+rather than to `mcv`.
+
+**Simpler but less secure — disable the restriction globally.** This re-enables
+unprivileged user namespaces for *all* programs, weakening a defense-in-depth
+protection against kernel exploits that abuse user namespaces. Prefer the scoped
+profile above; use this only on disposable/dev machines:
+
+```bash
+sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0
+
+# To persist across reboots:
+echo 'kernel.apparmor_restrict_unprivileged_userns=0' | sudo tee /etc/sysctl.d/99-userns.conf
+ ```
+
+### Build and Install
 
 Build the binary:
 
@@ -110,6 +158,7 @@ Two image variants are available:
 **How it works:** With `--no-gpu`, MCV extracts GPU information (backend, architecture, warp size) from cache metadata rather than detecting actual hardware. The cache files created by vLLM/Triton already contain all necessary GPU information in environment variables.
 
 **GPU access flags** (e.g., `--gpus all` for NVIDIA, `--device /dev/kfd --device /dev/dri` for AMD) are **ONLY** required for GPU validation/preflight checks. They are **NOT** needed when using `--no-gpu` for cache creation or extraction.
+When using `podman run --device`, `--group-add keep-groups` may be needed for device access. But `--group-add keep-groups` requires crun (not runc).
 
 For detailed usage examples, container configuration, GPU access requirements, and CI/CD integration, see [docs/no-gpu-usage.md](./docs/no-gpu-usage.md).
 
@@ -552,26 +601,55 @@ To use docker on the host with an MCV image, you need to mount the cache
 directory to the container and run the following command:
 
 ```bash
-docker run --rm -it --privileged \
-  -v <path-to-cache>/example:/example \
+docker run --rm -it \
+  --user $(id -u):$(id -g) \
+  --security-opt seccomp=unconfined \
+  --security-opt apparmor=unconfined \
+  -v <path-to-cache>/example:/example:Z \
   quay.io/gkm/mcv bash -lc '
     /mcv -c -i quay.io/gkm/vector-add-cache:rocm \
         -d /example/vector-add-cache-rocm --no-gpu &&
     buildah push containers-storage:quay.io/gkm/vector-add-cache:rocm \
         docker-archive:/example/vector-add-cache-rocm.tar:quay.io/gkm/vector-add-cache:rocm
   '
+WARN[2025-09-11 16:46:54] running newgidmap: exit status 1: newgidmap: write to gid_map failed: Operation not permitted
+WARN[2025-09-11 16:46:54] /usr/bin/newgidmap should be setgid or have filecaps setgid
+WARN[2025-09-11 16:46:54] Falling back to single mapping
+WARN[2025-09-11 16:46:54] Error running newuidmap: exit status 1: newuidmap: write to uid_map failed: Operation not permitted
+WARN[2025-09-11 16:46:54] Falling back to single mapping
 INFO[2025-09-11 16:46:54] Setting log level: info
 INFO[2025-09-11 16:46:54] Using buildah to build the image
 INFO[2025-09-11 16:46:54] Detected cache components: [triton]
 INFO[2025-09-11 16:46:55] Image built! 8ce4bc2e98abfa8c0a5a6f6046c1c7bc8ac09805ecb029427a995dc2897828f8
 INFO[2025-09-11 16:46:55] OCI image created successfully.
+WARN[0000] running newgidmap: exit status 1: newgidmap: write to gid_map failed: Operation not permitted
+WARN[0000] /usr/bin/newgidmap should be setgid or have filecaps setgid
+WARN[0000] Falling back to single mapping
+WARN[0000] Error running newuidmap: exit status 1: newuidmap: write to uid_map failed: Operation not permitted
+WARN[0000] Falling back to single mapping
 Getting image source signatures
 Copying blob 24b82d6fef87 done
 Copying config 8ce4bc2e98 done
 Writing manifest to image destination
 Storing signatures
 ```
 
+> **NOTE:** The Warnings are known and everything still works fine.
+> An include library is making a system call that it doesn't have permission for,
+> so it fails and falls back to another method that succeeds.
+>
+> **Security note — `seccomp=unconfined` / `apparmor=unconfined`.** MCV runs
+> buildah *inside* the container to assemble the OCI image, which needs
+> `mount`/`unshare`/`pivot_root` and user-namespace operations that Docker's
+> default seccomp and AppArmor profiles block for non-privileged containers.
+> Disabling both is a deliberate, security-reviewed fallback — it is far narrower
+> than `--privileged` (no added capabilities, host devices, or host namespaces),
+> and the blast radius is bounded: the container runs as a non-root user
+> (`--user`), performs a single packaging task, and is removed on exit (`--rm`).
+> To harden further, replace these flags with scoped seccomp and AppArmor
+> profiles that allow only buildah's required syscalls/operations, or run the
+> same command under **rootless Podman**, which needs neither flag.
+
 Then on host:
 
 ```bash
@@ -596,8 +674,8 @@ To use podman on the host with an MCV image, you need to mount the cache
 directory to the container and run the following command:
 
 ```bash
-podman run --rm -it --privileged \
-  -v <path-to-cache>/example:/example \
+podman run --rm -it \
+  -v <path-to-cache>/example:/example:Z,U \
   quay.io/gkm/mcv bash -lc '
     /mcv -c -i quay.io/gkm/vector-add-cache:rocm \
         -d /example/vector-add-cache-rocm --no-gpu &&

mcv/docs/no-gpu-usage.md

@@ -40,13 +40,17 @@ make build-image-mcv-no-gpu
 # or directly:
 podman build --target mcv-minimal -t quay.io/gkm/mcv:no-gpu -f mcv/images/Containerfile .
 
-# Use (MCV pushes directly to the registry; mount auth and fuse device for buildah)
+# Use — --create builds the image into the container's local containers-storage;
+# chain `buildah push` in the same run so it reaches the registry before --rm
+# removes the container (and its store). Mount registry auth for buildah (the
+# image runs as appuser, UID 1000, so mount it under that user's home).
 podman run --rm \
-  --device /dev/fuse \
-  -v /path/to/cache:/cache \
-  -v ${HOME}/.config/containers/auth.json:/run/containers/0/auth.json:ro \
+  -v /path/to/cache:/cache:ro \
+  -v ${HOME}/.config/containers/auth.json:/home/appuser/.config/containers/auth.json:ro \
+  --entrypoint sh \
   quay.io/gkm/mcv:no-gpu \
-  --create --image quay.io/myorg/cache:v1 --dir /cache --no-gpu
+  -c "/mcv --create --image quay.io/myorg/cache:v1 --dir /cache --no-gpu \
+      && buildah push quay.io/myorg/cache:v1"
 ```
 
 ### Unified Image (For GPU validation - NVIDIA + AMD)
@@ -61,17 +65,32 @@ make build-image-mcv
 # or directly:
 podman build --target mcv-unified -t quay.io/gkm/mcv:unified -f mcv/images/Containerfile .
 
-# Use with NVIDIA GPU
+# Use with NVIDIA GPU (device nodes are world-accessible; no group needed)
 podman run --rm --device nvidia.com/gpu=all \
-  -v /path/to/cache:/cache quay.io/gkm/mcv:unified \
+  -v /path/to/cache:/cache:U quay.io/gkm/mcv:unified \
   --extract --image quay.io/myorg/cache:v1 --dir /cache
 
 # Use with AMD GPU
-podman run --rm --device /dev/kfd --device /dev/dri \
-  -v /path/to/cache:/cache quay.io/gkm/mcv:unified \
+podman run --rm --device /dev/kfd --device /dev/dri --group-add keep-groups \
+  -v /path/to/cache:/cache:U quay.io/gkm/mcv:unified \
   --extract --image quay.io/myorg/cache:v1 --dir /cache
 ```
 
+On hosts with group-restricted GPU devices, the documented device mappings can make
+amd-smi or rocm-smi fail. Add a runtime-supported mapping, such as Podman
+`--group-add keep-groups` with crun or explicit host device GIDs, and run GPU preflight
+as appuser.
+
+**Writable mounts (rootless).** Extraction writes into the mounted directory as the
+image's non-root user, so a plain rootless bind mount can fail with `permission
+denied`. Append `:U` to every writable mount (e.g. `-v /cache:/cache:U`) so Podman
+chowns the mount to the container user; on SELinux-enforcing hosts also add `:Z`
+(private to this container) or `:z` (shared) — for example `:U,Z`. Note that `:U`
+recursively changes ownership of the host directory; if that is undesirable, use
+`--userns=keep-id` instead to map your host UID into the container so existing
+ownership works without a chown. Docker has no `:U` — use `--user $(id -u):$(id -g)`
+there. Read-only mounts (`--create`) do not need `:U`.
+
 ## Usage Examples
 
 ### Creating Cache Images (No GPU Required)
@@ -82,13 +101,16 @@ mcv --create --image quay.io/myorg/vllm-cache:v1 \
     --dir ~/.cache/vllm/torch_compile_cache \
     --no-gpu
 
-# In container (minimal image; MCV pushes to the registry directly)
+# In container (minimal image) — --create stores the image in the container's
+# local containers-storage; chain `buildah push` so it reaches the registry
+# before --rm removes the store. Mount registry auth under appuser's home.
 podman run --rm \
-  --device /dev/fuse \
   -v ~/.cache/vllm:/cache:ro \
-  -v ${HOME}/.config/containers/auth.json:/run/containers/0/auth.json:ro \
+  -v ${HOME}/.config/containers/auth.json:/home/appuser/.config/containers/auth.json:ro \
+  --entrypoint sh \
   quay.io/gkm/mcv:no-gpu \
-  --create --image quay.io/myorg/vllm-cache:v1 --dir /cache --no-gpu
+  -c "/mcv --create --image quay.io/myorg/vllm-cache:v1 --dir /cache --no-gpu \
+      && buildah push quay.io/myorg/vllm-cache:v1"
 ```
 
 ### Extracting Cache (No GPU Required)
@@ -101,7 +123,7 @@ mcv --extract --image quay.io/myorg/vllm-cache:v1 \
 
 # In container (minimal image)
 podman run --rm \
-  -v ~/.cache/vllm:/cache \
+  -v ~/.cache/vllm:/cache:U \
   quay.io/gkm/mcv:no-gpu \
   --extract --image quay.io/myorg/vllm-cache:v1 --dir /cache --no-gpu
 ```
@@ -116,8 +138,8 @@ mcv --extract --image quay.io/myorg/vllm-cache:v1 \
 
 # In container (full image)
 podman run --rm \
-  --device /dev/kfd --device /dev/dri \
-  -v ~/.cache/vllm:/cache \
+  --device /dev/kfd --device /dev/dri --group-add keep-groups \
+  -v ~/.cache/vllm:/cache:U \
   quay.io/gkm/mcv:unified \
   --extract --image quay.io/myorg/vllm-cache:v1 --dir /cache
 ```
@@ -130,7 +152,7 @@ mcv --check-compat --image quay.io/myorg/vllm-cache:v1
 
 # In container
 podman run --rm \
-  --device /dev/kfd --device /dev/dri \
+  --device /dev/kfd --device /dev/dri --group-add keep-groups \
   quay.io/gkm/mcv:unified \
   --check-compat --image quay.io/myorg/vllm-cache:v1
 ```
@@ -150,31 +172,52 @@ build-cache-image:
         # Run vLLM to generate cache
         python generate_cache.py
 
-    - name: Build cache OCI image
+    - name: Build and push cache OCI image
       run: |
-        podman run --rm --privileged \
+        # --create stores the image in the container's local containers-storage,
+        # so push it in the same run — the store is gone once --rm removes the
+        # container. Mount registry auth under appuser's home (image runs as UID 1000).
+        podman run --rm \
           -v $(pwd)/.cache/vllm:/cache:ro \
+          -v ${HOME}/.config/containers/auth.json:/home/appuser/.config/containers/auth.json:ro \
+          --entrypoint sh \
           quay.io/gkm/mcv:no-gpu \
-          --create --image quay.io/myorg/vllm-cache:${{ github.sha }} \
-          --dir /cache --no-gpu
-
-    - name: Push cache image
-      run: |
-        podman push quay.io/myorg/vllm-cache:${{ github.sha }}
+          -c "/mcv --create --image quay.io/myorg/vllm-cache:${{ github.sha }} \
+                --dir /cache --no-gpu \
+              && buildah push quay.io/myorg/vllm-cache:${{ github.sha }}"
 ```
 
 ## GPU Access Requirements
 
 ### When GPU Access Flags Are Required
 
-**NVIDIA GPUs:**
+**NVIDIA GPUs:** (device nodes are world-accessible — no group needed)
 - Docker: `--gpus all` (or `--gpus device=0,1` for specific devices)
 - Podman: `--device nvidia.com/gpu=all`
 - **ONLY needed for**: GPU validation/preflight checks (extract without `--no-gpu`, compatibility checks)
 - **NOT needed for**: Creating or extracting with `--no-gpu` flag
 
-**AMD GPUs:**
-- Docker & Podman: `--device /dev/kfd --device /dev/dri`
+**AMD GPUs:** (`/dev/kfd` and `/dev/dri/*` are group-owned — the non-root user must join the render/video group)
+- Podman (crun): `--device /dev/kfd --device /dev/dri --group-add keep-groups` (preserves the host user's render/video groups)
+- Docker: `--device /dev/kfd --device /dev/dri`, plus a `--group-add <gid>` for each mapped device node's owning group (`keep-groups` is Podman/crun-only). Device paths and GIDs are host-specific — `/dev/dri/renderD128` and the `video` group may be absent, and `/dev/kfd` and the various `/dev/dri/*` nodes can each be owned by a different GID — so **derive a GID for each node that actually exists** and pass only the ones you found (never a bare or empty `--group-add`). For example:
+  ```bash
+  # Collect the distinct owning GIDs of the GPU nodes present on this host.
+  gpu_gids=$(for d in /dev/kfd /dev/dri/renderD* /dev/dri/card*; do
+    [ -e "$d" ] && stat -c '%g' "$d"
+  done | sort -u)
+  [ -n "$gpu_gids" ] || { echo "no GPU device nodes found" >&2; exit 1; }
+
+  docker run --rm \
+    --user "$(id -u):$(id -g)" \
+    --device /dev/kfd --device /dev/dri \
+    $(printf -- '--group-add %s ' $gpu_gids) \
+    -v /path/to/cache:/cache \
+    quay.io/gkm/mcv:unified \
+    --extract --image quay.io/myorg/cache:v1 --dir /cache
+  ```
+  Docker has no `:U`; run as your host UID/GID with `--user "$(id -u):$(id -g)"`
+  and ensure the host cache directory is writable by that UID/GID.
+  Alternatively, require operators to supply the GIDs explicitly (e.g. `--group-add 44 --group-add 993`) when the device layout is known ahead of time.
 - **ONLY needed for**: GPU validation/preflight checks (extract without `--no-gpu`, compatibility checks)
 - **NOT needed for**: Creating or extracting with `--no-gpu` flag
 
@@ -187,16 +230,16 @@ podman run --rm \
   quay.io/gkm/mcv:no-gpu \
   --create --image quay.io/myorg/cache:v1 --dir /cache --no-gpu
 
-# NVIDIA GPU access required for validation
+# NVIDIA GPU access required for validation (world-accessible nodes; no group needed)
 podman run --rm --device nvidia.com/gpu=all \
-  -v /path/to/cache:/cache \
+  -v /path/to/cache:/cache:U \
   quay.io/gkm/mcv:unified \
   --extract --image quay.io/myorg/cache:v1 --dir /cache
 
 # AMD GPU access required for validation
 podman run --rm \
-  --device /dev/kfd --device /dev/dri \
-  -v /path/to/cache:/cache \
+  --device /dev/kfd --device /dev/dri --group-add keep-groups \
+  -v /path/to/cache:/cache:U \
   quay.io/gkm/mcv:unified \
   --extract --image quay.io/myorg/cache:v1 --dir /cache
 ```