Skip to content

fix(mcv): switch to vfs storage and remove fuse-overlayfs dependencies - #173

Merged
Billy99 merged 1 commit into
redhat-et:mainfrom
cmagina:mcv-image-overlayfs-fix
Aug 20, 2026
Merged

fix(mcv): switch to vfs storage and remove fuse-overlayfs dependencies#173
Billy99 merged 1 commit into
redhat-et:mainfrom
cmagina:mcv-image-overlayfs-fix

Conversation

@cmagina

@cmagina cmagina commented Aug 18, 2026

Copy link
Copy Markdown
Collaborator

VFS storage driver avoids nested overlay-on-overlay issues when running buildah inside containers. Remove fuse-overlayfs/fuse3 packages as they are only needed for overlay storage driver.

Resolves "mkdir /io.triton.manifest: operation not permitted" errors when running MCV in Docker/Podman environments (common on Fedora where docker is podman).

VFS storage driver avoids nested overlay-on-overlay issues when running
buildah inside containers. Remove fuse-overlayfs/fuse3 packages as they
are only needed for overlay storage driver.

Resolves "mkdir /io.triton.manifest: operation not permitted" errors
when running MCV in Docker/Podman environments (common on Fedora where
docker is podman).

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Signed-off-by: Craig Magina <cmagina@redhat.com>
@cmagina
cmagina requested a review from maryamtahhan August 18, 2026 19:31
@cmagina cmagina self-assigned this Aug 18, 2026
@coderabbitai

coderabbitai Bot commented Aug 18, 2026

Copy link
Copy Markdown

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 46f03eaf-74df-4173-abf9-a21a532c5d85

📥 Commits

Reviewing files that changed from the base of the PR and between 135a404 and ecaa35d.

📒 Files selected for processing (1)
  • mcv/images/Containerfile

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.


📝 Walkthrough

Summary by CodeRabbit

  • Bug Fixes
    • Updated container storage configuration for improved compatibility and reliability.
    • Removed obsolete FUSE-related package requirements from container targets.

Walkthrough

The container targets remove FUSE storage dependencies and change container storage configuration from overlay with fuse-overlayfs to the vfs driver.

Changes

Container storage configuration

Layer / File(s) Summary
Replace FUSE-based storage configuration
mcv/images/Containerfile
The minimal, NVIDIA, and unified targets remove fuse-overlayfs and fuse3. Their storage configuration uses the vfs driver without a mount_program setting.

Estimated code review effort: 1 (Trivial) | ~3 minutes

Merge Risk: ⚪ Minimal · up to ecaa3

The PR switches MCV image storage to VFS and removes unnecessary fuse packages; no actionable merge-blocking risk remains after normal checks and review.

Suggested reviewers: maryamtahhan, billy99

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly summarizes the switch to VFS storage and removal of obsolete dependencies.
Description check ✅ Passed The description explains the VFS change, dependency removal, and the container runtime error it resolves.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@Billy99 Billy99 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@Billy99
Billy99 merged commit f0e65de into redhat-et:main Aug 20, 2026
8 checks passed
Billy99 added a commit to Billy99/GKM that referenced this pull request Aug 20, 2026
Problem:

MCV uses buildah internally to build/push OCI cache images. Buildah traditionally
requires `--privileged` because it needs to mount overlay filesystems and run as root.
This is a security concern in production Kubernetes and CI environments.

What changed in the Containerfile

1. Storage driver: FUSE/overlay → VFS (redhat-et#173)

  driver="vfs"

  FUSE-overlayfs requires either --privileged or CAP_SYS_ADMIN + /dev/fuse access.
  VFS is a naive copy-based storage driver that needs no special kernel capabilities.
  It's slower (copies instead of overlays), but MCV only builds small single-layer
  cache images, so the performance difference is negligible.

2. Non-root user (UID/GID 1000)

  ```
  RUN groupadd -g 1000 appgroup && \
      useradd -u 1000 -g appgroup -m -s /bin/bash appuser
  USER appuser
  ```

  Running as root inside the container is unnecessary and a security risk. The
  fixed UID/GID 1000 also makes volume mount permissions predictable.

  Runtime flags: Podman vs Docker

  Podman:
  `podman run -v /path/to/cache:/tests:Z,U <image> ...`

  - :Z — relabels the volume for SELinux (private to this container)
  - :U — remaps the volume ownership to match the in-container user (UID 1000).
    Podman runs rootless with user namespaces, so the host UID and container UID
    differ. :U bridges that gap so appuser can read/write the mount.

  You only need :Z,U on volumes the container needs to write to (the cache output
  directory). Read-only mounts like model files only need :Z (or :ro,Z).

  Docker:

  ```
  docker run --user $(id -u):$(id -g) \
      --security-opt seccomp=unconfined \
      --security-opt apparmor=unconfined \
      -v /path/to/cache:/tests:Z \
      <image> ...
  ```

  - `--user $(id -u):$(id -g)` — Docker doesn't have Podman's user-namespace
    remapping, so you explicitly run as your host UID/GID to match volume ownership.
    Without this, the container runs as UID 1000 (appuser) which may not own the
    host-side mount.
  - `--security-opt seccomp=unconfined` — buildah makes syscalls (like mount,
    unshare) that Docker's default seccomp profile blocks. Disabling seccomp allows
    these without granting full --privileged.
  - `--security-opt apparmor=unconfined` — on Ubuntu (base container), AppArmor's
    default Docker profile also blocks some of buildah's mount/namespace operations.
    Disabling it is the minimal escalation needed.
  - `:Z` — SELinux relabeling, same as Podman. No :U needed because --user handles
    ownership directly.

  Why the difference

  Podman is rootless-native — it uses user namespaces automatically, so :U remaps
  ownership transparently. Docker doesn't do user-namespace remapping by default,
  so you need --user to align UIDs, and --security-opt to relax seccomp/AppArmor
  enough for buildah's syscalls without going full --privileged.

  What you're NOT granting

  Neither approach uses --privileged. The container does NOT get:
  - Full device access
  - CAP_SYS_ADMIN
  - Host PID/network namespace
  - Write access to /dev, /proc, /sys

  It's a targeted relaxation: let buildah do its mounts and namespace operations,
  nothing more.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Signed-off-by: Billy McFall <22157057+Billy99@users.noreply.github.com>
Billy99 added a commit to Billy99/GKM that referenced this pull request Aug 24, 2026
Problem:

MCV uses buildah internally to build/push OCI cache images. Buildah traditionally
requires `--privileged` because it needs to mount overlay filesystems and run as root.
This is a security concern in production Kubernetes and CI environments.

What changed in the Containerfile

1. Storage driver: FUSE/overlay → VFS (redhat-et#173)

  driver="vfs"

  FUSE-overlayfs requires either --privileged or CAP_SYS_ADMIN + /dev/fuse access.
  VFS is a naive copy-based storage driver that needs no special kernel capabilities.
  It's slower (copies instead of overlays), but MCV only builds small single-layer
  cache images, so the performance difference is negligible.

2. Non-root user (UID/GID 1000)

  ```
  RUN groupadd -g 1000 appgroup && \
      useradd -u 1000 -g appgroup -m -s /bin/bash appuser
  USER appuser
  ```

  Running as root inside the container is unnecessary and a security risk. The
  fixed UID/GID 1000 also makes volume mount permissions predictable.

  Runtime flags: Podman vs Docker

  Podman:
  `podman run -v /path/to/cache:/tests:Z,U <image> ...`

  - :Z — relabels the volume for SELinux (private to this container)
  - :U — remaps the volume ownership to match the in-container user (UID 1000).
    Podman runs rootless with user namespaces, so the host UID and container UID
    differ. :U bridges that gap so appuser can read/write the mount.

  You only need :Z,U on volumes the container needs to write to (the cache output
  directory). Read-only mounts like model files only need :Z (or :ro,Z).

  Docker:

  ```
  docker run --user $(id -u):$(id -g) \
      --security-opt seccomp=unconfined \
      --security-opt apparmor=unconfined \
      -v /path/to/cache:/tests:Z \
      <image> ...
  ```

  - `--user $(id -u):$(id -g)` — Docker doesn't have Podman's user-namespace
    remapping, so you explicitly run as your host UID/GID to match volume ownership.
    Without this, the container runs as UID 1000 (appuser) which may not own the
    host-side mount.
  - `--security-opt seccomp=unconfined` — buildah makes syscalls (like mount,
    unshare) that Docker's default seccomp profile blocks. Disabling seccomp allows
    these without granting full --privileged.
  - `--security-opt apparmor=unconfined` — on Ubuntu (base container), AppArmor's
    default Docker profile also blocks some of buildah's mount/namespace operations.
    Disabling it is the minimal escalation needed.
  - `:Z` — SELinux relabeling, same as Podman. No :U needed because --user handles
    ownership directly.

  Why the difference

  Podman is rootless-native — it uses user namespaces automatically, so :U remaps
  ownership transparently. Docker doesn't do user-namespace remapping by default,
  so you need --user to align UIDs, and --security-opt to relax seccomp/AppArmor
  enough for buildah's syscalls without going full --privileged.

  What you're NOT granting

  Neither approach uses --privileged. The container does NOT get:
  - Full device access
  - CAP_SYS_ADMIN
  - Host PID/network namespace
  - Write access to /dev, /proc, /sys

  It's a targeted relaxation: let buildah do its mounts and namespace operations,
  nothing more.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Signed-off-by: Billy McFall <22157057+Billy99@users.noreply.github.com>
Billy99 added a commit to Billy99/GKM that referenced this pull request Aug 25, 2026
Problem:

MCV uses buildah internally to build/push OCI cache images. Buildah traditionally
requires `--privileged` because it needs to mount overlay filesystems and run as root.
This is a security concern in production Kubernetes and CI environments.

What changed in the Containerfile

1. Storage driver: FUSE/overlay → VFS (redhat-et#173)

  driver="vfs"

  FUSE-overlayfs requires either --privileged or CAP_SYS_ADMIN + /dev/fuse access.
  VFS is a naive copy-based storage driver that needs no special kernel capabilities.
  It's slower (copies instead of overlays), but MCV only builds small single-layer
  cache images, so the performance difference is negligible.

2. Non-root user (UID/GID 1000)

  ```
  RUN groupadd -g 1000 appgroup && \
      useradd -u 1000 -g appgroup -m -s /bin/bash appuser
  USER appuser
  ```

  Running as root inside the container is unnecessary and a security risk. The
  fixed UID/GID 1000 also makes volume mount permissions predictable.

  Runtime flags: Podman vs Docker

  Podman:
  `podman run -v /path/to/cache:/tests:Z,U <image> ...`

  - :Z — relabels the volume for SELinux (private to this container)
  - :U — remaps the volume ownership to match the in-container user (UID 1000).
    Podman runs rootless with user namespaces, so the host UID and container UID
    differ. :U bridges that gap so appuser can read/write the mount.

  You only need :Z,U on volumes the container needs to write to (the cache output
  directory). Read-only mounts like model files only need :Z (or :ro,Z).

  Docker:

  ```
  docker run --user $(id -u):$(id -g) \
      --security-opt seccomp=unconfined \
      --security-opt apparmor=unconfined \
      -v /path/to/cache:/tests:Z \
      <image> ...
  ```

  - `--user $(id -u):$(id -g)` — Docker doesn't have Podman's user-namespace
    remapping, so you explicitly run as your host UID/GID to match volume ownership.
    Without this, the container runs as UID 1000 (appuser) which may not own the
    host-side mount.
  - `--security-opt seccomp=unconfined` — buildah makes syscalls (like mount,
    unshare) that Docker's default seccomp profile blocks. Disabling seccomp allows
    these without granting full --privileged.
  - `--security-opt apparmor=unconfined` — on Ubuntu (base container), AppArmor's
    default Docker profile also blocks some of buildah's mount/namespace operations.
    Disabling it is the minimal escalation needed.
  - `:Z` — SELinux relabeling, same as Podman. No :U needed because --user handles
    ownership directly.

  Why the difference

  Podman is rootless-native — it uses user namespaces automatically, so :U remaps
  ownership transparently. Docker doesn't do user-namespace remapping by default,
  so you need --user to align UIDs, and --security-opt to relax seccomp/AppArmor
  enough for buildah's syscalls without going full --privileged.

  What you're NOT granting

  Neither approach uses --privileged. The container does NOT get:
  - Full device access
  - CAP_SYS_ADMIN
  - Host PID/network namespace
  - Write access to /dev, /proc, /sys

  It's a targeted relaxation: let buildah do its mounts and namespace operations,
  nothing more.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Signed-off-by: Billy McFall <22157057+Billy99@users.noreply.github.com>
Billy99 added a commit to Billy99/GKM that referenced this pull request Aug 25, 2026
Problem:

MCV uses buildah internally to build/push OCI cache images. Buildah traditionally
requires `--privileged` because it needs to mount overlay filesystems and run as root.
This is a security concern in production Kubernetes and CI environments.

What changed in the Containerfile

1. Storage driver: FUSE/overlay → VFS (redhat-et#173)

  driver="vfs"

  FUSE-overlayfs requires either --privileged or CAP_SYS_ADMIN + /dev/fuse access.
  VFS is a naive copy-based storage driver that needs no special kernel capabilities.
  It's slower (copies instead of overlays), but MCV only builds small single-layer
  cache images, so the performance difference is negligible.

2. Non-root user (UID/GID 1000)

  ```
  RUN groupadd -g 1000 appgroup && \
      useradd -u 1000 -g appgroup -m -s /bin/bash appuser
  USER appuser
  ```

  Running as root inside the container is unnecessary and a security risk. The
  fixed UID/GID 1000 also makes volume mount permissions predictable.

  Runtime flags: Podman vs Docker

  Podman:
  `podman run -v /path/to/cache:/tests:Z,U <image> ...`

  - :Z — relabels the volume for SELinux (private to this container)
  - :U — remaps the volume ownership to match the in-container user (UID 1000).
    Podman runs rootless with user namespaces, so the host UID and container UID
    differ. :U bridges that gap so appuser can read/write the mount.

  You only need :Z,U on volumes the container needs to write to (the cache output
  directory). Read-only mounts like model files only need :Z (or :ro,Z).

  Docker:

  ```
  docker run --user $(id -u):$(id -g) \
      --security-opt seccomp=unconfined \
      --security-opt apparmor=unconfined \
      -v /path/to/cache:/tests:Z \
      <image> ...
  ```

  - `--user $(id -u):$(id -g)` — Docker doesn't have Podman's user-namespace
    remapping, so you explicitly run as your host UID/GID to match volume ownership.
    Without this, the container runs as UID 1000 (appuser) which may not own the
    host-side mount.
  - `--security-opt seccomp=unconfined` — buildah makes syscalls (like mount,
    unshare) that Docker's default seccomp profile blocks. Disabling seccomp allows
    these without granting full --privileged.
  - `--security-opt apparmor=unconfined` — on Ubuntu (base container), AppArmor's
    default Docker profile also blocks some of buildah's mount/namespace operations.
    Disabling it is the minimal escalation needed.
  - `:Z` — SELinux relabeling, same as Podman. No :U needed because --user handles
    ownership directly.

  Why the difference

  Podman is rootless-native — it uses user namespaces automatically, so :U remaps
  ownership transparently. Docker doesn't do user-namespace remapping by default,
  so you need --user to align UIDs, and --security-opt to relax seccomp/AppArmor
  enough for buildah's syscalls without going full --privileged.

  What you're NOT granting

  Neither approach uses --privileged. The container does NOT get:
  - Full device access
  - CAP_SYS_ADMIN
  - Host PID/network namespace
  - Write access to /dev, /proc, /sys

  It's a targeted relaxation: let buildah do its mounts and namespace operations,
  nothing more.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Signed-off-by: Billy McFall <22157057+Billy99@users.noreply.github.com>
Billy99 added a commit to Billy99/GKM that referenced this pull request Aug 25, 2026
Problem:

MCV uses buildah internally to build/push OCI cache images. Buildah traditionally
requires `--privileged` because it needs to mount overlay filesystems and run as root.
This is a security concern in production Kubernetes and CI environments.

What changed in the Containerfile

1. Storage driver: FUSE/overlay → VFS (redhat-et#173)

  driver="vfs"

  FUSE-overlayfs requires either --privileged or CAP_SYS_ADMIN + /dev/fuse access.
  VFS is a naive copy-based storage driver that needs no special kernel capabilities.
  It's slower (copies instead of overlays), but MCV only builds small single-layer
  cache images, so the performance difference is negligible.

2. Non-root user (UID/GID 1000)

  ```
  RUN groupadd -g 1000 appgroup && \
      useradd -u 1000 -g appgroup -m -s /bin/bash appuser
  USER appuser
  ```

  Running as root inside the container is unnecessary and a security risk. The
  fixed UID/GID 1000 also makes volume mount permissions predictable.

  Runtime flags: Podman vs Docker

  Podman:
  `podman run -v /path/to/cache:/tests:Z,U <image> ...`

  - :Z — relabels the volume for SELinux (private to this container)
  - :U — remaps the volume ownership to match the in-container user (UID 1000).
    Podman runs rootless with user namespaces, so the host UID and container UID
    differ. :U bridges that gap so appuser can read/write the mount.

  You only need :Z,U on volumes the container needs to write to (the cache output
  directory). Read-only mounts like model files only need :Z (or :ro,Z).

  Docker:

  ```
  docker run --user $(id -u):$(id -g) \
      --security-opt seccomp=unconfined \
      --security-opt apparmor=unconfined \
      -v /path/to/cache:/tests:Z \
      <image> ...
  ```

  - `--user $(id -u):$(id -g)` — Docker doesn't have Podman's user-namespace
    remapping, so you explicitly run as your host UID/GID to match volume ownership.
    Without this, the container runs as UID 1000 (appuser) which may not own the
    host-side mount.
  - `--security-opt seccomp=unconfined` — buildah makes syscalls (like mount,
    unshare) that Docker's default seccomp profile blocks. Disabling seccomp allows
    these without granting full --privileged.
  - `--security-opt apparmor=unconfined` — on Ubuntu (base container), AppArmor's
    default Docker profile also blocks some of buildah's mount/namespace operations.
    Disabling it is the minimal escalation needed.
  - `:Z` — SELinux relabeling, same as Podman. No :U needed because --user handles
    ownership directly.

  Why the difference

  Podman is rootless-native — it uses user namespaces automatically, so :U remaps
  ownership transparently. Docker doesn't do user-namespace remapping by default,
  so you need --user to align UIDs, and --security-opt to relax seccomp/AppArmor
  enough for buildah's syscalls without going full --privileged.

  What you're NOT granting

  Neither approach uses --privileged. The container does NOT get:
  - Full device access
  - CAP_SYS_ADMIN
  - Host PID/network namespace
  - Write access to /dev, /proc, /sys

  It's a targeted relaxation: let buildah do its mounts and namespace operations,
  nothing more.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Signed-off-by: Billy McFall <22157057+Billy99@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants