Build image : operation not permitted

[Mathie01]

Description

With this Dockerfile :

FROM docker.io/nginx:1.23.3-alpine

RUN apk add --update --no-cache nodejs npm libc6-compat

I get the following error:

STEP 2/14: RUN apk add --update --no-cache nodejs npm libc6-compat
error running subprocess: creating new mount namespace for [/bin/sh -c apk add --update --no-cache nodejs npm libc6-compat]: operation not permitted
Error: building at STEP "RUN apk add --update --no-cache nodejs npm libc6-compat": exit status 1

Steps to reproduce the issue:

1. Get the above Dockerfile
2. Use buildah build command

Describe the results you received:

I get the following error:

STEP 2/14: RUN apk add --update --no-cache nodejs npm libc6-compat
error running subprocess: creating new mount namespace for [/bin/sh -c apk add --update --no-cache nodejs npm libc6-compat]: operation not permitted
Error: building at STEP "RUN apk add --update --no-cache nodejs npm libc6-compat": exit status 1

Describe the results you expected:

I am waiting for the image to build

Output of rpm -q buildah or apt list buildah:

I can't, I use Gitlab CI / CD for this

Output of buildah version:

I use Gitlab CI / CD with buildah image

v1.28

My Gitlab CI / CD configuration :

docker_build:
  stage: docker_build
  image: quay.io/buildah/stable:v1.28
  variables:
    # Use vfs with buildah. Docker offers overlayfs as a default, but buildah
    # cannot stack overlayfs on top of another overlayfs filesystem.
    STORAGE_DRIVER: vfs
    # Write all image metadata in the docker format, not the standard OCI format.
    # Newer versions of docker can handle the OCI format, but older versions, like
    # the one shipped with Fedora 30, cannot handle the format.
    BUILDAH_FORMAT: docker
    # You may need this workaround for some errors: https://stackoverflow.com/a/70438141/1233435
    BUILDAH_ISOLATION: chroot
  before_script:
    - buildah login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
  script:
    - buildah build -t frontend:$CI_COMMIT_SHA -f Dockerfile.dashboard
    - buildah push frontend:$CI_COMMIT_TAG

[rhatdan]

Could you try with --isolation=chroot?

[Mathie01]

Hello !

Same thing :

[rhatdan]

It looks like you are running buildah within a confined environment, while docker is probably leaking the docker.sock into the container and using it.

[Mathie01]

I can't find any information on the Internet: it seems that nobody builds his image with Gitlab and Buildah

[MartinX3]

I've got the same problem.
We use buildah as build agent in jenkins in kubernetes and since 1.28 I can't use anymore RUN addgroup -g 1001 -S appuser in the dockerfile to build the image
error running subprocess: creating new mount namespace for [/bin/sh -c addgroup -g 1001 -S appuser]: operation not permitted

[rhatdan]

@flouthoc PTAL

[slonopotamus]

Experiencing the same issue. Unable to RUN anything when running buildah v1.28.0 inside Docker container:

error running subprocess: creating new mount namespace for [/bin/sh -c dotnet tool install -g dotnet-trace]: operation not permitted
Error: building at STEP "RUN dotnet tool install -g dotnet-trace": exit status 1

Note that neither 1.28.3 nor 1.29.0 are available via https://quay.io/repository/buildah/stable so I am not sure how to test them.

[flouthoc]

I think issue is more with the version v1.28.0, I'd request to bump buildah version please see comment here: #4553 (comment)

[flouthoc]

@slonopotamus Please try with quay.io/buildah/upstream:latest

[slonopotamus]

quay.io/buildah/upstream:latest fails in a much worse way:

Error during unshare(CLONE_NEWUSER): Operation not permitted
ERRO[0000] parsing PID "": strconv.Atoi: parsing "": invalid syntax 
ERRO[0000] (Unable to determine exit status)

Note that this happens even with --isolation=chroot. I'm using Docker 20.10.22.

More tests

quay.io/buildah/stable:v1.28

$ docker run --rm -it quay.io/buildah/stable:v1.28 buildah unshare echo hi
hi

quay.io/buildah/upstream:latest

$ docker run --rm -it quay.io/buildah/upstream:latest buildah unshare echo hi
Error during unshare(CLONE_NEWUSER): Operation not permitted
ERRO[0000] parsing PID "": strconv.Atoi: parsing "": invalid syntax 
ERRO[0000] (Unable to determine exit status)

[flouthoc]

@slonopotamus Could you try

[fl@fedora ~]$ sudo docker run --privileged --user 1000 --rm -it quay.io/buildah/upstream:latest buildah unshare echo hi

[slonopotamus]

It works both with and without sudo.

$ docker run --privileged --user 1000 --rm -it quay.io/buildah/upstream:latest buildah unshare echo hi
hi

$ su -c "docker run --privileged --user 1000 --rm -it quay.io/buildah/upstream:latest buildah unshare echo hi"
Password: 
hi

Obviously, this is not a way to go due because of security reasons.

[slonopotamus]

So, I'm not sure we're talking about a single issue here. I see that 1.28.0 works better than 1.27.0 or 1.29.0 in some aspect (only 1.28.0 passes buildah unshare echo hi inside unprivileged Docker container). But even 1.28.0 fails to RUN anything in Dockerfile.

[slonopotamus]

Okay, I know why buildah unshare echo hi works in 1.28.0 and doesn't work on 1.29.0 (and main branch). It was broken by containers/storage#1415.

[giuseppe]

how were you able to run nested containers without CAP_SYS_ADMIN and without the possibility to create a new user namespace? Were you using buildah run?