permission denied while creating rootless podman container

[cdeepali]

I am trying to using rootless podman, but I am unable to start container with it.

The error that I get is:
Failed to add conmon to cgroupfs sandbox cgroup: creating cgroup for blkio: mkdir /sys/fs/cgroup/blkio/conmon: permission denied

and podman info | cgroup shows the following:

  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1

podman version: 4.9.4
conmon verson: 2.1.10

[baude]

it might help the conversation if you cite the distribution and other relevant details, especially for this error.

[cdeepali]

System details: RHEL 9.4
Arch - x86_64

[giuseppe]

rootless podman won't attempt to use cgroups with cgroup v1 unless it is asked to do so.

What is the command you are using? In what environment are you running the command (sudo, local user session, ssh, ...)?

[cdeepali]

When we login as root user, we are able to pull images and spin containers.

however when logged in as normal (local) user, I am able to pull image (same image as used in case of root), but unable to spin container. While creating container it gives error:

DEBU[0001] ExitCode msg: "container create failed (no logs from conmon): conmon bytes \"\": readobjectstart: expect { or n, but found \x00, error found in #0 byte of ...||..., bigger context ...||..."
Error: container create failed (no logs from conmon): conmon bytes "": readObjectStart: expect { or n, but found , error found in #0 byte of ...||..., bigger context ...||...
DEBU[0001] Shutting down engines

[giuseppe]

that use case works with the default configuration, so it depends on your system and the way it is configured.

You are not providing enough information to understand what is going on. How did you login with the unprivileged user? Can you show the full command line? Have you performed any customization to the unprivileged user config files?

[cdeepali]

I have added my user with useradd command:

sudo useradd -s /bin/bash -m -d /home/deepali deepali

and then I ssh to the machine using my user.

[cdeepali]

podman pull docker.io/nvidia/cuda:12.6.3-devel-ubi9
Trying to pull docker.io/nvidia/cuda:12.6.3-devel-ubi9...
Getting image source signatures
Copying blob 18761c2a6242 done   |
Copying blob 32ebb53b046b done   |
Copying blob b01410969803 done   |
Copying blob 0a73aee7e570 skipped: already exists
Copying blob c471e60df11b done   |
Copying blob d1626b5b9f7a skipped: already exists
Copying blob 4cae7acd3be1 done   |
Copying blob e1b482bd8ff2 done   |
Copying blob d9457b301242 done   |
Copying blob 475a15c266ad done   |
Copying blob 1dd5e0013585 done   |
Copying config aaf54081e0 done   |
Writing manifest to image destination
aaf54081e0faa0be7107d3fc1179257fb3cdabd96f43972c921e01b2e902f827

 podman run -it --name mycontainer aaf54081e0faa0be7107d3fc1179257fb3cdabd96f43972c921e01b2e902f827 bash
Error: container create failed (no logs from conmon): conmon bytes "": readObjectStart: expect { or n, but found , error found in #0 byte of ...||..., bigger context ...||...

[giuseppe]

thanks, can you please run the same command with --log-level debug to podman and trying once without passing -it? It usually gives better error messages

[cdeepali]

Sure, it shows permission denied.

INFO[0000] Running conmon under slice user.slice and unitName libpod-conmon-f1df9929d20702169c376ee27db0d756f26d8938c539819cc007e9b9a916a91e.scope
[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied

DEBU[0000] Received: -1
DEBU[0000] Cleaning up container f1df9929d20702169c376ee27db0d756f26d8938c539819cc007e9b9a916a91e
DEBU[0000] Tearing down network namespace at /run/user/1002/netns/netns-8f22b959-42ba-f93d-f852-bd8875f60dbd for container f1df9929d20702169c376ee27db0d756f26d8938c539819cc007e9b9a916a91e
DEBU[0001] Unmounted container "f1df9929d20702169c376ee27db0d756f26d8938c539819cc007e9b9a916a91e"

[giuseppe]

do you get anything different if you run systemd-run --scope --user podman run -it --name mycontainer aaf54081e0faa0be7107d3fc1179257fb3cdabd96f43972c921e01b2e902f827 bash ?

[cdeepali]

systemd-run --scope --user  podman run -it --name mycontainer aaf54081e0faa0be7107d3fc1179257fb3cdabd96f43972c921e01b2e902f827 bash
Running scope as unit: run-r04e585f2262648c6aab3e88f7f37330d.scope
Error: runc: nvidia-container-cli: mount error: failed to add device rules: unable to find any existing device filters attached to the cgroup: bpf_prog_query(BPF_CGROUP_DEVICE) failed: operation not permitted: OCI permission denied

[giuseppe]

that error is coming from the nvidia runtime, it is not generated by Podman, so please report it to the appropriate project

[cdeepali]

but same error does not occur with root user. So I was thinking if it has to do with some configuration.

[cdeepali]

Thanks @giuseppe I will report the issue in nvidia project.

[giuseppe]

Probably you can't access the device without root privileges

[cdeepali]

Yes you are correct, the issue finally got resolved with the solution provided at NVIDIA/nvidia-container-toolkit#139 (comment)..

Thank you so much for your help on this @giuseppe