[Bug]: failed to drop capabilities in youki exec

[stepancheg]

Bug Description

Youki exec fails with "failed to drop capabilities" where runc run (not exec) fails.

Steps to Reproduce

config.json lists all possible capabilities:

{
  "ociVersion": "1.2.0",
  "process": {
    "terminal": false,
    "user": {
      "uid": 0,
      "gid": 0,
      "umask": 18
    },
    "args": [
      "/bin/sh"
    ],
    "env": [
      "DEBIAN_FRONTEND=noninteractive",
      "HOSTNAME=hds-c2ftv93mtzsz",
      "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
    ],
    "cwd": "/",
    "capabilities": {
      "effective": [
        "CAP_CHOWN",
        "CAP_DAC_OVERRIDE",
        "CAP_DAC_READ_SEARCH",
        "CAP_FOWNER",
        "CAP_FSETID",
        "CAP_KILL",
        "CAP_SETGID",
        "CAP_SETUID",
        "CAP_SETPCAP",
        "CAP_LINUX_IMMUTABLE",
        "CAP_NET_BIND_SERVICE",
        "CAP_NET_BROADCAST",
        "CAP_NET_ADMIN",
        "CAP_NET_RAW",
        "CAP_IPC_LOCK",
        "CAP_IPC_OWNER",
        "CAP_SYS_MODULE",
        "CAP_SYS_RAWIO",
        "CAP_SYS_CHROOT",
        "CAP_SYS_PTRACE",
        "CAP_SYS_PACCT",
        "CAP_SYS_ADMIN",
        "CAP_SYS_BOOT",
        "CAP_SYS_NICE",
        "CAP_SYS_RESOURCE",
        "CAP_SYS_TIME",
        "CAP_SYS_TTY_CONFIG",
        "CAP_MKNOD",
        "CAP_LEASE",
        "CAP_AUDIT_WRITE",
        "CAP_AUDIT_CONTROL",
        "CAP_SETFCAP",
        "CAP_MAC_OVERRIDE",
        "CAP_MAC_ADMIN",
        "CAP_SYSLOG",
        "CAP_WAKE_ALARM",
        "CAP_BLOCK_SUSPEND",
        "CAP_AUDIT_READ",
        "CAP_PERFMON",
        "CAP_BPF",
        "CAP_CHECKPOINT_RESTORE"
      ],
      "inheritable": [
        "CAP_CHOWN",
        "CAP_DAC_OVERRIDE",
        "CAP_DAC_READ_SEARCH",
        "CAP_FOWNER",
        "CAP_FSETID",
        "CAP_KILL",
        "CAP_SETGID",
        "CAP_SETUID",
        "CAP_SETPCAP",
        "CAP_LINUX_IMMUTABLE",
        "CAP_NET_BIND_SERVICE",
        "CAP_NET_BROADCAST",
        "CAP_NET_ADMIN",
        "CAP_NET_RAW",
        "CAP_IPC_LOCK",
        "CAP_IPC_OWNER",
        "CAP_SYS_MODULE",
        "CAP_SYS_RAWIO",
        "CAP_SYS_CHROOT",
        "CAP_SYS_PTRACE",
        "CAP_SYS_PACCT",
        "CAP_SYS_ADMIN",
        "CAP_SYS_BOOT",
        "CAP_SYS_NICE",
        "CAP_SYS_RESOURCE",
        "CAP_SYS_TIME",
        "CAP_SYS_TTY_CONFIG",
        "CAP_MKNOD",
        "CAP_LEASE",
        "CAP_AUDIT_WRITE",
        "CAP_AUDIT_CONTROL",
        "CAP_SETFCAP",
        "CAP_MAC_OVERRIDE",
        "CAP_MAC_ADMIN",
        "CAP_SYSLOG",
        "CAP_WAKE_ALARM",
        "CAP_BLOCK_SUSPEND",
        "CAP_AUDIT_READ",
        "CAP_PERFMON",
        "CAP_BPF",
        "CAP_CHECKPOINT_RESTORE"
      ],
      "permitted": [
        "CAP_CHOWN",
        "CAP_DAC_OVERRIDE",
        "CAP_DAC_READ_SEARCH",
        "CAP_FOWNER",
        "CAP_FSETID",
        "CAP_KILL",
        "CAP_SETGID",
        "CAP_SETUID",
        "CAP_SETPCAP",
        "CAP_LINUX_IMMUTABLE",
        "CAP_NET_BIND_SERVICE",
        "CAP_NET_BROADCAST",
        "CAP_NET_ADMIN",
        "CAP_NET_RAW",
        "CAP_IPC_LOCK",
        "CAP_IPC_OWNER",
        "CAP_SYS_MODULE",
        "CAP_SYS_RAWIO",
        "CAP_SYS_CHROOT",
        "CAP_SYS_PTRACE",
        "CAP_SYS_PACCT",
        "CAP_SYS_ADMIN",
        "CAP_SYS_BOOT",
        "CAP_SYS_NICE",
        "CAP_SYS_RESOURCE",
        "CAP_SYS_TIME",
        "CAP_SYS_TTY_CONFIG",
        "CAP_MKNOD",
        "CAP_LEASE",
        "CAP_AUDIT_WRITE",
        "CAP_AUDIT_CONTROL",
        "CAP_SETFCAP",
        "CAP_MAC_OVERRIDE",
        "CAP_MAC_ADMIN",
        "CAP_SYSLOG",
        "CAP_WAKE_ALARM",
        "CAP_BLOCK_SUSPEND",
        "CAP_AUDIT_READ",
        "CAP_PERFMON",
        "CAP_BPF",
        "CAP_CHECKPOINT_RESTORE"
      ]
    }
  },
  "root": {
    "path": "rootfs"
  },
  "hostname": null,
  "mounts": [
    {
      "destination": "/proc",
      "type": "proc",
      "source": "proc",
      "options": [
        "nosuid",
        "noexec",
        "nodev"
      ]
    },
    {
      "destination": "/dev",
      "type": "tmpfs",
      "source": "tmpfs",
      "options": [
        "nosuid",
        "strictatime",
        "mode=755",
        "size=65536k"
      ]
    },
    {
      "destination": "/sys",
      "type": "sysfs",
      "source": "sysfs",
      "options": [
        "nosuid",
        "noexec",
        "nodev",
        "ro"
      ]
    },
    {
      "destination": "/dev/pts",
      "type": "devpts",
      "source": "devpts",
      "options": [
        "nosuid",
        "noexec",
        "newinstance",
        "ptmxmode=0666",
        "mode=0620",
        "gid=5"
      ]
    },
    {
      "destination": "/dev/shm",
      "type": "tmpfs",
      "source": "shm",
      "options": [
        "nosuid",
        "noexec",
        "nodev",
        "mode=1777",
        "size=67108864"
      ]
    },
    {
      "destination": "/dev/mqueue",
      "type": "mqueue",
      "source": "mqueue",
      "options": [
        "nosuid",
        "noexec",
        "nodev"
      ]
    },
    {
      "destination": "/sys/fs/cgroup",
      "type": "cgroup",
      "source": "cgroup",
      "options": [
        "rprivate",
        "nosuid",
        "noexec",
        "nodev",
        "relatime",
        "rw"
      ]
    }
  ],
  "linux": {
    "namespaces": [
      {
        "type": "pid"
      },
      {
        "type": "ipc"
      },
      {
        "type": "uts"
      },
      {
        "type": "mount"
      },
      {
        "type": "cgroup"
      }
    ],
    "cgroupsPath": "/ch-inner"
  }
}

Not sure what is the right behavior here. Anyway,

$ youki run xx

starts fine, but then

$ youki exec xx pwd
ERROR libcontainer::process::init::process: failed to drop capabilities err=SetCaps(CapsError("capset failure: Operation not permitted (os error 1)"))
ERROR libcontainer::process::container_intermediate_process: failed to initialize container process: failed syscall
ERROR libcontainer::process::container_main_process: failed to wait for init ready: exec process failed with error error in executing process : failed syscall
ERROR libcontainer::container::builder_impl: failed to run container process exec process failed with error error in executing process : failed syscall
ERROR youki: error in executing command: failed to create container: exec process failed with error error in executing process : failed syscall
exec failed : failed to create container: exec process failed with error error in executing process : failed syscall

But

$ runc run xx
ERRO[0000] runc run failed: unable to start container process: error during container init: unable to apply caps: can't apply capabilities: operation not permitted

Expectation

I presume that either

• container should not be able to start
• or if container started, then exec should be successful

System and Setup Info

youki version: 0.5.7
commit: 0.5.7-bd54457ba9de1629f21fe678687f433aec63e7c7

runc version 1.3.0
commit: v1.3.0-0-g4ca628d1
spec: 1.2.1
go: go1.24.7
libseccomp: 2.5.5

6.14.0-1015-gcp #16-Ubuntu
(running on GCE VM)

Additional Context

No response

[stepancheg]

Logs (added in #3436) show dropping bounding capabilities to {}. But bounding is unset. Could this be an issue?

DEBUG libcontainer::capabilities: reset all caps
DEBUG libcontainer::capabilities: dropping bounding capabilities to {}
DEBUG libcontainer::capabilities: dropping effective capabilities to {Setgid, IpcOwner, AuditWrite, Chown, Bpf, CheckpointRestore, BlockSuspend, NetRaw, Setfcap, SysPtrace, SysAdmin, NetAdmin, SysModule, MacOverride, DacReadSearch, Mknod, NetBroadcast, AuditControl, Setuid, IpcLock, Setpcap, Fowner, SysTime, SysNice, Syslog, SysBoot, Lease, SysRawio, SysPacct, LinuxImmutable, WakeAlarm, Kill, SysChroot, SysResource, AuditRead, SysTtyConfig, MacAdmin, Perfmon, NetBindService, Fsetid, DacOverride}
DEBUG libcontainer::capabilities: dropping permitted capabilities to {SysAdmin, NetBindService, AuditControl, SysTime, Fowner, CheckpointRestore, Bpf, Setuid, Lease, WakeAlarm, Chown, Mknod, IpcOwner, Fsetid, Setfcap, Setpcap, SysNice, SysChroot, DacReadSearch, Syslog, AuditRead, Setgid, SysRawio, Kill, IpcLock, SysBoot, SysPtrace, NetBroadcast, SysPacct, SysResource, LinuxImmutable, Perfmon, NetAdmin, SysTtyConfig, BlockSuspend, AuditWrite, SysModule, MacAdmin, MacOverride, NetRaw, DacOverride}
DEBUG libcontainer::capabilities: dropping inheritable capabilities to {BlockSuspend, SysTtyConfig, Chown, SysChroot, SysPacct, NetBindService, SysNice, DacOverride, Mknod, Syslog, Perfmon, Bpf, NetRaw, Setgid, NetAdmin, DacReadSearch, SysBoot, NetBroadcast, Fsetid, SysResource, Kill, IpcOwner, Lease, IpcLock, AuditRead, Setfcap, MacAdmin, SysModule, Fowner, Setpcap, Setuid, WakeAlarm, SysAdmin, SysPtrace, AuditControl, SysRawio, SysTime, CheckpointRestore, LinuxImmutable, AuditWrite, MacOverride}
ERROR libcontainer::process::init::process: failed to drop capabilities err=SetCaps(CapsError("capset failure: Operation not permitted (os error 1)"))

[stepancheg]

Yes, looks like it:

youki/crates/libcontainer/src/container/tenant_builder.rs

Lines 94 to 97 in b777886

	let bounding: SpecCapabilities = match spec_caps.bounding() {
	Some(bounding) => bounding.union(&caps).copied().collect(),
	None => SpecCapabilities::new().union(&caps).copied().collect(),
	};

here if bounding is unset, tenant builder resets it to empty.

[YJDoc2]

Hey, I don't think bounding caps itself here is an issue. checking your PR and the logs you shared above, it seems that bounding caps are in fact dropped correctly to empty, but setting ambient caps is failing. This can be due to either

1. because we set bounding to {}, we are having error setting ambient
2. the kernel / system setup you are running does not support user setting ambient caps

I doubt the case is 1 because we would also have has same problem for effective or permitted caps. I will request you to check these two things -

1. In youki run logs, see if there is any warn logs saying failed to set cap or like
2. if possible, can you run the runc with strace and see which syscall they are failing at and do same for youki and compare?

If you need any help with this, I can do that, or I can try doing this myself, but might not be quick. Once the exact problem is known, we can see if, what and how we want to fix it.

[stepancheg]

it seems that bounding caps are in fact dropped correctly to empty

AFAIU, unset is not the same as empty.

Spec is unclear about it unfortunately, need to check what runc does.

[utam0k]

@stepancheg Thanks for raising the issue. Do you want to take this on?

[stepancheg]

Probably not, sorry. We evaluated youki, and found too many issues with it. For most problems there are workarounds, but the issue #3437 is non-trivial to fix and it blocks us, so we switched back to runc.

[utam0k]

Probably not, sorry. We evaluated youki, and found too many issues with it. For most problems there are workarounds, but the issue #3437 is non-trivial to fix and it blocks us, so we switched back to runc.

@stepancheg Thank you for trying it out. May I let you know once the fix is complete?
We are currently in a phase where we want to listen to user feedback and experiment with various new features. We would be very happy if we could collaborate with you.