Linux podman info - stat /bin/fuse-overlayfs: no such file or directory #12888

Constantin07 · 2021-12-15T19:45:50Z

Constantin07
Dec 15, 2021

/kind bug

Description

On CentOS7.9 with podman build from sources it fails to find fuse-overlayfs under /usr/local/bin. This happens when invoked as non-privileged user.
As root user it works just fine if I specify this config settings but for some reason this config is not being picked up when run by non-privileged user.

# cat /etc/containers/storage.conf
[storage]

driver = "overlay"

runroot = "/run/containers/storage"

graphroot = "/var/lib/containers/storage"

rootless_storage_path = "/var/lib/containers/storage"

[storage.options]
mount_program = "/usr/local/bin/fuse-overlayfs"

[storage.options.overlay]
ignore_chown_errors = "false"
mountopt = "nodev"

Note: The fuse-overlayfs v1.7.1 has been compiled from sources and installed in /usr/local/bin.
The PATH variable includes /usr/local/bin for both root and non-privileged user.

Steps to reproduce the issue:

Update $PATH to include /usr/local/bin
Make sure to have the fuse-overlayfs installed under /usr/local/bin:

# /usr/local/bin/fuse-overlayfs -V
fuse-overlayfs: version 1.7.1
FUSE library version 3.6.1
using FUSE kernel interface version 7.29
fusermount3 version: 3.6.1

Make sure to install podman v.3.4.4 under /usr/local/bin:
Run podman info:

Describe the results you received:

$ podman info
Error: overlay: can't stat program "/bin/fuse-overlayfs": stat /bin/fuse-overlayfs: no such file or directory

$ echo $PATH
/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin

$ ls -la /usr/local/bin/podman
-rwxr-xr-x 1 root root 52604016 Dec 15 19:10 /usr/local/bin/podman

$ ls -la /usr/local/bin/fuse-overlayfs
-rwxr-xr-x 1 root root 389096 Dec 15 08:06 /usr/local/bin/fuse-overlayfs

Describe the results you expected:

podman to work just fine and find the path to fuse-overlayfs either via $PATH env var. or the mount_program config in the /etc/containers/storage.conf file.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

podman version 3.4.4

Output of podman info --debug:

As non-privileged user:

Error: overlay: can't stat program "/bin/fuse-overlayfs": stat /bin/fuse-overlayfs: no such file or directory

If I run as root:

host:
  arch: amd64
  buildahVersion: 1.23.1
  cgroupControllers:
  - cpuset
  - cpu
  - cpuacct
  - blkio
  - memory
  - devices
  - freezer
  - net_cls
  - perf_event
  - net_prio
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: Unknown
    path: /usr/local/bin/conmon
    version: 'conmon version 2.0.31, commit: 7e7eb74e52abf65a6d46807eeaea75425cc8a36c'
  cpus: 2
  distribution:
    distribution: '"centos"'
    version: "7"
  eventLogger: file
  hostname: *******************
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.15.8-1.el7.elrepo.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 317079552
  memTotal: 4739059712
  ociRuntime:
    name: runc
    package: containerd.io-1.4.12-3.1.el7.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.2
      commit: v1.0.2-0-g52b36a2
      spec: 1.0.2-dev
      go: go1.16.10
      libseccomp: 2.3.1
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_AUDIT_WRITE,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_MKNOD,CAP_NET_BIND_SERVICE,CAP_NET_RAW,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: ""
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-0.4.3-4.el7_8.x86_64
    version: |-
      slirp4netns version 0.4.3
      commit: 2244b9b6461afeccad1678fac3d6e478c28b4ad6
  swapFree: 10174464
  swapTotal: 2147479552
  uptime: 6h 20m 47.84s (Approximately 0.25 days)
plugins:
  log:
  - k8s-file
  - none
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  ********************:
    Blocked: false
    Insecure: true
    Location: **************************
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: ****************:5000
  search:
  - docker.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.ignore_chown_errors: "false"
    overlay.mount_program:
      Executable: /usr/local/bin/fuse-overlayfs
      Package: Unknown
      Version: |-
        fusermount3 version: 3.6.1
        fuse-overlayfs: version 1.7.1
        FUSE library version 3.6.1
        using FUSE kernel interface version 7.29
    overlay.mountopt: nodev
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 13
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 3.4.4
  Built: 1639595345
  BuiltTime: Wed Dec 15 19:09:05 2021
  GitCommit: 8a94e1484b9c204457171e868d6b9401357427c3
  GoVersion: go1.17.4
  OsArch: linux/amd64
  Version: 3.4.4

Package info (e.g. output of rpm -q podman or apt list podman):

N/A because podman was build from sources.

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes. 4.0.0-dev

Additional environment details (AWS, VirtualBox, physical, etc.):

CentOS Linux release 7.9.2009 (Core)

giuseppe · 2021-12-16T08:54:02Z

giuseppe
Dec 16, 2021
Maintainer

does the user have a custom configuration file under ~/.config/containers that overrides the mount_program?

0 replies

Constantin07 · 2021-12-16T13:01:04Z

Constantin07
Dec 16, 2021
Author

You are right, @giuseppe I did found in ~/.config/containers a storage config file though not sure how it was created. Probably from old podman rpm package which I did install initially and then removed because it was very old.

The above file indeed had mount_program = "/bin/fuse-overlayfs". Now If I rename this file I get:

$ podman info
Error: error acquiring runtime init lock: error opening "/run/libpod/alive.lck": permission denied

Wasn't supposed podman to look into /etc/containers/storage.conf if user's config file is absent ? Reason: I don't want to create user's config file for every user on the system.
root user doesn't have ~/.config/containers and is able to use the "global" config file.

0 replies

rhatdan · 2021-12-16T13:07:53Z

rhatdan
Dec 16, 2021
Maintainer

Looks like XDG_RUNTIME_DIR is set incorrectly.

0 replies

giuseppe · 2021-12-16T14:44:48Z

giuseppe
Dec 16, 2021
Maintainer

if you don't override the configuration in ~/.config/containers then what is set in /etc/containers is used.

Older versions of Podman were always creating files under ~/.config. Can you try removing storage.conf as well?

0 replies

Constantin07 · 2021-12-16T18:34:44Z

Constantin07
Dec 16, 2021
Author

@rhatdan looks like that's the case. As root logged in on a box I get:

# echo $XDG_RUNTIME_DIR
/run/user/0

Now, when I do sudu su - jenkins I get that it's not set:

$ echo $XDG_RUNTIME_DIR

but I do get XDG_SESSION_ID=2 env. var set. This is a login/interactive shell.
Now I'm wondering if XDG_RUNTIME_DIR is supposed to be set when run as service account, e.g. running Jenkins with jenkins non-priviledged user ?

0 replies

Constantin07 · 2021-12-16T18:42:45Z

Constantin07
Dec 16, 2021
Author

@giuseppe I did remove entirely the ~/.config/containers directory for regular user (let's call it jenkins).
but unfortunately no luck - getting the same error:

$ podman info
Error: error acquiring runtime init lock: error opening "/run/libpod/alive.lck": permission denied

Here is my global config:

# ls -la /etc/containers
-rw-r--r--  1 root root   74 Dec  9 20:24 policy.json
-rw-r--r--  1 root root  132 Dec 16 13:08 registries.conf
-rw-r--r--  1 root root  308 Dec 15 00:32 storage.conf

# cat policy.json
{
  "default": [
    {
      "type": "insecureAcceptAnything"
    }
  ]
}

# cat storage.conf
[storage]

driver = "overlay"

runroot = "/run/containers/storage"

graphroot = "/var/lib/containers/storage"

rootless_storage_path = "/var/lib/containers/storage"

[storage.options]
mount_program = "/usr/local/bin/fuse-overlayfs"

[storage.options.overlay]
ignore_chown_errors = "false"
mountopt = "nodev"


# cat registries.conf
unqualified-search-registries = ['docker.io']

0 replies

Constantin07 · 2021-12-16T19:17:57Z

Constantin07
Dec 16, 2021
Author

defining XDG_RUNTIME_DIR env. var. doesn't help:

$ export XDG_RUNTIME_DIR=/run/user/$UID
$ echo $XDG_RUNTIME_DIR
/run/user/995
$ podman -v
podman version 4.0.0-dev
$ podman info
Error: error acquiring runtime init lock: error opening "/run/libpod/alive.lck": permission denied

0 replies

rhatdan · 2021-12-16T21:53:28Z

rhatdan
Dec 16, 2021
Maintainer

@mheon WDYT?

0 replies

mheon · 2021-12-17T14:31:49Z

mheon
Dec 17, 2021
Maintainer

Please remove ~/.config/containers again after ensuring that $XDG_RUNTIME_DIR is defined; Podman is using the old paths, cached in the DB, which contain an invalid tmp directory.

0 replies

Constantin07 · 2021-12-17T15:06:25Z

Constantin07
Dec 17, 2021
Author

@mheon I did remove it. But I have the same result.
I've observed that it does create an empty file ~/.config/cni/net.d/cni.lock.

Please note that I'm using podman only for building container images, not for running them.

The below error message is misleading: the file /run/libpod/alive.lck does exist, is empty and has read permission allowed for everyone. If it tries to open this file in write mode, probably it should be more explicit about this.

$ podman info
Error: error acquiring runtime init lock: error opening "/run/libpod/alive.lck": permission denied
-bash-4.2$ ls -la /run/libpod/alive.lck
-rw-r--r-- 1 root root 0 Dec 17 14:41 /run/libpod/alive.lck

0 replies

mheon · 2021-12-17T16:22:27Z

mheon
Dec 17, 2021
Maintainer

That directory should not be in use by rootless Podman. We should be using a directory in /run/user/$UID (or, failing that, a directory in /tmp unique to the user in question). That is the root Podman temporary files directory. It cannot be used by rootless Podman.

0 replies

Constantin07 · 2021-12-17T16:35:57Z

Constantin07
Dec 17, 2021
Author

Thanks @mheon for explaining, I'm wondering what other thing could make podman when run as privileged user to connect to too root resources ?

0 replies

2022-01-17T00:04:47Z

github-actions[bot]
bot Jan 17, 2022

A friendly reminder that this issue had no activity for 30 days.

0 replies

rhatdan · 2022-01-17T12:12:18Z

rhatdan
Jan 17, 2022
Maintainer

Since this is a setup issue on a non supported platform, moving this to discussion.

0 replies

bademux · 2025-05-31T18:04:04Z

bademux
May 31, 2025

@Constantin07 have you find the reason for that behavior?
I'm having similar issues on Openwrt.

@mheon What will be the best way to debug, why rootless podman "ignores" XDG_RUNTIME_DIR ?

2 replies

mheon May 31, 2025
Maintainer

Please file a new issue, this discussion is 3 years old. It's very difficult to help folks without more information on the problem they're encountering, that we can get from the issue template.

bademux May 31, 2025

yep, fully understand.
I just want to try dig into the problem by myself first.
Anyway, thanks for fast response!

Linux podman info - stat /bin/fuse-overlayfs: no such file or directory #12888

Uh oh!

Uh oh!

Constantin07 Dec 15, 2021

Replies: 15 comments · 2 replies

Uh oh!

giuseppe Dec 16, 2021 Maintainer

Uh oh!

Uh oh!

Constantin07 Dec 16, 2021 Author

Uh oh!

rhatdan Dec 16, 2021 Maintainer

Uh oh!

giuseppe Dec 16, 2021 Maintainer

Uh oh!

Constantin07 Dec 16, 2021 Author

Uh oh!

Uh oh!

Constantin07 Dec 16, 2021 Author

Uh oh!

Constantin07 Dec 16, 2021 Author

Uh oh!

rhatdan Dec 16, 2021 Maintainer

Uh oh!

mheon Dec 17, 2021 Maintainer

Uh oh!

Constantin07 Dec 17, 2021 Author

Uh oh!

mheon Dec 17, 2021 Maintainer

Uh oh!

Constantin07 Dec 17, 2021 Author

Uh oh!

github-actions[bot] bot Jan 17, 2022

Uh oh!

rhatdan Jan 17, 2022 Maintainer

Uh oh!

bademux May 31, 2025

Uh oh!

mheon May 31, 2025 Maintainer

Uh oh!

bademux May 31, 2025

Constantin07
Dec 15, 2021

Replies: 15 comments 2 replies

giuseppe
Dec 16, 2021
Maintainer

Constantin07
Dec 16, 2021
Author

rhatdan
Dec 16, 2021
Maintainer

giuseppe
Dec 16, 2021
Maintainer

Constantin07
Dec 16, 2021
Author

Constantin07
Dec 16, 2021
Author

Constantin07
Dec 16, 2021
Author

rhatdan
Dec 16, 2021
Maintainer

mheon
Dec 17, 2021
Maintainer

Constantin07
Dec 17, 2021
Author

mheon
Dec 17, 2021
Maintainer

Constantin07
Dec 17, 2021
Author

github-actions[bot]
bot Jan 17, 2022

rhatdan
Jan 17, 2022
Maintainer

bademux
May 31, 2025

mheon May 31, 2025
Maintainer