Podman : no CPU stats, no groups

[t4nki]

🔍 Check for existing issues

• Completed

How is Dozzle deployed?

Standalone Deployment

📦 Dozzle version

8.10.4

✅ Command used to run Dozzle

Dozzle is deployed through Ansible:

- name: Déploiement de Dozzle
  containers.podman.podman_container:
    name: dozzle
    image: docker.io/amir20/dozzle:latest
    label:
      io.containers.autoupdate=registry
      PODMAN_SYSTEMD_UNIT=pod_metrics.service
      diun.enable=true
      traefik.enable=true
      "traefik.http.routers.dozzle.rule=Host(`logs.{{ ansible_hostname }}.{{ ansible_domain}}`)"
      traefik.http.routers.dozzle.entrypoints=websecure
      traefik.http.services.dozzle.loadbalancer.server.port=9000
      traefik.http.routers.dozzle.middlewares=authelia@docker
      traefik.http.routers.dozzle.tls='true'
      dev.dozzle.group=metrics
    privileged: true
    env:
      TZ: 'Europe/Paris'
      DOZZLE_HOSTNAME: "{{ ansible_hostname }}.{{ ansible_domain }}"
      DOZZLE_ADDR: ":9000"
      DOZZLE_ENABLE_ACTIONS: true
      DOZZLE_FILTER: label=diun.enable
      DOZZLE_AUTH_PROVIDER: forward-proxy
    volumes:
      - /run/podman/podman.sock:/var/run/docker.sock
    state: present
    restart_policy: "on-failure:5"
    pod: "metrics"

🐛 Describe the bug / provide steps to reproduce it

The CPU average is always 0 be it global or containers
Also, I group my containers in pods (my main raison for using podman BTW) I tried using dev.dozzle.group label but i does not seem to be honored by dozzle
but, by accessing /group/<my_group> it works fine :)
Could you take a look at this, it's not a critical issue, it's just that it could be useful
Other than that, dozzle works like a charm and it's a great piece of software 👍

💻 Environment

host:
  arch: arm64
  buildahVersion: 1.28.2
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon_2.1.6+ds1-1_arm64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.6, commit: unknown'
  cpuUtilization:
    idlePercent: 82.56
    systemPercent: 2.23
    userPercent: 15.22
  cpus: 4
  distribution:
    codename: bookworm
    distribution: debian
    version: "12"
  eventLogger: journald
  hostname: rpi5
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.6.62+rpt-rpi-2712
  linkmode: dynamic
  logDriver: journald
  memFree: 12245450752
  memTotal: 17008033792
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun_1.8.1-1+deb12u1_arm64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.1
      commit: f8a096be060b22ccd3d5f3ebe44108517fbf6c30
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_1.2.0-1_arm64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 536854528
  swapTotal: 536854528
  uptime: 2h 25m 48.00s (Approximately 0.08 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries: {}
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 30
    paused: 0
    running: 30
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 30834601984
  graphRootUsed: 10774724608
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 25
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.3.1
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.19.8
  Os: linux
  OsArch: linux/arm64
  Version: 4.3.1

📸 If applicable, add screenshots to help explain your bug

No response

📜 If applicable, attach your Dozzle logs. You many need to enable debug mode. See https://dozzle.dev/guide/debugging.

No response

[amir20]

I see you are on rpi. Seems to be related to https://dozzle.dev/guide/faq#i-installed-dozzle-but-memory-consumption-doesn-t-show-up

[t4nki]

well funily enough, memory consumption shows up albeit it takes some time to be processed :

[amir20]

ah then I don't know 🫤 I don't have RPI handy to test it. Does docker stats work?

[t4nki]

I use podman and podman stats works fine
I'm willing to help you debug these issues
I have several rpis available for testing

[amir20]

I use Docker API directly. So if you could use the Docker CLI and test with it then that's the best way to know what is happening. I have deployed Podman here and there but it has always been a pain on Mac so I just use Orbstack now.

Let me know what you find.

[t4nki]

I will try with docker and a few containers to see if the issue is related to podman or raspbian

[t4nki]

Okay, I tried Dozzle with docker and cpu and ram stats are correctly displayed
To help you debugging, here are the stats from docker and podman, respectively, that I got while querying both sockets

Docker query : curl --unix-socket /var/run/docker.sock http://localhost/containers/dozzle/stats?stream=false
Docker response :

{
  "read": "2025-01-15T19:46:33.654557292Z",
  "preread": "2025-01-15T19:46:32.649839502Z",
  "pids_stats": {
    "current": 10,
    "limit": 19372
  },
  "blkio_stats": {
    "io_service_bytes_recursive": null,
    "io_serviced_recursive": null,
    "io_queue_recursive": null,
    "io_service_time_recursive": null,
    "io_wait_time_recursive": null,
    "io_merged_recursive": null,
    "io_time_recursive": null,
    "sectors_recursive": null
  },
  "num_procs": 0,
  "storage_stats": {},
  "cpu_stats": {
    "cpu_usage": {
      "total_usage": 1179210000,
      "usage_in_kernelmode": 389300000,
      "usage_in_usermode": 789910000
    },
    "system_cpu_usage": 340662000000000,
    "online_cpus": 4,
    "throttling_data": {
      "periods": 0,
      "throttled_periods": 0,
      "throttled_time": 0
    }
  },
  "precpu_stats": {
    "cpu_usage": {
      "total_usage": 1178139000,
      "usage_in_kernelmode": 389300000,
      "usage_in_usermode": 788839000
    },
    "system_cpu_usage": 340657990000000,
    "online_cpus": 4,
    "throttling_data": {
      "periods": 0,
      "throttled_periods": 0,
      "throttled_time": 0
    }
  },
  "memory_stats": {
    "usage": 10108928,
    "stats": {
      "active_anon": 9437184,
      "active_file": 0,
      "anon": 9437184,
      "anon_thp": 0,
      "file": 0,
      "file_dirty": 0,
      "file_mapped": 0,
      "file_writeback": 0,
      "inactive_anon": 0,
      "inactive_file": 0,
      "kernel_stack": 163840,
      "pgactivate": 0,
      "pgdeactivate": 0,
      "pgfault": 1533,
      "pglazyfree": 0,
      "pglazyfreed": 0,
      "pgmajfault": 0,
      "pgrefill": 0,
      "pgscan": 0,
      "pgsteal": 0,
      "shmem": 0,
      "slab": 256664,
      "slab_reclaimable": 71008,
      "slab_unreclaimable": 185656,
      "sock": 0,
      "thp_collapse_alloc": 0,
      "thp_fault_alloc": 0,
      "unevictable": 0,
      "workingset_activate": 0,
      "workingset_nodereclaim": 0,
      "workingset_refault": 0
    },
    "limit": 17008033792
  },
  "name": "/dozzle",
  "id": "fcff1902ebf9ff113c8c1c8168ad13fa127c742a3a633739911f2a2fa19d5627",
  "networks": {
    "eth0": {
      "rx_bytes": 209666,
      "rx_packets": 3452,
      "rx_errors": 0,
      "rx_dropped": 0,
      "tx_bytes": 1754576,
      "tx_packets": 3144,
      "tx_errors": 0,
      "tx_dropped": 0
    }
  }
}

Podman query : curl --unix-socket /run/podman/podman.sock -s http://localhost/v1.0.0/containers/dozzle/stats?stream=false
Podman response :

{
  "read": "2025-01-15T20:50:06.12877464+01:00",
  "preread": "0001-01-01T00:00:00Z",
  "pids_stats": {
    "current": 11
  },
  "blkio_stats": {
    "io_service_bytes_recursive": [
      {
        "major": 0,
        "minor": 8,
        "op": "read",
        "value": 13443072
      },
      {
        "major": 0,
        "minor": 8,
        "op": "write",
        "value": 0
      },
      {
        "major": 0,
        "minor": 8,
        "op": "rios",
        "value": 458
      },
      {
        "major": 0,
        "minor": 8,
        "op": "wios",
        "value": 0
      },
      {
        "major": 0,
        "minor": 8,
        "op": "dbytes",
        "value": 0
      },
      {
        "major": 0,
        "minor": 8,
        "op": "dios",
        "value": 0
      }
    ],
    "io_serviced_recursive": null,
    "io_queue_recursive": null,
    "io_service_time_recursive": null,
    "io_wait_time_recursive": null,
    "io_merged_recursive": null,
    "io_time_recursive": null,
    "sectors_recursive": null
  },
  "num_procs": 0,
  "storage_stats": {},
  "cpu_stats": {
    "cpu_usage": {
      "total_usage": 396043058000,
      "usage_in_kernelmode": 131908272,
      "usage_in_usermode": 395911149728
    },
    "system_cpu_usage": 93913262081000,
    "cpu": 0,
    "throttling_data": {
      "periods": 0,
      "throttled_periods": 0,
      "throttled_time": 0
    }
  },
  "precpu_stats": {
    "cpu_usage": {
      "total_usage": 0,
      "usage_in_kernelmode": 0,
      "usage_in_usermode": 0
    },
    "cpu": 0,
    "throttling_data": {
      "periods": 0,
      "throttled_periods": 0,
      "throttled_time": 0
    }
  },
  "memory_stats": {
    "usage": 16551936,
    "limit": 8236683264
  },
  "name": "dozzle",
  "Id": "31531c6b12a1fc46705b392e85f046aa7bf7735ea0fa9cf0f6b7a66e32c8bebe",
  "networks": {
    "network": {
      "rx_bytes": 108397197,
      "rx_packets": 0,
      "rx_errors": 0,
      "rx_dropped": 0,
      "tx_bytes": 801751318,
      "tx_packets": 0,
      "tx_errors": 0,
      "tx_dropped": 0
    }
  }
}

Regarding the version of each :
docker system info :

Client: Docker Engine - Community
 Version:    27.5.0
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.19.3
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.32.3
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 3
  Running: 3
  Paused: 0
  Stopped: 0
 Images: 4
 Server Version: 27.5.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: bcc810d6b9066471b0b6fa75f557a15a1cbf31bb
 runc version: v1.2.4-0-g6c52b3f
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.6.62+rpt-rpi-2712
 Operating System: Debian GNU/Linux 12 (bookworm)
 OSType: linux
 Architecture: aarch64
 CPUs: 4
 Total Memory: 15.84GiB
 Name: rpi5
 ID: d9728e62-7fc8-4901-b07b-4d83deb1b1a0

 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

podman system info :

host:
  arch: arm64
  buildahVersion: 1.28.2
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon_2.1.6+ds1-1_arm64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.6, commit: unknown'
  cpuUtilization:
    idlePercent: 93.95
    systemPercent: 2.08
    userPercent: 3.97
  cpus: 4
  distribution:
    codename: bookworm
    distribution: debian
    version: "12"
  eventLogger: journald
  hostname: mirage
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.1.0-29-arm64
  linkmode: dynamic
  logDriver: journald
  memFree: 258195456
  memTotal: 8236683264
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun_1.8.1-1+deb12u1_arm64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.1
      commit: f8a096be060b22ccd3d5f3ebe44108517fbf6c30
      rundir: /run/user/0/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 0
  swapTotal: 0
  uptime: 103h 47m 34.00s (Approximately 4.29 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries: {}
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 28
    paused: 0
    running: 28
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 251583135744
  graphRootUsed: 25522040832
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 23
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.3.1
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.19.8
  Os: linux
  OsArch: linux/arm64
  Version: 4.3.1

I hope all the details provided here make sense and will help you isolate the problem

Also, please apologize if anything I say is not clear, English is not my first language :)

[amir20]

Hey @t4nki so it seems like Podman is the culprit. Docker API returns the right data, while podman returns all zeros for CPU.

I don't think there is anything I can do on my side to fix it. There wasn't a lot either reported on this. I found this issue portainer/portainer#4818. I assume not a lot of people use RPI and Podman.

I think this is an unfortunate parts of using Podman some times.

[t4nki]

I guess dozzle tries to retrieve precpu{} values, am I right ?

[amir20]

Yes, see

dozzle/internal/docker/client.go

Lines 231 to 232 in 4cfd3d0

	previousCPU = v.PreCPUStats.CPUUsage.TotalUsage
	previousSystem = v.PreCPUStats.SystemUsage

[t4nki]

I see, let's hope this will be fixed in future versions, podman on Debian is pretty outdated anyway

Did you have time to have a look at the grouping bug I talked about in the first post ?

[amir20]

Going to move this to discussion since it's not a Dozzle bug.

Did you have time to have a look at the grouping bug I talked about in the first post ?

It should work as documented https://dozzle.dev/guide/container-groups#custom-groups

I am not sure what you mean by not working. I may have a bug because I changed custom groups recently but was working.

It works there with docker run -d --label dev.dozzle.group=test ...

[ivoshm]

I can confirm that in the Podman environment containers are not grouped in the user interface.
The /group/<group_name> URL still works.

I'm sure my configuration based on the 'dev.dozzle.group' label worked before, but it stopped working a few days ago.

I tried downgrading Dozzle to bi-sect the problem but I failed.

[amir20]

I think I know what the problem might be.

Do you ONLY have groups and nothing else?

[t4nki]

it has never worked for me, but, I'm a fresh dozzle user (for the last month I might say)
All the containers appear correctly in dozzle, but the groups are a no show, but I can still access them while changing the URL to https://<dozzle_host>/group/<target_group>

[amir20]

I think it's fixed here #3573 but I need someone to test it.

There is a new style of menus in Dozzle where you can horizontal scroll. Right now it is 1) hosts & containers, 2) custom groups and services.

I had a bug where it would only show up with services. I was thinking of moving custom groups to its own menu so that it would be:

1. host and containers
2. custom groups and containers
3. services and stacks

But I never got around to doing that and that's why the bug is there.

[amir20]

This is merged in master now.