Problems running nanocl on Ubuntu wth podman

[izderadicka]

Describe the bug
When playing with nanocl on Ubuntu 24.04 I've encountered couple of issues:

1. It's related to existing Ubuntu bug:
   https://bugs.launchpad.net/ubuntu/+source/libpod/+bug/2040483 /
   This prevents podman and thus nanocl from stopping containers.
   Disabling apparmor heled.

2. ndns exists with this error:

dnsmasq: failed to create listening socket for 10.89.0.1: Address in use

It's because it is used by podman's own dns service:

$ sudo lsof -nP -iUDP
COMMAND    PID            USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
aardvark- 9305            root    9u  IPv4  84026      0t0  UDP 10.89.0.1:53

As described here https://www.redhat.com/sysadmin/podman-new-network-stack

3. After reboot of host majority of containers were gone - nanocld, ....
   podman does not have daemon by default - so restart of containers should be managed by systemd - not exactly sure why some containers restarted and some not . Did not look into details

While first issue is purely Canonical problem, second and third one I think will be more generic - I assume this will affect nanocl on other platforms too.

To Reproduce
Clean install of Ubuntu 24.04 and sudo apt install podman
Install nanocl as per it's documentation (must do sudo nanocl install because of right on podman socket in /var/run/podman/podman.sock")

failing ndns cn be seen via sudo podman ps -a

reboot system, and see what containers are still running

Expected behavior
podman should works same as docker

Desktop (please complete the following information):

• OS: Linux
• Distribution: Ubuntu
• Version 24.04

[leon3s]

Hey thanks to have took time to open a very detailed issue, i'll investigate and provide a fix!

[leon3s]

So i investigated why on system reboot container aren't restarting, and it's seems there is a problem with conmon this is the error i get:

Error: unable to start container "e4404aaae8a9fec8047942b6b7d9617edd6ff2b69b8220402b84df2e9d068d5d": container create failed (no logs from conmon): conmon bytes "": readObjectStart: expect { or n, but found , error found in #0 byte of ...||..., bigger context ...||...

I was able to fix it on a fresh ubuntu 24 install using this script:

It came from this issue: containers/podman#21024

wget https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_22.04/amd64/conmon_2.1.2~0_amd64.deb -O /tmp/conmon_2.1.2.deb
sudo apt install /tmp/conmon_2.1.2.deb

After i got another error:

Error: unable to start container "614b6b451087c3966f93b7e25b6fbceeb7d27bbf630e96bc010f474418330d5c": crun: cannot stat `/run/nanocl`: No such file or directory: OCI runtime attempted to invoke a command that was not found

Looks like podman don't create mounted directory on restart.
The only workarround i see right now is to create a systemd profile that run before podman-restart that create the directory..

[leon3s]

I'll add this later in the documentation but this is the steps to add the systemd profile:

sudo vim /etc/systemd/system/nanocl-podman-restart.service

The profile:

[Unit]
Description=Create /run/nanocl directory before podman-restart
Before=podman-restart.service

[Service]
Type=oneshot
ExecStart=/bin/mkdir -p /run/nanocl

[Install]
WantedBy=multi-user.target

Then we enable the profile:

sudo systemctl daemon-reload
sudo systemctl enable nanocl-podman-restart

[leon3s]

And it's look like there is deeper problem with podman.

Some of the data cannot be serialized correctly because they didn't correctly followed the docker structures.

Failed to deserialize JSON: unknown variant stopped, expected one of ``, created, `running`, `paused`, `restarting`, `removing`, `exited`, `dead` at line 1 column 192

I'll see if it come from the library i use but it generate the structure from the openapi spec from docker.

[izderadicka]

I'd recommend to try on distro with first class support for podman - like Fedora - behavior there might be different. Ubuntu support of podman seems to be bit dubious ...

[danielwernerio]

Has this podman situation improved since?