Best practice for Rootless Podman Socket as a dependency for System-level Units (Rocky 9)

[schlitzered]

Hello Podman Team,

I am trying to set up a Forgejo Runner on Rocky 9 that runs as a system-level unit but interacts with a rootless Podman instance. Getting this to work reliably across reboots has been extremely difficult, and the resulting unit file feels like it contains far too many workarounds.

I am looking for the "officially recommended" way to handle this dependency chain, as my current solution feels like I am fighting the OS.

The Setup:

• Host: Rocky Linux 9
• Service: Forgejo Runner (System-level unit: /etc/systemd/system/forgejo-runner@.service)
• Requirement: The runner must be active at boot and connect to the Podman API via the rootless socket for the forgejo user (UID 1000).

[Unit]
Description=Forgejo Runner: %i
After=network.target user-runtime-dir@1000.service user@1000.service
Requires=user-runtime-dir@1000.service user@1000.service

[Service]
User=forgejo
Group=forgejo
WorkingDirectory=/home/forgejo/%i
Delegate=yes
KillMode=process

Environment=HOME=/home/forgejo
Environment=XDG_RUNTIME_DIR=/run/user/1000
Environment=DOCKER_HOST=unix:///run/user/1000/podman/podman.sock
Environment=DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus

# 1. Force Linger (as root) to ensure the runtime dir exists
ExecStartPre=+/usr/bin/loginctl enable-linger forgejo

# 2. Force Enable the user-level socket (as root via machinectl)
ExecStartPre=+/usr/bin/machinectl shell forgejo@.host /usr/bin/systemctl --user enable podman.socket

# 3. Cleanup stale lockfiles and wait for the API to respond
ExecStartPre=/usr/bin/bash -c ' \
    rm -rf /tmp/storage-run-1000/containers /tmp/storage-run-1000/libpod/tmp; \
    until curl -s --unix-socket /run/user/1000/podman/podman.sock http://localhost/_ping | grep -q "OK"; do \
        sleep 1; \
    done'

ExecStart=/home/forgejo/%i/forgejo daemon --config /home/forgejo/%i/config.yaml

Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

Questions

• Is there a cleaner way to handle the cross-dependency? Having a system unit reach into a user session via machinectl and loginctl feels like a hack.
• The "Boot ID" Error: Why is it necessary to manually rm -rf the storage directories in /tmp to avoid the "current system boot ID differs from cached boot ID" error? Is there a Podman command that handles this cleanup safely during a service start?
• Is Rootless the right tool here? If the goal is a headless, persistent system service, should I be using a system-wide socket (/run/podman/podman.sock) with group permissions instead of fighting the user-session lifecycle?

I also attempted to pivot to a Rootful setup to see if it offered a more 'native' systemd experience. While it removed the loginctl and machinectl hacks, it introduced a different kind of configuration sprawl. To get a non-root user (forgejo) to reliably communicate with the rootful socket across reboots on Rocky 9, I had to implement the following:

1. A Systemd Socket Override to fix permissions:

# /etc/systemd/system/podman.socket.d/override.conf
[Socket]
SocketMode=0660
SocketUser=root
SocketGroup=podman
DirectoryMode=0750

2. A tmpfiles.d entry because the socket unit alone wouldn't reliably set the group ownership of the parent /run/podman directory on boot:

# /etc/tmpfiles.d/podman-rootful.conf
d /run/podman 0750 root podman - -

3. The Final Runner Unit:

/etc/systemd/system/forgejo-runner@.service
[Unit]
Description=Forgejo Runner: %i
# 1. Strict requirements
After=network.target podman.socket
Requires=podman.socket

[Service]
User=forgejo
Group=forgejo
WorkingDirectory=/home/forgejo/%i

# 2. Point to the STABLE system-wide socket
Environment=DOCKER_HOST=unix:///run/podman/podman.sock

# 3. The "Patience" Step
# Instead of an insane loop, we just ensure the socket is actually responsive 
# before the binary starts.

ExecStart=/home/forgejo/%i/forgejo daemon --config /home/forgejo/%i/config.yaml

# 4. Give it a bit more breathing room on failure
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

The Core Frustration:
Whether I go Rootless or Rootful, I am fighting the ephemeral nature of /run, the default restrictive directory permissions of the Podman installer, and the lack of a 'clean' dependency chain between a system unit and the Podman API.

Is there a 'blessed' way to do this that doesn't involve manually wiring up tmpfiles.d or escaping into user-sessions via machinectl? It feels like Podman is missing a 'System Service Mode' that just works for headless server automation."

[NezSez]

FYI: I posted this ticket in the Rocky Linux MatterMost #sig/containers

[NezSez]

Running rootless podman in an unprivileged container in kubernetes #28123 might have some useful info for you.