Half the containers don't come back after a reboot or docker restart #591
Description
Hi, me again sorry!
Describe the bug
docker-mailserver does not come back properly after a reboot, or even just a restart of Docker.
Beforehand it's running nicely. Then I reboot my VPS, or do systemctl restart docker. When it comes back only 4 of the containers are running (mda, mta, web, filter), and they are not happy - listed as "unhealthy" in docker ps:
# docker ps
020f67b25da8 jeboehm/mailserver-mda:latest "/usr/bin/tini -- /u…" 10 minutes ago Up About a minute (unhealthy) 2003-2004/tcp, 8080/tcp, 9090/tcp, 31024/tcp, 31587/tcp, 31990/tcp, 34190/tcp, 0.0.0.0:110->31110/tcp, 0.0.0.0:143->31143/tcp, 0.0.0.0:993->31993/tcp, 0.0.0.0:995->31995/tcp docker-mailserver-mda-1
aa718f9fb4da jeboehm/mailserver-web:latest "docker-php-entrypoi…" 10 minutes ago Up About a minute (unhealthy) 8080/tcp, 9000/tcp docker-mailserver-web-1
00ec34d62de7 jeboehm/mailserver-filter:latest "/usr/local/bin/entr…" 10 minutes ago Up 9 minutes (unhealthy) 11332-11334/tcp docker-mailserver-filter-1
02f46523d084 jeboehm/mailserver-mta:latest "/usr/local/bin/entr…" 10 minutes ago Up About a minute (unhealthy) 0.0.0.0:25->25/tcp, 0.0.0.0:465->465/tcp, 0.0.0.0:587->587/tcp docker-mailserver-mta-1
fetchmail, db, redis, and unbound are not running.
docker compose logs is full of complaints, most of which seem to revolve around being unable to resolve names - unsurprising as Unbound is one of the containers that failed to start - but of course nothing's going to get very far without the database either.
I thought the cause might be the use of a wrapper script (bin/production.sh) instead of using docker compose with a single top-level compose file, so I moved the port definitions from docker-compose.production.yml into the relevant files in deploy/compose/*, but it's exactly the same.
Obviously I can just down and up them and the problem is solved, so it's not the hardest thing to work around by having a boot script - though I'm not sure how you'd ensure that a docker restart always runs it. But it would be cleaner if the containers came back by themselves, or if none of them did rather than having a broken half-up system until that script runs.
I do have a solution: In all the deploy/compose/* files I replaced
restart: on-failure:5
with
restart: always
then restarted the containers. This seems to fix the problem - they now come back properly after a docker restart or system reboot. I have no idea if it's the "right" solution though, or if it could have bad consequences. But it seems to me that a production mailserver should probably keep trying if at first it doesn't succeed.
To Reproduce
bin/production.sh up
docker ps - observe all containers running
reboot
ssh back in after rebot
docker ps - observe only 4 containers running and they're not happy
Expected behavior
All containers come back online and working after reboot (preferred).
Docker environment (please complete the following information):
- Output of
docker ps -a(filter relevant containers)
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ca1350f5a5d4 ghcr.io/jeboehm/fetchmailmgr:0.4.0 "node src/index.js -…" 3 minutes ago Exited (0) 2 minutes ago docker-mailserver-fetchmail-1
e38c1c0ee320 jeboehm/mailserver-filter:latest "/usr/local/bin/entr…" 3 minutes ago Up About a minute (unhealthy) 11332-11334/tcp docker-mailserver-filter-1
eb94e0930c5f jeboehm/mailserver-web:latest "docker-php-entrypoi…" 3 minutes ago Up About a minute (unhealthy) 8080/tcp, 9000/tcp docker-mailserver-web-1
c6b1a0b658b1 jeboehm/mailserver-mda:latest "/usr/bin/tini -- /u…" 3 minutes ago Up About a minute (unhealthy) 2003-2004/tcp, 8080/tcp, 9090/tcp, 31024/tcp, 31587/tcp, 31990/tcp, 34190/tcp, 0.0.0.0:110->31110/tcp, 0.0.0.0:143->31143/tcp, 0.0.0.0:993->31993/tcp, 0.0.0.0:995->31995/tcp docker-mailserver-mda-1
9068ae52fdb8 jeboehm/mailserver-ssl:latest "/usr/local/bin/crea…" 3 minutes ago Exited (0) 3 minutes ago docker-mailserver-ssl-1
8a055a206e67 mysql:lts "docker-entrypoint.s…" 3 minutes ago Exited (0) 2 minutes ago docker-mailserver-db-1
f153cc724430 jeboehm/mailserver-mta:latest "/usr/local/bin/entr…" 3 minutes ago Up About a minute (unhealthy) 0.0.0.0:25->25/tcp, 0.0.0.0:465->465/tcp, 0.0.0.0:587->587/tcp docker-mailserver-mta-1
989b0d4eefa8 redis:8.2-alpine "docker-entrypoint.s…" 3 minutes ago Exited (0) 2 minutes ago docker-mailserver-redis-1
645b1db6504a jeboehm/mailserver-unbound:latest "entrypoint.sh" 3 minutes ago Exited (0) 2 minutes ago docker-mailserver-unbound-1
- Output of
docker info
Client:
Version: 28.2.2
Context: default
Debug Mode: false
Plugins:
compose: Docker Compose (Docker Inc.)
Version: 2.37.1+ds1-0ubuntu2~24.04.1
Path: /usr/libexec/docker/cli-plugins/docker-compose
Server:
Containers: 10
Running: 4
Paused: 0
Stopped: 6
Images: 13
Server Version: 28.2.2
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
CDI spec directories:
/etc/cdi
/var/run/cdi
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version:
runc version:
init version:
Security Options:
apparmor
seccomp
Profile: builtin
cgroupns
Kernel Version: 6.8.0-88-generic
Operating System: Ubuntu 24.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 7.755GiB
Name: her
ID: c461bfb1-f197-4610-b46b-70d709e8a2ac
Docker Root Dir: /var/lib/docker
Debug Mode: false
Experimental: false
Insecure Registries:
::1/128
127.0.0.0/8
Live Restore Enabled: false
- Output of
docker-compose version
Docker Compose version 2.37.1+ds1-0ubuntu2~24.04.1