Skip to content

fix: Replace a test container whose port Docker did not publish, and terminate every container that failed to start. - #188

Merged
goloroden merged 2 commits into
mainfrom
fix/replace-unpublished-test-container
Oct 6, 2026
Merged

goloroden merged 2 commits into
mainfrom
fix/replace-unpublished-test-container

Conversation

@goloroden

Copy link
Copy Markdown
Member

Fixes a flaky start of the test container on macOS.

The problem. Docker Desktop picks a random host port for the container, and opens it on the Mac only afterwards. If something on the Mac takes that port in between, publishing it fails silently: Docker Desktop logs "bind: address already in use".

  • The container keeps running, but its port is never published.
  • From then on, every inspect hangs about 3 seconds in Docker Desktop's proxy, so Start fails after its 10-second timeout with start container: … get state … context deadline exceeded.
  • A longer timeout does not help: with 60 seconds, 6 of 900 starts still failed the same way.
  • It only happens with Docker Desktop, not on Linux.

The fix.

  • Replacing a broken container. If a start fails while the container runs without a published port for the database, Start terminates it and starts a new one, up to 3 attempts. Every other error is returned at once and unchanged, so this reacts only to that one state, not to failures in general.
  • Cleaning up. A container that failed to start is now always terminated. Before, it was left behind, sometimes stuck in "Created".
  • A request per attempt. Each attempt builds its own request, because the signing-key file of a request is read to the end on the first start.
  • No panic on a typed nil. A failed start that hands out a container of a concrete type that is nil is treated as no container, as testcontainers does.

Tests. The tests produce the unpublished state on purpose: with Networks: ["none"], which reproduces the real hang on Docker Desktop, and with only another port published. They cover:

  • replacing the container;
  • giving up after 3 attempts;
  • returning every other failure at once;
  • the context ending while waiting and while terminating;
  • a failure to terminate;
  • a typed nil.

All new cases fail on main. 23 mutations of the new code are all caught. make qa and -race are green.

Not in this PR: the other flaky start, wait for reaper …: unexpected container status "created". It is a bug in testcontainers-go, which upstream PR testcontainers/testcontainers-go#3868 fixes. The version will be bumped once that fix is released.

🤖 Generated with Claude Code

https://claude.ai/code/session_01TCdPte6nY6ToUVd2prb9GM

goloroden and others added 2 commits October 6, 2026 11:55
…terminate every container that failed to start.

Under load, Docker Desktop sometimes fails to publish the host port it
chose for a container, because something on the host took that port in
the meantime. The container keeps running, but the database in it can
never be reached, so Start ran into its timeout, however long it was.
Start now terminates such a container and starts a new one, up to three
attempts in total, and its error says that Docker did not publish the
port if all of them fail. It makes no new attempt once the context has
ended. Every other failure is returned at once, as before.

Start also terminates every container that failed to start, so that
they no longer pile up, and reports if terminating one fails. Each
attempt builds a request of its own, as starting a container reads the
signing key file of its request to the end.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TCdPte6nY6ToUVd2prb9GM
…an inspect it.

A container is an interface, so a failed start may hand out a nil
pointer to a concrete container, such as a nil *DockerContainer, which
is not nil as an interface. Inspecting it to tell whether Docker
published its port panicked. Such a container now counts as no
container, as testcontainers.TerminateContainer checks it, so Start
returns the error of the attempt at once, as it is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TCdPte6nY6ToUVd2prb9GM
@goloroden goloroden self-assigned this Oct 6, 2026
@goloroden
goloroden requested a review from a team as a code owner October 6, 2026 10:02
@goloroden
goloroden merged commit 83613bc into main Oct 6, 2026
2 checks passed
@goloroden
goloroden deleted the fix/replace-unpublished-test-container branch October 6, 2026 10:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant