Load kind images per platform with Docker's containerd image store by wu-sheng · Pull Request #150 · apache/skywalking-infra-e2e

wu-sheng · 2026-10-22T15:28:45Z

Why

On ubuntu-26.04 runners (Docker 29.4.2), setup.kind.import-images fails for any multi-platform image pulled from a registry, before a test runs. apache/skywalking-horizon-ui#205 hit it in its kind cases (istio, deployment):

import docker images: ghcr.io/apache/skywalking/oap:<sha>
ERROR: failed to load image: command "docker exec --privileged -i kind-control-plane ctr --namespace=k8s.io images import --all-platforms --digests --snapshotter=overlayfs -" failed with error: exit status 1
Command Output: ctr: content digest sha256:bfb54eac...: not found

Docker 29 enables the containerd image store by default. A pulled multi-platform image there only holds the content of the host platform, but docker save still exports the index that references every platform, so the nodes' ctr images import --all-platforms asks for content the host never had. This is kubernetes-sigs/kind#3795, still open; kind main has the same save and import. ubuntu-24.04 (Docker 28.0.4, classic store) is unaffected. ubuntu-latest moves to 26.04 between Oct 19 and Nov 19, so every consumer with kind cases on ubuntu-latest will hit it.

The API version is not the cause: action.yaml already sets DOCKER_API_VERSION from the server, and the failed run used 1.54.

#140 had turned the containerd image store off in action.yaml; #146 removed that, so kind mode lost its protection. Putting the daemon switch back is not an option: consumers build images before calling the action, and switching stores hides those images.

What

New internal/components/setup/kind_image.go. It asks the daemon for its image store (driver-type: io.containerd.snapshotter.v1) and reads the client's negotiated API version.
- containerd store, API ≥ 1.48 (Docker 28+): save only the nodes' platform (ImageSave with the platform from ServerVersion().Arch, normalized) into a temp archive under the work dir, then kind load image-archive. This is the workaround kind documents in its known issues.
- classic store: kind load docker-image, as before.
- containerd store, API < 1.48: kind load docker-image, with a hint if the nodes report missing content.
Failed loads now carry the node command's output (kind leaves it out of the error), and the temp archive is removed on every path.
E2E: the kind job runs on ubuntu-24.04 and ubuntu-26.04. It imports pinned busybox:1.37.0 (an OCI index with 9 platforms and attestations) and a locally built image, and the cluster gains a worker. One pod runs on the worker and one is pinned to the control-plane, so a load that skipped a node fails.
containerd/platforms and opencontainers/image-spec become direct dependencies, at the versions already in the build.

Validation

make test, golangci-lint run (0 issues, Go 1.26.9), Linux and macOS builds, and license-eye header check pass locally. Unit tests cover store detection, mode choice by API version, platform normalization, and finding the node output through kind's and our wrapping.
The kind e2e has not run locally (the local Docker engine is not answering), so this PR's CI matrix is its first run.
Reviewed with Codex (gpt-6-astra) twice: on the plan, and on the code (ultra). The plan findings are applied (gate on the negotiated client API, read node output through RunError, normalize the platform, a deterministic fixture). The code review found nothing blocking; its one finding, no check on the control-plane, is fixed with the pinned pod.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CsP9pv9o23vwAUH88Qfe22

Docker 29 enables the containerd image store by default (ubuntu-26.04 runners). A pulled multi-platform image there only holds the content of the host platform, but `docker save` still exports an index referencing every platform, so `kind load docker-image` fails in the nodes' `ctr images import --all-platforms` with "content digest ... not found" (kubernetes-sigs/kind#3795). When the daemon reports the containerd image store and the client's negotiated API is 1.48 or later (Docker 28+), save only the kind nodes' platform and load it with `kind load image-archive`, the workaround kind documents. The classic image store keeps `kind load docker-image`. Failed loads now carry the node command output, with a hint when an older Docker's containerd store is missing content. The kind e2e now runs on ubuntu-24.04 (Docker 28, classic store) and ubuntu-26.04 (Docker 29, containerd store), imports a pinned multi-platform image and a locally built one, and loads into a worker node too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CsP9pv9o23vwAUH88Qfe22

…ages per platform Docker 29 (ubuntu-26.04 runners) uses the containerd image store, where a pulled multi-platform image holds only the host platform; importing every platform into kind failed for the OAP image ("content digest ... not found") and, on the containerd 1.7 kindest/node:v1.28.15 these cases pin, for the Rover image, whose attestation manifests do not unpack as images ("mismatched image rootfs and manifest layers"). infra-e2e now saves and imports only the node platform (apache/skywalking-infra-e2e#150, #151), so the e2e cases follow ubuntu-latest again instead of the ubuntu-24.04 pin #205 added.

…ntainerd 1.7 node #150 saves images for the kind nodes' platform with Docker's containerd image store and loads them with `kind load image-archive`. The saved archive still lists the image's attestation manifests at its top level, and kind imports it with `ctr images import --all-platforms --digests`. A containerd 1.7 node records and unpacks every top-level manifest as an image, whatever the platform flags, so an attestation manifest whose config lists its in-toto payloads as layers fails with "mismatched image rootfs and manifest layers" - apache/skywalking-horizon-ui's istio case (kindest/node:v1.28.15) failed so on the SkyWalking Rover image. The kind e2e runs kind's default node, containerd 2, which unpacks only the default platform, so it passed. Attestation manifests - marked as such, or manifests whose layers are all in-toto statements - are now dropped from the archive's index.json before kind loads it; every other entry is kept as saved. A new kind e2e case runs a containerd 1.7 node, imports the SkyWalking OAP and Rover images that failed in real use, as pulled by the run itself, and checks both reached the node. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ages per platform Docker 29 (ubuntu-26.04 runners) uses the containerd image store, where a pulled multi-platform image holds only the host platform; importing every platform into kind failed for the OAP image ("content digest ... not found") and, on the containerd 1.7 kindest/node:v1.28.15 these cases pin, for the Rover image, whose attestation manifests do not unpack as images ("mismatched image rootfs and manifest layers"). infra-e2e now saves only the node platform and leaves attestation manifests out of the archive (apache/skywalking-infra-e2e#150, #151), so the e2e cases follow ubuntu-latest again instead of the ubuntu-24.04 pin #205 added.

…rd 1.7 node #150 saves images for the kind nodes' platform with Docker's containerd image store and loads them with `kind load image-archive`. Docker saves the referrers of the image beside it in the archive's top-level index - its attestations, and the Cosign / Sigstore signatures Docker 29 pulls for them - each marked with the `io.containerd.manifest.subject` annotation. kind imports the archive with `ctr images import --all-platforms --digests`. containerd 2 skips the marked descriptors, but a containerd 1.7 node (node images built by kind before v0.27) creates an image for every top-level descriptor and unpacks it, and a referrer has no filesystem layers, so the import fails with "mismatched image rootfs and manifest layers" - apache/skywalking-horizon-ui's istio case (kindest/node:v1.28.15) failed so on the SkyWalking Rover image. The kind e2e runs kind's default node, containerd 2, so it passed. The referrers are now removed from the archive's index.json before kind loads it, as containerd 2 would skip them; the kept descriptors and every other entry are copied as saved. Both archive files are closed before the rewritten one replaces the original, which Windows requires. A new kind e2e case runs a containerd 1.7 node, imports the SkyWalking OAP and Rover images that failed in real use, as pulled by the run itself, and checks both reached the node. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ages per platform Docker 29 (ubuntu-26.04 runners) uses the containerd image store, where a pulled multi-platform image holds only the host platform; importing every platform into kind failed for the OAP image ("content digest ... not found"). Saved for one platform, the archive also lists the image's referrers, such as attestations, at its top level, and the containerd 1.7 kindest/node:v1.28.15 these cases pin imports each one as an image, which failed for the Rover image ("mismatched image rootfs and manifest layers"). infra-e2e now saves only the node platform and leaves the referrers out of the archive (apache/skywalking-infra-e2e#150, #151), so the e2e cases follow ubuntu-latest again instead of the ubuntu-24.04 pin #205 added.

…rd 1.7 node #150 saves images for the kind nodes' platform with Docker's containerd image store and loads them with `kind load image-archive`. Docker saves the referrers of the image beside it in the archive's top-level index - its attestations, and the Cosign / Sigstore signatures Docker 29 pulls for them - each marked with the `io.containerd.manifest.subject` annotation. kind imports the archive with `ctr images import --all-platforms --digests`. A containerd 1.7 node (node images built by kind v0.20 to v0.26) creates an image for every top-level descriptor and unpacks it, and a referrer has no filesystem layers, so the import fails with "mismatched image rootfs and manifest layers" - apache/skywalking-horizon-ui's istio case (kind v0.25's kindest/node:v1.28.15) failed so on the SkyWalking Rover image. containerd 2, kind's default node, only unpacks manifests that have filesystem layers, so the kind e2e passed. The referrers are now removed from the archive's index.json before kind loads it; a descriptor that names an image is kept even when marked. The kept descriptors and every other entry are copied as saved. Both archive files are closed before the rewritten one replaces the original, which Windows requires. A new kind e2e case runs a containerd 1.7 node, imports the SkyWalking OAP and Rover images that failed in real use, as pulled by the run itself, and checks both reached the node. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ages per platform Docker 29 (ubuntu-26.04 runners) uses the containerd image store, where a pulled multi-platform image holds only the host platform; importing every platform into kind failed for the OAP image ("content digest ... not found"). Saved for one platform, the archive also lists the image's referrers, such as attestations, at its top level, and the containerd 1.7 kindest/node:v1.28.15 these cases pin imports each one as an image, which failed for the Rover image ("mismatched image rootfs and manifest layers"). infra-e2e now saves only the node platform and leaves the referrers out of the archive (apache/skywalking-infra-e2e#150, #151), so the e2e cases follow ubuntu-latest again instead of the ubuntu-24.04 pin #205 added.

…rd 1.7 node #150 saves images for the kind nodes' platform with Docker's containerd image store and loads them with `kind load image-archive`. Docker saves the referrers of the image beside it in the archive's top-level index - its attestations, and the Cosign / Sigstore signatures Docker 29 pulls for them - each marked with the `io.containerd.manifest.subject` annotation. kind imports the archive with `ctr images import --all-platforms --digests`. A containerd 1.7 node (node images built by kind v0.20 to v0.26) creates an image for every top-level descriptor and unpacks it, and a referrer has no filesystem layers, so the import fails with "mismatched image rootfs and manifest layers" - apache/skywalking-horizon-ui's istio case (kind v0.25's kindest/node:v1.28.15) failed so on the SkyWalking Rover image. containerd 2, kind's default node, only unpacks manifests that have filesystem layers, so the kind e2e passed. The referrers are now removed from the archive's index.json before kind loads it; a descriptor that names an image is kept even when marked. The kept descriptors and every other entry are copied as saved. Both archive files are closed before the rewritten one replaces the original, which Windows requires. A new kind e2e case runs a containerd 1.7 node, imports the SkyWalking OAP and Rover images that failed in real use, as pulled by the run itself, and checks both reached the node. The Build workflow's Linux jobs run on ubuntu-26.04 instead of ubuntu-latest: ubuntu-latest jobs sat queued with no runner while the ubuntu-24.04 / ubuntu-26.04 jobs started within seconds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ges per platform Docker 29 (ubuntu-26.04 runners) uses the containerd image store, where a pulled multi-platform image holds only the host platform; importing every platform into kind failed for the OAP image ("content digest ... not found"). Saved for one platform, the archive also lists the image's referrers, such as attestations, at its top level, and the containerd 1.7 kindest/node:v1.28.15 these cases pin imports each one as an image, which failed for the Rover image ("mismatched image rootfs and manifest layers"). infra-e2e now saves only the node platform and leaves the referrers out of the archive (apache/skywalking-infra-e2e#150, #151), so the e2e cases leave the ubuntu-24.04 pin #205 added. The CI and e2e jobs name ubuntu-26.04 rather than ubuntu-latest: ubuntu-latest jobs sat queued with no runner while jobs naming ubuntu-24.04 or ubuntu-26.04 started within seconds.

…ges per platform Docker 29 (ubuntu-26.04 runners) uses the containerd image store, where a pulled multi-platform image holds only the host platform; importing every platform into kind failed for the OAP image ("content digest ... not found"). Saved for one platform, the archive also lists the image's referrers, such as attestations, at its top level, and the containerd 1.7 kindest/node:v1.28.15 these cases pin imports each one as an image, which failed for the Rover image ("mismatched image rootfs and manifest layers"). infra-e2e now saves only the node platform and leaves the referrers out of the archive (apache/skywalking-infra-e2e#150, #151), so the e2e cases leave the ubuntu-24.04 pin #205 added. The CI and e2e jobs name ubuntu-26.04 rather than ubuntu-latest: ubuntu-latest jobs sat queued with no runner while jobs naming ubuntu-24.04 or ubuntu-26.04 started within seconds. The istio mesh spec accepts `UNKNOWN` as the service the MESH header opens on. Since the e2e OAP pin moved past apache/skywalking#14135 (#204), OAP reads the Kubernetes metadata of a new pod address in the background, so bookinfo's first calls count under `UNKNOWN`; in the first minute of traffic that burst can rank first by calls per minute, and the header then picks it. It failed so on main at ed6171d (ubuntu-24.04) as well as here.

…ges per platform Docker 29 (ubuntu-26.04 runners) uses the containerd image store, where a pulled multi-platform image holds only the host platform; importing every platform into kind failed for the OAP image ("content digest ... not found"). Saved for one platform, the archive also lists the image's referrers, such as attestations, at its top level, and the containerd 1.7 kindest/node:v1.28.15 these cases pin imports each one as an image, which failed for the Rover image ("mismatched image rootfs and manifest layers"). infra-e2e now saves only the node platform and leaves the referrers out of the archive (apache/skywalking-infra-e2e#150, #151), so the e2e cases leave the ubuntu-24.04 pin #205 added. The CI and e2e jobs name ubuntu-26.04 rather than ubuntu-latest: ubuntu-latest jobs sat queued with no runner while jobs naming ubuntu-24.04 or ubuntu-26.04 started within seconds. The istio mesh and eBPF profiling specs accept `UNKNOWN` as the service the MESH header opens on (MESH_HEADER_SERVICES; MESH_SERVICES still lists only the real workloads). Since the e2e OAP pin moved past apache/skywalking#14135 (#204), OAP reads the Kubernetes metadata of a new pod address in the background, so bookinfo's first calls count under `UNKNOWN`; in the first minute of traffic that burst can rank first by calls per minute, and the header then picks it. It failed so on main at ed6171d (ubuntu-24.04) as well as here. A mesh whose calls all count under `UNKNOWN` still fails: the mesh spec requires every bookinfo peer in the service picker.

…ges per platform Docker 29 (ubuntu-26.04 runners) uses the containerd image store, where a pulled multi-platform image holds only the host platform; importing every platform into kind failed for the OAP image ("content digest ... not found"). Saved for one platform, the archive also lists the image's referrers, such as attestations, at its top level, and the containerd 1.7 kindest/node:v1.28.15 these cases pin imports each one as an image, which failed for the Rover image ("mismatched image rootfs and manifest layers"). infra-e2e now saves only the node platform and leaves the referrers out of the archive (apache/skywalking-infra-e2e#150, #151), so the e2e cases leave the ubuntu-24.04 pin #205 added. The CI and e2e jobs name ubuntu-26.04 rather than ubuntu-latest: ubuntu-latest jobs sat queued with no runner while jobs naming ubuntu-24.04 or ubuntu-26.04 started within seconds. The istio mesh fixture is fully known, so its specs now require all of it and accept only `UNKNOWN` beyond it. The mesh spec asserts the service picker lists exactly the five workloads (bookinfo's four and the ingress gateway) plus, optionally, `UNKNOWN`; the topology spec requires a node for each workload by name; and the roster readiness check waits until Horizon lists all five. MESH_PEERS, which left `ratings` out, is gone. `UNKNOWN` is accepted as the service the MESH header opens on (MESH_HEADER_SERVICES, in the mesh and eBPF profiling specs). Since the e2e OAP pin moved past apache/skywalking#14135 (#204), OAP reads the Kubernetes metadata of a new pod address in the background, so bookinfo's first calls count under `UNKNOWN`; in the first minute of traffic that burst can rank first by calls per minute, and the header then picks it. It failed so on main at ed6171d (ubuntu-24.04) as well as here.

…rd 1.7 node (#151) #150 saves images for the kind nodes' platform with Docker's containerd image store and loads them with `kind load image-archive`. Docker saves the referrers of the image beside it in the archive's top-level index - its attestations, and the Cosign / Sigstore signatures Docker 29 pulls for them - each marked with the `io.containerd.manifest.subject` annotation. kind imports the archive with `ctr images import --all-platforms --digests`. A containerd 1.7 node (node images built by kind v0.20 to v0.26) creates an image for every top-level descriptor and unpacks it, and a referrer has no filesystem layers, so the import fails with "mismatched image rootfs and manifest layers" - apache/skywalking-horizon-ui's istio case (kind v0.25's kindest/node:v1.28.15) failed so on the SkyWalking Rover image. containerd 2, kind's default node, only unpacks manifests that have filesystem layers, so the kind e2e passed. The referrers are now removed from the archive's index.json before kind loads it; a descriptor that names an image is kept even when marked. The kept descriptors and every other entry are copied as saved. Both archive files are closed before the rewritten one replaces the original, which Windows requires. A new kind e2e case runs a containerd 1.7 node, imports the SkyWalking OAP and Rover images that failed in real use, as pulled by the run itself, and checks both reached the node. The Build workflow's Linux jobs run on ubuntu-26.04 instead of ubuntu-latest: ubuntu-latest jobs sat queued with no runner while the ubuntu-24.04 / ubuntu-26.04 jobs started within seconds.

…ges per platform Docker 29 (ubuntu-26.04 runners) uses the containerd image store, where a pulled multi-platform image holds only the host platform; importing every platform into kind failed for the OAP image ("content digest ... not found"). Saved for one platform, the archive also lists the image's referrers, such as attestations, at its top level, and the containerd 1.7 kindest/node:v1.28.15 these cases pin imports each one as an image, which failed for the Rover image ("mismatched image rootfs and manifest layers"). infra-e2e now saves only the node platform and leaves the referrers out of the archive (apache/skywalking-infra-e2e#150, #151), so the e2e cases leave the ubuntu-24.04 pin #205 added. The CI and e2e jobs name ubuntu-26.04 rather than ubuntu-latest: ubuntu-latest jobs sat queued with no runner while jobs naming ubuntu-24.04 or ubuntu-26.04 started within seconds. The istio mesh fixture is fully known, so its specs now require all of it and accept only `UNKNOWN` beyond it. The mesh spec asserts the service picker lists exactly the five workloads (bookinfo's four and the ingress gateway) plus, optionally, `UNKNOWN`; the topology spec requires a node for each workload by name; and the roster readiness check waits until Horizon lists all five. MESH_PEERS, which left `ratings` out, is gone. `UNKNOWN` is accepted as the service the MESH header opens on (MESH_HEADER_SERVICES, in the mesh and eBPF profiling specs). Since the e2e OAP pin moved past apache/skywalking#14135 (#204), OAP reads the Kubernetes metadata of a new pod address in the background, so bookinfo's first calls count under `UNKNOWN`; in the first minute of traffic that burst can rank first by calls per minute, and the header then picks it. It failed so on main at ed6171d (ubuntu-24.04) as well as here.

…ges per platform (#206) Docker 29 (ubuntu-26.04 runners) uses the containerd image store, where a pulled multi-platform image holds only the host platform; importing every platform into kind failed for the OAP image ("content digest ... not found"). Saved for one platform, the archive also lists the image's referrers, such as attestations, at its top level, and the containerd 1.7 kindest/node:v1.28.15 these cases pin imports each one as an image, which failed for the Rover image ("mismatched image rootfs and manifest layers"). infra-e2e now saves only the node platform and leaves the referrers out of the archive (apache/skywalking-infra-e2e#150, #151), so the e2e cases leave the ubuntu-24.04 pin #205 added. The CI and e2e jobs name ubuntu-26.04 rather than ubuntu-latest: ubuntu-latest jobs sat queued with no runner while jobs naming ubuntu-24.04 or ubuntu-26.04 started within seconds. The istio mesh fixture is fully known, so its specs now require all of it and accept only `UNKNOWN` beyond it. The mesh spec asserts the service picker lists exactly the five workloads (bookinfo's four and the ingress gateway) plus, optionally, `UNKNOWN`; the topology spec requires a node for each workload by name; and the roster readiness check waits until Horizon lists all five. MESH_PEERS, which left `ratings` out, is gone. `UNKNOWN` is accepted as the service the MESH header opens on (MESH_HEADER_SERVICES, in the mesh and eBPF profiling specs). Since the e2e OAP pin moved past apache/skywalking#14135 (#204), OAP reads the Kubernetes metadata of a new pod address in the background, so bookinfo's first calls count under `UNKNOWN`; in the first minute of traffic that burst can rank first by calls per minute, and the header then picks it. It failed so on main at ed6171d (ubuntu-24.04) as well as here.

wu-sheng requested review from fgksgf and kezhenxu94 October 22, 2026 15:34

wu-sheng added this to the 1.4.0 milestone Oct 22, 2026

wu-sheng added the feature label Oct 22, 2026

wu-sheng mentioned this pull request Oct 22, 2026

fix: upgrade E2E harness and stabilize native compaction apache/skywalking-banyandb#1427

Merged

kezhenxu94 approved these changes Oct 23, 2026

View reviewed changes

wu-sheng merged commit c45db27 into main Oct 23, 2026
9 of 12 checks passed

wu-sheng deleted the fix/kind-load-containerd-image-store branch October 23, 2026 00:39

This was referenced Oct 23, 2026

ci: run CI and e2e on ubuntu-26.04, with infra-e2e importing kind images per platform apache/skywalking-horizon-ui#206

Merged

Leave image referrers out of kind image archives, and test a containerd 1.7 node #151

Merged

Navigation Menu

Repository navigation

Load kind images per platform with Docker's containerd image store - #150

Load kind images per platform with Docker's containerd image store#150
wu-sheng merged 1 commit into
mainfrom
fix/kind-load-containerd-image-store

wu-sheng commented Oct 22, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

wu-sheng commented Oct 22, 2026

Why

What

Validation

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants