Commit and building completely broken (wrong uids...) when userns+idmapping is used #2396

[lyphyser]

[The first part is written by me, then an AI-written report follows, which I deem correct]

Apparently containers/storage expects that containers are stored with 0-based UIDs on disk, but also fails to idmap the upper layers so that this actually happens, and even fails to record the fact in the layer.

The result is that images created by podman save have UID values in the uidmap range, containers cannot start on those images, and ALL building with buildah with userns enabled is broken.

In general, this shows complete carelessness in implementation and testing. Userns with mount idmapping is the only code path that provides both security and performance, and needs to work and be the main tested and used path.

Here is what I think probably needs to be fixed:

1. Hardcoding storage to have 0-based UIDs is completely absurd. Instead, it should be allowed to be in ANY UID range, and then idmapped back using the composition of the storage layer idmap and the current container idmap (e.g. if it's stored as UID 200000 based and the current container uses UID 300000 based, idmap UID 200000 to 300000 in the mount idmap - or perhaps map UID 200000 to 0 or some fixed high range in the idmap for that layer, and then map 0 or some fixed high range to 300000 into an idmap for the whole overlayfs stack). This is needed since files on disk must reflect the container who created them for security reasons. When pulling images, it should use a fixed high UID range used for this purpose.
2. The UID map needs to be always stored. Currently it seems that if shifting is supported, then the UID map is not even stored.

It's pretty unbelievable that an issue of this magnitude exists.

Using podman 5.8.2.

[text removed as CoC violation]

Claude Fable 5 report

Committing a container that uses an ID mapping produces image layers with host-shifted UIDs/GIDs when idmapped mounts are in use

Description

When the overlay driver uses idmapped mounts (SupportsShifting() true: rootful, native overlay, no mount_program, recent kernel), committing a mapped container emits host-shifted UIDs/GIDs into the image layer instead of canonical 0-based ones.

Consequences:

• podman run --userns=auto <committed-image> fails: the container needs a user namespace with size <huge> that is bigger than the maximum value allowed with userns=auto 65536.
• With a fixed mapping, files from the committed layer appear as nobody:nogroup in the next container (the idmapped lower mount shifts already-shifted IDs out of range).
• podman build / buildah build --userns-uid-map breaks at every step boundary, since each step commits an intermediate image that the next step consumes as an idmapped lower.
• Host subordinate-ID layout leaks into pushable OCI blobs.

Reproducible with plain podman run + podman commit, so the defect is in containers/storage; podman and buildah are affected consumers.

Steps to reproduce

# podman run --name t --userns=auto alpine touch /x
# podman commit t t2
# podman run --rm --userns=auto t2 ls -ln /x
Error: creating container storage: the container needs a user namespace with
size 1074136065 that is bigger than the maximum value allowed with userns=auto 65536

Top layer tar of podman save t2 (everything written through the mapped container carries the auto-allocated host ID; untouched base content is correctly 0-based):

-rw-r--r-- 1074136064/1074136064 0 2026-06-11 20:15 x
-rwxr-xr-x 1074136064/1074136064 0 2026-06-11 20:15 run/.containerenv
drwxr-xr-x 0/0                   0 2026-04-15 04:51 bin/

Expected results

Committed layers contain 0-based IDs, as they do when idmapped mounts are unavailable.

Root cause (from current main)

1. store.CreateContainer: when canUseShifting(), the container's layer is created with HostUIDMapping: true, UIDMap: nil — the real map is kept only on the container record, not the layer.
2. overlay.get(): idmapped mounts are applied only to lowers; upperdir is the raw diff. All writes (RUN output through the userns; buildah COPY, which chowns ToHost) land physically shifted in the diff.
3. layerStore.Diff untranslates via layerMappings(toLayer) — nil because of (1) — so the diff tar keeps the shifted IDs and the committed layer is shifted on disk while recorded as unmapped.
4. Next use: the layer is accepted as-is (layerMatchesMappingOptions: shifting + no maps) and mounted with an idmapped mount from a 0-based source → double shift → overflow ID; --userns=auto sizing reads the same shifted IDs.

Writers still follow the legacy chown-scheme contract (shifted on disk, unshift at Diff via recorded maps); the shifting scheme blanks those maps and assumes 0-based content. The unshift silently no-ops.

Workaround (untested, per code)

_CONTAINERS_OVERLAY_DISABLE_IDMAP=yes should force the legacy chown path, where layer maps are recorded and Diff untranslates.

Possible fix

Record the true maps on the container's layer even under shifting — the upper is never idmapped at mount time, so this is truthful and lets Diff untranslate, without changing the 0-based scheme for shared image lowers.

Prior sighting

podman-container-tools/podman#16795 (2022): same signature (podman build --userns=auto, step 2 demands userns size = step 1's auto base + offset); closed as a subuid-range/size configuration problem. Raising auto-userns-max-size only masks it — files get wrong high in-container UIDs and the shift compounds per step.

Versions

$ podman --version
podman version 5.8.2

[lyphyser]

Claude Fable produced this patch. I have not read it beyond vague skimming and not tried it and cannot vouch in any way for its correctness. However, the model is very good, so it's probably a reasonable starting point; it was however done from the Chat UI with not really much context curation/etc., no guidance, etc. so you can do a better job using Claude Code yourself

Subject: [PATCH] layers: record container-layer ID maps and compose them with
 mount-time idmapped mounts

Status: compile-untested draft against containers/storage main (2026-06-11),
written to accompany the bug report "Committing a container that uses an ID
mapping produces image layers with host-shifted UIDs/GIDs when idmapped
mounts are in use".

Two logical changes:

1. store.CreateContainer: stop blanking the container layer's UIDMap/GIDMap
   when canUseShifting() is true.  The overlay driver never idmaps the
   upperdir, so the container's writable layer physically contains shifted
   IDs; recording the map makes the metadata truthful and lets
   layerStore.Diff untranslate at commit time.  This alone fixes the
   corruption: committed image layers come out canonical 0-based.

2. Compose recorded layer maps into mount-time idmapped mounts.  A layer
   whose on-disk content is mapped (recorded UIDMap/GIDMap, e.g. the
   chown-scheme mapped copies, or container layers after change 1, in mixed
   stores) must not be idmapped from a 0-based source.  The translation for
   such a lower is layer-map^-1 composed with the requested container map.
   When the composition is the identity (layer already chowned to the
   requested mapping) the idmapped mount is skipped entirely.

Known limitations of this draft:
- layers in additional (read-only) image stores are not visible to
  layerStore.Mount's ancestry walk, so their recorded maps (if any) are not
  propagated; canonical 0-based layers (the overwhelmingly common case) are
  unaffected since an absent entry means identity source.
- imageTopLayerForMapping still blanks maps on mapped template copies; with
  composition those copies become unnecessary under shifting and that branch
  could be removed in a follow-up.

---
 drivers/driver.go          | 10 ++++++
 drivers/overlay/overlay.go | 63 +++++++++++++++++++++++++++++++-------
 layers.go                  | 23 ++++++++++++++
 pkg/idtools/idtools.go     | 46 +++++++++++++++++++++++++++
 store.go                   | 16 +++-------
 5 files changed, 136 insertions(+), 22 deletions(-)

diff --git a/store.go b/store.go
--- a/store.go
+++ b/store.go
@@ -1890,18 +1890,12 @@ func (s *store) CreateContainer(id string, names []string, image, layer, metada
 		// But in transient store mode, all container layers are volatile.
 		Volatile: options.Volatile || s.transientStore,
 	}
-	if s.canUseShifting(uidMap, gidMap) {
-		layerOptions.IDMappingOptions = types.IDMappingOptions{
-			HostUIDMapping: true,
-			HostGIDMapping: true,
-			UIDMap:         nil,
-			GIDMap:         nil,
-		}
-	} else {
-		layerOptions.IDMappingOptions = types.IDMappingOptions{
-			HostUIDMapping: idMappingsOptions.HostUIDMapping,
-			HostGIDMapping: idMappingsOptions.HostGIDMapping,
-			UIDMap:         copySlicePreferringNil(uidMap),
-			GIDMap:         copySlicePreferringNil(gidMap),
-		}
+	// Always record the real mapping on the container's layer, even when
+	// mount-time shifting is available: the overlay upperdir is never
+	// idmapped, so content written through the container's user namespace
+	// is physically shifted on disk.  Recording the map keeps the metadata
+	// truthful and lets Diff untranslate it (e.g. at commit time).
+	layerOptions.IDMappingOptions = types.IDMappingOptions{
+		HostUIDMapping: idMappingsOptions.HostUIDMapping,
+		HostGIDMapping: idMappingsOptions.HostGIDMapping,
+		UIDMap:         copySlicePreferringNil(uidMap),
+		GIDMap:         copySlicePreferringNil(gidMap),
 	}
 	if options.Flags == nil {
 		options.Flags = make(map[string]any)

diff --git a/pkg/idtools/idtools.go b/pkg/idtools/idtools.go
--- a/pkg/idtools/idtools.go
+++ b/pkg/idtools/idtools.go
@@ -..,0 +..,46 @@
+// ComposeMaps composes a layer's recorded on-disk ID mapping with a
+// requested container mapping, producing the mapping that a mount-time
+// idmapped mount must apply to that layer's content.
+//
+// layerMap maps logical (in-container) IDs to the IDs physically stored on
+// disk; targetMap maps logical IDs to the IDs the content should appear as.
+// The result maps on-disk IDs (ContainerID field) to target IDs (HostID
+// field).  An empty layerMap means the on-disk content is canonical
+// (identity), so targetMap is returned unchanged.  Logical IDs not covered
+// by layerMap cannot exist on disk and are omitted.
+func ComposeMaps(layerMap, targetMap []IDMap) []IDMap {
+	if len(layerMap) == 0 {
+		return slices.Clone(targetMap)
+	}
+	var out []IDMap
+	for _, t := range targetMap {
+		for _, l := range layerMap {
+			// intersect the logical ranges
+			start := max(t.ContainerID, l.ContainerID)
+			end := min(t.ContainerID+t.Size, l.ContainerID+l.Size)
+			if start >= end {
+				continue
+			}
+			out = append(out, IDMap{
+				ContainerID: l.HostID + (start - l.ContainerID), // on-disk
+				HostID:      t.HostID + (start - t.ContainerID), // target
+				Size:        end - start,
+			})
+		}
+	}
+	sort.Slice(out, func(i, j int) bool { return out[i].ContainerID < out[j].ContainerID })
+	return out
+}
+
+// IsIdentityMapping reports whether every entry maps a range onto itself,
+// i.e. an idmapped mount applying it would be a no-op.
+func IsIdentityMapping(m []IDMap) bool {
+	for _, e := range m {
+		if e.ContainerID != e.HostID {
+			return false
+		}
+	}
+	return true
+}

diff --git a/drivers/driver.go b/drivers/driver.go
--- a/drivers/driver.go
+++ b/drivers/driver.go
@@ -58,6 +58,16 @@ type MountOpts struct {
 	UidMaps []idtools.IDMap //nolint: revive
 	GidMaps []idtools.IDMap //nolint: revive
 	Options []string
+
+	// LayerUIDMaps/LayerGIDMaps optionally provide, keyed by layer ID, the
+	// ID mappings recorded for a layer's on-disk content.  Drivers which
+	// implement mount-time ID shifting must compose these with
+	// UidMaps/GidMaps instead of assuming a 0-based on-disk source.  A
+	// missing entry means the layer's content is canonical (identity
+	// source).
+	LayerUIDMaps map[string][]idtools.IDMap
+	LayerGIDMaps map[string][]idtools.IDMap
 
 	// Volatile specifies whether the container storage can be optimized
 	// at the cost of not syncing all the dirty files in memory.

diff --git a/layers.go b/layers.go
--- a/layers.go
+++ b/layers.go
@@ -1662,6 +1662,29 @@ func (r *layerStore) Mount(id string, options drivers.MountOpts) (string, error
 	if (options.UidMaps != nil || options.GidMaps != nil) && !r.driver.SupportsShifting(options.UidMaps, options.GidMaps) {
 		if !reflect.DeepEqual(options.UidMaps, layer.UIDMap) || !reflect.DeepEqual(options.GidMaps, layer.GIDMap) {
 			return "", fmt.Errorf("cannot mount layer %v: shifting not enabled", layer.ID)
 		}
 	}
+	// Propagate the recorded on-disk mappings of this layer and its
+	// ancestors, so that drivers using mount-time idmapping can compose
+	// them with the requested mapping rather than assume 0-based content.
+	if (options.UidMaps != nil || options.GidMaps != nil) && r.driver.SupportsShifting(options.UidMaps, options.GidMaps) {
+		for l := layer; l != nil; {
+			if len(l.UIDMap) > 0 || len(l.GIDMap) > 0 {
+				if options.LayerUIDMaps == nil {
+					options.LayerUIDMaps = map[string][]idtools.IDMap{}
+					options.LayerGIDMaps = map[string][]idtools.IDMap{}
+				}
+				options.LayerUIDMaps[l.ID] = l.UIDMap
+				options.LayerGIDMaps[l.ID] = l.GIDMap
+			}
+			if l.Parent == "" {
+				break
+			}
+			parent, ok := r.lookup(l.Parent)
+			if !ok {
+				break // parent may live in an additional store; treated as canonical
+			}
+			l = parent
+		}
+	}
 	mountpoint, err := r.driver.Get(id, options)
 	if mountpoint != "" && err == nil {

diff --git a/drivers/overlay/overlay.go b/drivers/overlay/overlay.go
--- a/drivers/overlay/overlay.go
+++ b/drivers/overlay/overlay.go
@@ -1597,14 +1597,36 @@ func (d *Driver) get(id string, disableShifting bool, options graphdriver.Mount
 	}
 
-	idmappedMountProcessPid := -1
-	if needsIDMapping {
-		pid, cleanupFunc, err := idmap.CreateUsernsProcess(options.UidMaps, options.GidMaps)
-		if err != nil {
-			return "", err
-		}
-		idmappedMountProcessPid = pid
-		defer cleanupFunc()
-	}
+	// usernsForLayer returns the pid of a process whose user namespace
+	// expresses the idmapped-mount translation for the given lower, i.e.
+	// the layer's recorded on-disk mapping composed with the requested
+	// mapping.  Layers without recorded maps use the plain requested
+	// mapping (0-based source).  Returns -1 if the composition is the
+	// identity and the layer must not be idmapped at all.  Namespaces are
+	// created lazily and deduplicated per distinct composition.
+	type usernsEntry struct{ pid int }
+	usernsCache := map[string]usernsEntry{}
+	composeKey := func(u, g []idtools.IDMap) string { return fmt.Sprintf("%v/%v", u, g) }
+	defer func() {
+		// cleanups are registered per-namespace below via cleanupFuncs
+	}()
+	var cleanupFuncs []func()
+	defer func() {
+		for _, f := range cleanupFuncs {
+			f()
+		}
+	}()
+	usernsForLayer := func(layerID string) (int, error) {
+		u := idtools.ComposeMaps(options.LayerUIDMaps[layerID], options.UidMaps)
+		g := idtools.ComposeMaps(options.LayerGIDMaps[layerID], options.GidMaps)
+		if idtools.IsIdentityMapping(u) && idtools.IsIdentityMapping(g) {
+			return -1, nil
+		}
+		key := composeKey(u, g)
+		if e, ok := usernsCache[key]; ok {
+			return e.pid, nil
+		}
+		pid, cleanup, err := idmap.CreateUsernsProcess(u, g)
+		if err != nil {
+			return 0, err
+		}
+		cleanupFuncs = append(cleanupFuncs, cleanup)
+		usernsCache[key] = usernsEntry{pid: pid}
+		return pid, nil
+	}
 
 	skipIDMappingLayers := make(map[string]string)
 
@@ -1713,7 +1735,11 @@ func (d *Driver) get(id string, disableShifting bool, options graphdriver.Mount
 			if needsIDMapping {
-				if err := idmap.CreateIDMappedMount(composefsMount, composefsMount, idmappedMountProcessPid); err != nil {
+				pid, err := usernsForLayer(lowerID) // composefs blobs are canonical; identity layer map
+				if err != nil {
+					return "", err
+				}
+				if err := idmap.CreateIDMappedMount(composefsMount, composefsMount, pid); err != nil {
 					return "", fmt.Errorf("create mapped mount for %q: %w", composefsMount, err)
 				}
 				skipIDMappingLayers[composefsMount] = composefsMount
@@ -1779,7 +1805,7 @@ func (d *Driver) get(id string, disableShifting bool, options graphdriver.Mount
 	if needsIDMapping {
 		var newAbsDir []string
-		idMappedMounts := make(map[string]string)
+		idMappedMounts := make(map[string]string) // keyed by source dir + composition
 
 		mappedRoot := filepath.Join(d.home, id, "mapped")
 		if err := os.MkdirAll(mappedRoot, 0o700); err != nil {
@@ -1796,17 +1822,30 @@ func (d *Driver) get(id string, disableShifting bool, options graphdriver.Mount
 			if _, ok := skipIDMappingLayers[absLower]; ok {
 				newAbsDir = append(newAbsDir, absLower)
 				continue
 			}
 
-			root, found := idMappedMounts[mappedMountSrc]
+			// Identify the layer this lower belongs to (".../<id>/diff[N]")
+			// to look up its recorded on-disk mapping.
+			lowerLayerID := filepath.Base(filepath.Dir(absLower))
+			pid, err := usernsForLayer(lowerLayerID)
+			if err != nil {
+				return "", err
+			}
+			if pid == -1 {
+				// already chowned to the requested mapping: identity, do not idmap
+				newAbsDir = append(newAbsDir, absLower)
+				continue
+			}
+			cacheKey := mappedMountSrc + "\x00" + strconv.Itoa(pid)
+			root, found := idMappedMounts[cacheKey]
 			if !found {
 				root = filepath.Join(mappedRoot, fmt.Sprintf("%d", c))
 				c++
-				if err := idmap.CreateIDMappedMount(mappedMountSrc, root, idmappedMountProcessPid); err != nil {
+				if err := idmap.CreateIDMappedMount(mappedMountSrc, root, pid); err != nil {
 					return "", fmt.Errorf("create mapped mount for %q on %q: %w", mappedMountSrc, root, err)
 				}
-				idMappedMounts[mappedMountSrc] = root
+				idMappedMounts[cacheKey] = root
 
 				// overlay takes a reference on the mount, so it is safe to unmount
 				// the mapped idmounts as soon as the final overlay file system is mounted.
-- 
2.x

[lyphyser]

export _CONTAINERS_OVERLAY_DISABLE_IDMAP=yes does indeed workaround the issue, but obviously is completely unacceptable as a permanent solution since it results in iterating over everything in storage to change their UID/GID

[Luap99]

As CNCF project we follow the CNCF Code of Conduct, the language you used is not an acceptable tone to talk to any project maintainer so I removed that text.
https://github.com/cncf/foundation/blob/main/code-of-conduct.md

Also note our LLM usage policy https://github.com/podman-container-tools/podman/blob/main/LLM_POLICY.md#no-llm-generated-direct-communication

So please be respectful towards others, writing something like this will not make any maintainer want to work on this voluntary.