Buildah Cache Selection with --cache-from and Remote Image Layers

[dinhhoang1995]

Hello Buildah Community,

I'm seeking clarification on how Buildah selects cached layers when using the --cache-from option, particularly when the cache source is a remote image from a registry like Harbor.

Scenario:

I have a base image, <MY-HARBOR-REPOSITORY>/ubi9:latest, hosted in Harbor. I perform initial builds of an image with a RUN dnf install tomcat instruction.

1. Initial Build (No Cache):
   I build and push an image tagged my-image-no-cache:v1:

   buildah build -f Dockerfile -t $MY-HARBOR-REPOSITORY/my-image-no-cache:v1 .
   buildah push $MY-HARBOR-REPOSITORY/my-image-no-cache:v1

   This Dockerfile contains:

   FROM <MY-HARBOR-REPOSITORY>/ubi9:latest
   RUN dnf install tomcat

2. Subsequent Build (Updating Package):
   Periodically, I rebuild this image to pick up the latest version of tomcat, pushing it as my-image-no-cache:v2:

   buildah build -f Dockerfile -t $MY-HARBOR-REPOSITORY/my-image-no-cache:v2 .
   buildah push $MY-HARBOR-REPOSITORY/my-image-no-cache:v2

3. Build with --cache-from:
   Now, I have a second Dockerfile (Dockerfile.simple) that includes additional steps:

   FROM <MY-HARBOR-REPOSITORY>/ubi9:latest
   RUN dnf install tomcat
   COPY entrypoint.sh /var/tmp/entrypoint.sh
   ENTRYPOINT ["/var/tmp/entrypoint.sh"]

   When building Dockerfile.simple, I use --cache-from pointing to the image name (not a specific tag) that I've been updating:

   buildah build -f Dockerfile.simple --layers --cache-from $$MY-HARBOR-REPOSITORY/my-image-no-cache -t $$MY-HARBOR-REPOSITORY/my-image-build-with-cache:v1 .

The Question:

When Buildah encounters the RUN dnf install tomcat instruction in Dockerfile.simple, which cached layer will it attempt to use from the image specified by --cache-from $MY-HARBOR-REPOSITORY/my-image-no-cache?

Specifically, if I have pushed both v1 and v2 of my-image-no-cache to Harbor, will Buildah:

• Select the cached layer from the older v1 build?
• Or, will it prioritize and select the cached layer from the most recently pushed version (e.g., v2) of my-image-no-cache that matches the layer history?

My expectation is that it should attempt to use the cached layer from the latest available version of my-image-no-cache that provides a matching history for the dnf install tomcat step. Could someone confirm if this is the expected behavior and how Buildah determines which remote cache layer to use in such scenarios?

Thank you for your time and insights!

[nalind]

The argument to --cache-from is, currently, only ever interpreted as a repository name, in which it searches for images tagged using a unique value computed from information about the step for which the cache is being searched and the steps which preceded it.
The expectation is that the cached image, if there is a suitable one, was pushed there during a previous build which was run with the same repository specified to a --cache-to flag, and tagged with the matching value.

[dinhhoang1995]

Hi @nalind,

Following up on our discussion about --cache-from, I'd like to get a bit more granular on the specifics of how --cache-to manages intermediate layers, particularly when identical content hashes are generated across different builds.

My understanding is that when Buildah encounters an instruction like RUN dnf install tomcat, it computes a content hash for that step (let's call it HASH_TOMCAT). If --cache-to is enabled and configured to push to a registry (e.g., $MY-HARBOR-REPOSITORY/build-cache), Buildah pushes a layer corresponding to HASH_TOMCAT to that repository, tagged with HASH_TOMCAT.

Now, consider this scenario:

1. Build 1: An image is built, and the RUN dnf install tomcat step results in HASH_TOMCAT. Using --cache-to, a layer tagged HASH_TOMCAT is pushed to $MY-HARBOR-REPOSITORY/build-cache.
2. Build 2: A subsequent build is run after a period. The base image is the same, and the RUN dnf install tomcat instruction is also identical. Therefore, Buildah computes HASH_TOMCAT again for this step. The --cache-to option is also used for this Build 2. However, the tomcat package has since evolved.

My question is about how Buildah handles the remote cache repository ($MY-HARBOR-REPOSITORY/build-cache) during Build 2 when it again calculates HASH_TOMCAT. Specifically:

• Does Buildah attempt to push a new, distinct layer for HASH_TOMCAT in Build 2? If so, how does it manage the tag HASH_TOMCAT in the registry? Does it overwrite the existing tag, create a new tag or something else?
• Or, does Buildah detect that a layer with HASH_TOMCAT already exists in the remote cache repository from Build 1, and therefore does not push a redundant layer?

The core of my confusion is understanding the exact mechanism Buildah employs when duplicate content hashes are generated across sequential builds using --cache-to. The goal is to ensure that HASH_TOMCAT remains reliably discoverable for subsequent --cache-from operations.

Thanks for clarifying this aspect of Buildah's cache management.

[nalind]

The builder is pushing the image that was committed for that step to a tag in the specified repository. If a later run looks for an image using the same tag, it will find the image that was most recently pushed to that tag. The registry doesn't "know" that this has anything to do with caching, it's doing what registries are expected to do.

When pushing the image, if the most recent step produced a layer diff, the layer diff will be pushed if an exact copy of it doesn't already exist in the registry. That behavior is a general push-time optimization that the image library provides - it's going to do the same for all of the image's layers, taking into account information that it keeps track of in the blob info cache that it maintains when it can.

[dinhhoang1995]

Thank you for the replies 👍