Skip to content

fix image build not using cache #559

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

fix image build not using cache #559

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
a87861e
fix image build not using cache
fulminemizzega Jun 20, 2025
3cb64f5
document new defaults in images_build.py
fulminemizzega Jun 23, 2025
512c861
use kwargs.get default parameter for defaults
fulminemizzega Jun 23, 2025
9e37dc0
fix wrong mock URL in test_build unit test
fulminemizzega Jun 24, 2025
63f91a7
add unit test for images.build default parameters
fulminemizzega Jun 24, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 7 additions & 7 deletions podman/domain/images_build.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,9 +63,10 @@ def build(self, **kwargs) -> tuple[Image, Iterator[bytes]]:
isolation (str) – Isolation technology used during build. (ignored)
use_config_proxy (bool) – (ignored)
http_proxy (bool) - Inject http proxy environment variables into container (Podman only)
layers (bool) - Cache intermediate layers during build.
layers (bool) - Cache intermediate layers during build. Default True.
output (str) - specifies if any custom build output is selected for following build.
outputformat (str) - The format of the output image's manifest and configuration data.
Default to "application/vnd.oci.image.manifest.v1+json" (OCI format).

Returns:
first item is the podman.domain.images.Image built
Expand Down Expand Up @@ -167,7 +168,7 @@ def _render_params(kwargs) -> dict[str, list[Any]]:
raise PodmanError("Custom encoding not supported when gzip enabled.")

params = {
"dockerfile": kwargs.get("dockerfile"),
"dockerfile": kwargs.get("dockerfile", f".containerfile.{random.getrandbits(160):x}"),
"forcerm": kwargs.get("forcerm"),
"httpproxy": kwargs.get("http_proxy"),
"networkmode": kwargs.get("network_mode"),
Expand All @@ -181,9 +182,11 @@ def _render_params(kwargs) -> dict[str, list[Any]]:
"squash": kwargs.get("squash"),
"t": kwargs.get("tag"),
"target": kwargs.get("target"),
"layers": kwargs.get("layers"),
"layers": kwargs.get("layers", True),
"output": kwargs.get("output"),
"outputformat": kwargs.get("outputformat"),
"outputformat": kwargs.get(
"outputformat", "application/vnd.oci.image.manifest.v1+json"
),
}

if "buildargs" in kwargs:
Expand All @@ -204,8 +207,5 @@ def _render_params(kwargs) -> dict[str, list[Any]]:
if "labels" in kwargs:
params["labels"] = json.dumps(kwargs.get("labels"))

if params["dockerfile"] is None:
params["dockerfile"] = f".containerfile.{random.getrandbits(160):x}"

# Remove any unset parameters
return dict(filter(lambda i: i[1] is not None, params.items()))
14 changes: 14 additions & 0 deletions podman/tests/integration/test_images.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@
"""Images integration tests."""

import io
import json
import platform
import tarfile
import types
Expand Down Expand Up @@ -144,6 +145,19 @@ def test_build(self):
self.assertIsNotNone(image)
self.assertIsNotNone(image.id)

def test_build_cache(self):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

more tests are probably needed to ensure that the defaults are passed, and images.build also accepts the other parameters

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are defaults like those above, and then there is dockerfile that is generated randomly... maybe this is something that should be done in a unit test?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are defaults like those above, and then there is dockerfile that is generated randomly... maybe this is something that should be done in a unit test?

I think you can do all through integration tests by inspecting the request that are passed. if you want I can write you some pseudo-code to help understand how I would design it

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes please, I’m not familiar with this kind of stuff… how would I inspect what is requested? I’ve seen how it is mocked in unit tests, is it related?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes please, I’m not familiar with this kind of stuff… how would I inspect what is requested? I’ve seen how it is mocked in unit tests, is it related?

Actually my bad, I was looking into many things at the same time and got confused. You are correct, unit tests is the place where you check you function. Also yes, you said it right, you need to mock a request and test that your image build calls the parameters correctly in the default/non-default cases.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

more tests are probably needed to ensure that the defaults are passed, and images.build also accepts the other parameters

About the other parameters, you mean that, besides the new defaults, there should be a test should that uses all of them and inspects the request? In that case, there is another issue: while experimenting with the parameter-argument conversion (see above), I discovered that _render_params parses the "remote" key, it is not documented, it can not be used alone because _render_params requires either "path" or "fileobj" (and both are meaningless if remote is used), but other than that the build function is able to handle this, because body is initialized with None, and when all the if/elif fail it stays None which is right for remote. But if dockerfile is None, then it is generated randomly in _render_params and this will be yet another issue: "If the URI points to a tarball and the dockerfile parameter is also specified, there must be a file with the corresponding path inside the tarball" from https://docs.podman.io/en/latest/_static/api.html#tag/images/operation/ImageBuildLibpod
I understand this is going quite out of the "cache" issue scope, I apologize in advance.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added a unit test that checks the default parameters for images.build, with it also I corrected the mock URL for two other test cases

"""Check that building twice the same image uses caching"""
buffer = io.StringIO("""FROM quay.io/libpod/alpine_labels:latest\nLABEL test=value""")
image, _ = self.client.images.build(fileobj=buffer)
buffer.seek(0)
_, stream = self.client.images.build(fileobj=buffer)
for line in stream:
# Search for a line with contents "-> Using cache <image id>"
parsed = json.loads(line)['stream']
if "Using cache" in parsed:
break
self.assertEqual(parsed.split()[3], image.id)

def test_build_with_context(self):
context = io.BytesIO()
with tarfile.open(fileobj=context, mode="w") as tar:
Expand Down
70 changes: 52 additions & 18 deletions podman/tests/unit/test_build.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import io
import requests
import json
import unittest

Expand All @@ -16,6 +17,20 @@
from podman.domain.images import Image
from podman.errors import BuildError, DockerException

good_image_id = "032b8b2855fc"
good_stream = [
{"stream": " ---\u003e a9eb17255234"},
{"stream": "Step 1 : VOLUME /data"},
{"stream": " ---\u003e Running in abdc1e6896c6"},
{"stream": " ---\u003e 713bca62012e"},
{"stream": "Removing intermediate container abdc1e6896c6"},
{"stream": "Step 2 : CMD [\"/bin/sh\"]"},
{"stream": " ---\u003e Running in dba30f2a1a7e"},
{"stream": " ---\u003e 032b8b2855fc"},
{"stream": "Removing intermediate container dba30f2a1a7e"},
{"stream": f"{good_image_id}\n"},
]


class TestBuildCase(unittest.TestCase):
"""Test ImagesManager build().
Expand All @@ -41,19 +56,7 @@ def test_build(self, mock_prepare_containerfile, mock_create_tar):
mock_prepare_containerfile.return_value = "Containerfile"
mock_create_tar.return_value = b"This is a mocked tarball."

stream = [
{"stream": " ---\u003e a9eb17255234"},
{"stream": "Step 1 : VOLUME /data"},
{"stream": " ---\u003e Running in abdc1e6896c6"},
{"stream": " ---\u003e 713bca62012e"},
{"stream": "Removing intermediate container abdc1e6896c6"},
{"stream": "Step 2 : CMD [\"/bin/sh\"]"},
{"stream": " ---\u003e Running in dba30f2a1a7e"},
{"stream": " ---\u003e 032b8b2855fc"},
{"stream": "Removing intermediate container dba30f2a1a7e"},
{"stream": "032b8b2855fc\n"},
]

stream = good_stream
buffer = io.StringIO()
for entry in stream:
buffer.write(json.JSONEncoder().encode(entry))
Expand All @@ -70,9 +73,9 @@ def test_build(self, mock_prepare_containerfile, mock_create_tar):
text=buffer.getvalue(),
)
mock.get(
tests.LIBPOD_URL + "/images/032b8b2855fc/json",
tests.LIBPOD_URL + f"/images/{good_image_id}/json",
json={
"Id": "032b8b2855fc",
"Id": good_image_id,
"ParentId": "",
"RepoTags": ["fedora:latest", "fedora:33", "<none>:<none>"],
"RepoDigests": [
Expand Down Expand Up @@ -100,7 +103,7 @@ def test_build(self, mock_prepare_containerfile, mock_create_tar):
labels={"Unittest": "true"},
)
self.assertIsInstance(image, Image)
self.assertEqual(image.id, "032b8b2855fc")
self.assertEqual(image.id, good_image_id)
self.assertIsInstance(logs, Iterable)

@patch.object(api, "create_tar")
Expand Down Expand Up @@ -130,16 +133,47 @@ def test_build_logged_error(self, mock_prepare_containerfile, mock_create_tar):

@requests_mock.Mocker()
def test_build_no_context(self, mock):
mock.post(tests.LIBPOD_URL + "/images/build")
mock.post(tests.LIBPOD_URL + "/build")
with self.assertRaises(TypeError):
self.client.images.build()

@requests_mock.Mocker()
def test_build_encoding(self, mock):
mock.post(tests.LIBPOD_URL + "/images/build")
mock.post(tests.LIBPOD_URL + "/build")
with self.assertRaises(DockerException):
self.client.images.build(path="/root", gzip=True, encoding="utf-8")

@patch.object(api, "create_tar")
@patch.object(api, "prepare_containerfile")
def test_build_defaults(self, mock_prepare_containerfile, mock_create_tar):
"""Check the defaults used by images.build"""
mock_prepare_containerfile.return_value = "Containerfile"
mock_create_tar.return_value = b"This is a mocked tarball."

stream = good_stream
buffer = io.StringIO()
for entry in stream:
buffer.write(json.dumps(entry))
buffer.write("\n")

with requests_mock.Mocker() as mock:
query = "?outputformat=" + (
requests.utils.quote("application/vnd.oci.image.manifest.v1+json", safe='')
+ "&layers=True"
)
mock.post(
tests.LIBPOD_URL + "/build" + query,
text=buffer.getvalue(),
)
mock.get(
tests.LIBPOD_URL + f"/images/{good_image_id}/json",
json={
"Id": "unittest",
},
)
img, _ = self.client.images.build(path="/tmp/context_dir")
assert img.id == "unittest"


if __name__ == '__main__':
unittest.main()