Make docker images fully reproducible [DI-741] by ldziedziul · Pull Request #1236 · hazelcast/hazelcast-docker

ldziedziul · 2026-02-18T21:56:55Z

Both OSS and Enterprise Dockerfiles now produce bit-for-bit identical images across clean builds. Every layer hash is deterministic regardless of when or where the build runs.

Fixes https://hazelcast.atlassian.net/browse/DI-741

Benefits

Reproducible builds - identical image hashes across clean builds, regardless of when or where the build runs
Reduced storage - JFrog deduplicates identical layers, significantly reducing storage footprint
Faster pushes - rebuilds push only changed layers, all unchanged layers show "already exists"
Independent layer updates - updating hazelcast binaries, base system/JDK, or system libraries each only affects its own layer; the rest are reused without redundant storage or transfer
Cross-platform layer sharing - hazelcast binaries layer is shared across all architectures (amd64, arm64, s390x)

Changes

Reproducible distribution layer (OSS + EE)

Extract distribution into /build_root/ in a staging stage and COPY --link into the final image, producing a platform-independent layer that is shared across architectures
Normalize timestamps via SOURCE_DATE_EPOCH and --output type=docker,rewrite-timestamp=true
syntax=docker/dockerfile:1.7 - enables BuildKit's COPY --link support

Enterprise

Consolidate user creation, package upgrade, and cleanup into a single RUN

OSS

Extract apk upgrade into a separate layer with its own timestamp normalization
Regenerate Java cacerts with SOURCE_DATE_EPOCH for reproducibility

How it works

The COPY --link flag makes the distribution layer independent of the base image, allowing it to be shared across platforms.
rewrite-timestamp=true sets all files timestamps to SOURCE_DATE_EPOCH value

Verification

EE

Build image (requires local hazelcast-enterprise/hazelcast-enterprise-distribution-5.6.0.zip)

SOURCE_DATE_EPOCH=0 docker buildx build --progress=plain -f hazelcast-enterprise/Dockerfile hazelcast-enterprise --build-arg HAZELCAST_ZIP_FILE_NAME=hazelcast-enterprise-distribution-5.6.0.zip -t hazelcast.jfrog.io/sandbox-docker-preprod/lukasz-test-ee:5.6.0-1 --output type=docker,rewrite-timestamp=true

Initial push image to JFrog - all layers need push

docker push docker push hazelcast.jfrog.io/sandbox-docker-preprod/lukasz-test-ee:5.6.0-1
The push refers to repository [hazelcast.jfrog.io/sandbox-docker-preprod/lukasz-test-ee]
5f70bf18a086: Pushed
36a111f6d998: Pushed
989832e1d274: Pushed
b2366c6e0bab: Pushed
d77edb45729d: Pushed
eeeef98956cd: Pushed
b68de875dd7e: Pushed
3c3acb3a7af5: Pushed
4929137c7f9d: Pushed
5.6.0-1: digest: sha256:59fbeb2f8cbe4f3f1781df93d0e1b4e6c6426c63a05f6c7ecb23ed4314df90fb size: 220

Clean local layers

docker system prune -a

Rebuild image from scratch

SOURCE_DATE_EPOCH=0 docker buildx build --progress=plain -f hazelcast-enterprise/Dockerfile hazelcast-enterprise --build-arg HAZELCAST_ZIP_FILE_NAME=hazelcast-enterprise-distribution-5.6.0.zip -t hazelcast.jfrog.io/sandbox-docker-preprod/lukasz-test-ee:5.6.0-2 --output type=docker,rewrite-timestamp=true

Push image to JFrog - all layers already exists


docker push hazelcast.jfrog.io/sandbox-docker-preprod/lukasz-test-ee:5.6.0-2
The push refers to repository [hazelcast.jfrog.io/sandbox-docker-preprod/lukasz-test-ee]
5f70bf18a086: Layer already exists
36a111f6d998: Layer already exists
989832e1d274: Layer already exists
b2366c6e0bab: Layer already exists
d77edb45729d: Layer already exists
eeeef98956cd: Layer already exists
b68de875dd7e: Layer already exists
3c3acb3a7af5: Layer already exists
4929137c7f9d: Layer already exists
5.6.0-2: digest: sha256:c18647b6f0e8e385cd2e9766ca36c69a259d27e7589c36d2fd709065728e11d4 size: 2206

OSS

Build image (requires local hazelcast-oss/hazelcast-distribution-5.6.0.zip)

SOURCE_DATE_EPOCH=0 docker buildx build --progress=plain -f hazelcast-oss/Dockerfile hazelcast-oss --build-arg HAZELCAST_ZIP_FILE_NAME=hazelcast-distribution-5.6.0.zip -t hazelcast.jfrog.io/sandbox-docker-preprod/lukasz-test-oss:5.6.0-1 --output type=docker,rewrite-timestamp=true

Initial push image to JFrog - all layers need push

docker push hazelcast.jfrog.io/sandbox-docker-preprod/lukasz-test-oss:5.6.0-1
The push refers to repository [hazelcast.jfrog.io/sandbox-docker-preprod/lukasz-test-oss]
5f70bf18a086: Pushed
c99808c7ec85: Pushed
86895bee41d6: Pushed
d1356c4de94c: Pushed
45f3ea5848e8: Pushed
5.6.0: digest: sha256:44c0c2625f1d6417cf290169ce375bb6ef5a0421cc6dd128731c9adc434abd82 size: 1369

Clean local layers

docker system prune -a

Rebuild image from scratch

SOURCE_DATE_EPOCH=0 docker buildx build --progress=plain -f hazelcast-oss/Dockerfile hazelcast-oss --build-arg HAZELCAST_ZIP_FILE_NAME=hazelcast-distribution-5.6.0.zip -t hazelcast.jfrog.io/sandbox-docker-preprod/lukasz-test-oss:5.6.0-2  --output type=docker,rewrite-timestamp=true

Push image to JFrog - all layers already exists

docker push hazelcast.jfrog.io/sandbox-docker-preprod/lukasz-test-oss:5.6.0-2
The push refers to repository [hazelcast.jfrog.io/sandbox-docker-preprod/lukasz-test-oss]
5f70bf18a086: Layer already exists
c99808c7ec85: Layer already exists
86895bee41d6: Layer already exists
d1356c4de94c: Layer already exists
45f3ea5848e8: Layer already exists
5.6.0-2: digest: sha256:14fd1ab8e1bad71d990d41f87bb2c7195f339199e3eabdd211fc55386bf3b26a size: 1369

Automated CI verification

A new reusable workflow (.github/workflows/verify-layer-reproducibility.yml) runs on every PR via build-pr.yml. It builds each Dockerfile twice with --no-cache and compares all layer digests using docker inspect. If any layer differs between builds, the job fails.

JackPGreen · 2026-03-11T16:23:28Z

+
+            . .github/scripts/rhel.functions.sh
+
+            # Look up the existing image ID from Pyxis so downstream steps can still verify tags


Rather than trying-and-failing, could we lookup digest first and if not found then submit via preflight? Avoids ignoring preflight errors.

Good suggestion and makes sense not to hide other errors
I agree if its possible.
If not possible then we should fail if there are errors other than is published, published image can't be updated

Good idea, reworked: a70f24a

First build:

Rebuilding:

nishaatr · 2026-03-11T16:57:46Z

+
+            . .github/scripts/rhel.functions.sh
+
+            # Look up the existing image ID from Pyxis so downstream steps can still verify tags


Good suggestion and makes sense not to hide other errors
I agree if its possible.
If not possible then we should fail if there are errors other than is published, published image can't be updated

nishaatr · 2026-03-11T17:11:10Z

+
+          # Reproducible builds produce identical digests across rebuilds.
+          # If the same digest was already certified and published by a previous run,
+          # Pyxis rejects the submission with "is published, published image can't be updated".


Does this mean preflight check rejects already published images but continues with others?
as I don't see another call to preflight check
if thats the case then best to reflect in the comment as currently it implies the whole preflight fails

nishaatr · 2026-03-11T17:19:47Z

+
+          echo "${PREFLIGHT_OUTPUT}"
+
+          # Reproducible builds produce identical digests across rebuilds.


Little confused about what IMAGE, IMAGE_IDS and DIGEST actually mean exactly
How does this relate to layers? Is a layer an IMAGE? Does each architecture equate to an IMAGE?
would be good if you can briefly explain how are these connected?

ps: i looked at Sprint Review diagram but still not sure

See https://docs.docker.com/dhi/core-concepts/digests/ and https://docs.docker.com/get-started/docker-concepts/building-images/understanding-image-layers/

nishaatr · 2026-03-11T17:23:22Z

+            # Look up the existing image ID from Pyxis so downstream steps can still verify tags
+            DIGEST=$(grep -oP 'sha256:[a-f0-9]{64}' <<< "${PREFLIGHT_OUTPUT}" | head -1 || true)
+            if [[ -n "${DIGEST}" ]]; then
+              EXISTING_IMAGE_ID=$(get_image_id_by_digest \


this implies there is only single previously published DIGEST?
But we now have multiple layers that might have been published previously
I think I need a lesson as little sketchy on this. More than happy to discuss and clarify in stand-up to save you typing

Discussed on standup

nishaatr · 2026-03-11T17:33:35Z

In PR description I read this

But don't see this anymore. I think this was before rewrite-timestamp?

nishaatr

@ldziedziul
I have done first pass
Have some clarification questions so will do second pass after that

Also, has this been proven to work in the field?
May be not possible or may be needs to go live once before we see the net effect in local builds, PR builds and rebuilds (when base image changes etc)?

Co-authored-by: Jack Green <jack.green@hazelcast.com>

Co-authored-by: Nishaat Rajabali <12186256+nishaatr@users.noreply.github.com>

Reproducible builds produce identical digests across rebuilds. Before running preflight, check Pyxis for an existing certified image with the same digest. If found, skip the entire certification flow (unique tag push, preflight scan, await publishing, tag copying) and only verify that tags are published.

JackPGreen · 2026-03-12T11:36:45Z

+            "${REDHAT_ROBOT_RHEL_API_KEY}")
+
+          if [[ -n "${EXISTING_IMAGE_ID}" && "${EXISTING_IMAGE_ID}" != "null" ]]; then
+            echonotice "Image ${{ vars.DOCKERHUB_NAMESPACE }}/${{ vars.PREPROD_REGISTRY }}-${{ vars.DOCKERHUB_EE_IMAGE_NAME }}:${{ steps.get-tags-to-push.outputs.primary-tag }} already published with image ID: ${EXISTING_IMAGE_ID}, skipping certification"


We should be explicit it's not just the same image, but identical digest.
Also we should specify we aren't just skipping certification - publish, too.

Reworded d26cabd

JackPGreen · 2026-03-12T11:38:49Z

          PREFLIGHT_OUTPUT=$(preflight check container "${{ steps.scan_registry_secrets.outputs.SCAN_REPOSITORY }}:${{ steps.unique-destination-tag.outputs.tag }}" \
            --submit --pyxis-api-token=${REDHAT_ROBOT_RHEL_API_KEY} \
            --certification-component-id=${{ steps.scan_registry_secrets.outputs.PROJECT_ID }} \
            --docker-config ~/.docker/config.json \
            2>&1)

-          echodebug "${PREFLIGHT_OUTPUT}"
+          echo "${PREFLIGHT_OUTPUT}"


I don't understand why we changed this - but I'm aware this whole thing (...that I wrote...) is bad as in the case of errors, you get no output at all.

I think it's crucial information from this step and it's not that long

JackPGreen · 2026-03-12T11:39:42Z

          . .github/scripts/rhel.functions.sh

-          jq --raw-output '.[]' <<< '${{ steps.preflight.outputs.IMAGE_IDS }}' | while read IMAGE_ID; do
+          jq --raw-output '.[]' <<< '${{ env.IMAGE_IDS }}' | while read IMAGE_ID; do


Suggested change

jq --raw-output '.[]' <<< '${{ env.IMAGE_IDS }}' | while read IMAGE_ID; do

jq --raw-output '.[]' <<< '${IMAGE_IDS}' | while read IMAGE_ID; do

(biased, don't like env)
This uses the "escaped" version so is safer / less likely to break. And cleaner.

changed: 591b144

ldziedziul · 2026-03-14T09:25:47Z

🎉

...actually, did we resolve RedHat?

I see the build fails and the output isn't preserved which I think you had some ideas to fix as well:

Fixed 2b2aa91

JackPGreen

I think we're nearly there.

JackPGreen · 2026-03-14T09:33:51Z

+          # Extract the linux/amd64 manifest digest, matching how the image is pushed 
+          # to RedHat Container Registry (single-arch, not manifest list)
+          DIGEST=$(skopeo inspect --raw \
+            docker://${FULL_PRIMARY_TAG} \
+            | jq -r '.manifests[] | select(.platform.os == "linux" and .platform.architecture == "amd64") | .digest')
+          echo "Image ${FULL_PRIMARY_TAG} for linux/amd64 has digest: ${DIGEST}"


Could we avoid hardcoded platform here?
I.E. if we change the pushed platforms elsewhere this could break.

Fixed: 20549f0

JackPGreen · 2026-03-14T09:36:40Z

+              --override-os linux \
+              --override-arch amd64 \


Why do we need this now? Or it just fluke that we happen to run this job on an amd64/linux platform?

Because now we copy remaining tags using ${{ vars.DOCKERHUB_NAMESPACE }}/${{ vars.PREPROD_REGISTRY }}-${{ vars.DOCKERHUB_EE_IMAGE_NAME }}:${{ steps.get-tags-to-push.outputs.primary-tag }}(multi-arch) as the source, instead of ${{ steps.scan_registry_secrets.outputs.SCAN_REPOSITORY }}:${{ steps.unique-destination-tag.outputs.tag }} \. This way we can skip creation of unique tag when republishing the same content

sonarqubecloud · 2026-03-16T13:21:38Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

JackPGreen

🚀

ldziedziul self-assigned this Feb 18, 2026

ldziedziul temporarily deployed to test February 18, 2026 21:57 — with GitHub Actions Inactive

ldziedziul had a problem deploying to test February 18, 2026 21:57 — with GitHub Actions Error

ldziedziul had a problem deploying to test February 18, 2026 21:57 — with GitHub Actions Failure

ldziedziul temporarily deployed to test February 18, 2026 21:57 — with GitHub Actions Inactive

ldziedziul had a problem deploying to test February 18, 2026 21:57 — with GitHub Actions Error

ldziedziul temporarily deployed to test February 18, 2026 21:57 — with GitHub Actions Inactive

ldziedziul temporarily deployed to test February 18, 2026 22:01 — with GitHub Actions Inactive

ldziedziul temporarily deployed to test February 18, 2026 22:02 — with GitHub Actions Inactive

ldziedziul temporarily deployed to test February 18, 2026 22:36 — with GitHub Actions Inactive

ldziedziul temporarily deployed to test February 18, 2026 22:40 — with GitHub Actions Inactive

JackPGreen reviewed Mar 11, 2026

View reviewed changes

Comment thread .github/scripts/rhel.functions.sh

JackPGreen reviewed Mar 11, 2026

View reviewed changes

nishaatr reviewed Mar 11, 2026

View reviewed changes

nishaatr requested changes Mar 11, 2026

View reviewed changes

ldziedziul and others added 3 commits March 11, 2026 21:48

Update .github/scripts/rhel.functions.sh

f3ed167

Co-authored-by: Jack Green <jack.green@hazelcast.com>

Update .github/workflows/ee-nlc-tag-package.yml

7b686f6

Co-authored-by: Nishaat Rajabali <12186256+nishaatr@users.noreply.github.com>

JackPGreen reviewed Mar 12, 2026

View reviewed changes

Comment thread .github/workflows/tag_image_promote_rhel.yml

JackPGreen reviewed Mar 12, 2026

View reviewed changes

Comment thread .github/workflows/tag_image_promote_rhel.yml

JackPGreen reviewed Mar 12, 2026

View reviewed changes

JackPGreen requested changes Mar 12, 2026

View reviewed changes

ldziedziul added 8 commits March 12, 2026 13:35

Do not exit after preflight check container to see the PREFLIGHT_OUTPUT

2b2aa91

Fix digest lookup to use linux/amd64 manifest digest

901b162

Improve logging

24258a9

Extract amd64 digest from manifest list

3057fe4

Improve logging

d26cabd

Use env variables directly

591b144

Skip deleted images in lookup by digest

1ad8b4e

Copy tags from DockerHub source to avoid multiple unique tags per image

5d77c97

JackPGreen requested changes Mar 14, 2026

View reviewed changes

ldziedziul added 2 commits March 16, 2026 13:58

Extract hardcoded platform into PLATFORM_OS and PLATFORM_ARCH env vars

20549f0

Add comment

314680e

JackPGreen approved these changes Mar 16, 2026

View reviewed changes

nishaatr approved these changes Mar 17, 2026

View reviewed changes

JackPGreen mentioned this pull request Mar 26, 2026

Fix vulnerability_scan_subworkflow execution on maintenance branches #1256

Merged

JackPGreen mentioned this pull request Jun 19, 2026

Fix reproducible builds + Docker rebuilds + RedHat #1281

Merged


		. .github/scripts/rhel.functions.sh

		# Look up the existing image ID from Pyxis so downstream steps can still verify tags


		echo "${PREFLIGHT_OUTPUT}"

		# Reproducible builds produce identical digests across rebuilds.

	jq --raw-output '.[]' <<< '${{ env.IMAGE_IDS }}' \| while read IMAGE_ID; do
	jq --raw-output '.[]' <<< '${IMAGE_IDS}' \| while read IMAGE_ID; do

Uh oh!

Conversation

ldziedziul commented Feb 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benefits

Changes

Reproducible distribution layer (OSS + EE)

Enterprise

OSS

How it works

Verification

EE

OSS

Automated CI verification

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nishaatr left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ldziedziul commented Mar 14, 2026

Uh oh!

JackPGreen left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud Bot commented Mar 16, 2026

Quality Gate passed

Uh oh!

JackPGreen left a comment

ldziedziul commented Feb 18, 2026 •

edited

Loading