feat: Add reproducible builds release workflows and push images to DockerHub #7614

MoeMahhouk · 2025-06-16T23:00:00Z

Issue Addressed

This pull request introduces workflows and updates to ensure reproducible builds for the Lighthouse project. It adds two GitHub Actions workflows for building and testing reproducible Docker images and binaries, updates the Makefile to streamline reproducible build configurations, and modifies the Dockerfile.reproducible to align with the new build process. Additionally, it removes the reproducible profile from Cargo.toml.

Proposed Changes

New GitHub Actions Workflows:

.github/workflows/release-reproducible.yml: Adds a workflow to build and push reproducible multi-architecture Docker images for releases, including support for dry runs.
.github/workflows/reproducible-build.yml: Adds a workflow to test reproducibility by comparing binaries built twice for x86_64 and aarch64 architectures, with failure reporting and artifact uploads.

Build Configuration Updates:

Makefile: Refactors reproducible build targets, centralizes environment variables for reproducibility, and updates Docker build arguments for x86_64 and aarch64 architectures.
Dockerfile.reproducible: Updates the base Rust image to version 1.86, removes hardcoded reproducibility settings, and delegates build logic to the Makefile.

Profile Removal:

Cargo.toml: Removes the reproducible profile, simplifying build configurations and relying on external tooling for reproducibility.

Additional Info

This is mainly a follow up to this work #6799 where I refine the reproducible build configuration to simplify the CI workflow to generate the reproducible images and pushes them to DockerHub. I also added a cron job workflow (inspired from the Reth repo) that checks every two days or pull requests that touches files that might affect reproducibility to catch potential regressions.
In case, this is too much, let me know and I can create a separate PR for this to be merged later when necessary

close #7486
close #7485

cla-assistant · 2025-06-16T23:00:16Z

All committers have signed the CLA.

cla-assistant · 2025-06-16T23:00:16Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ MoeMahhouk
❌ Ubuntu

Ubuntu seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

chong-he · 2025-06-19T07:48:09Z

Doing some testing on this, will post the comment when ready

mergify · 2025-06-23T23:32:50Z

Some required checks have failed. Could you please take a look @MoeMahhouk? 🙏

chong-he

Thanks for the PR. I left some comments after doing some testing

.github/workflows/reproducible-build.yml

.github/workflows/release-reproducible.yml

.github/workflows/reproducible-build.yml

MoeMahhouk · 2025-06-24T15:57:23Z

Thanks for the PR. I left some comments after doing some testing

Thanks for reviewing it.
I will do some rework and refinements to this PR to fix the issues and address your comments.

chong-he · 2025-07-01T13:04:21Z

Thanks for the PR. I left some comments after doing some testing

Thanks for reviewing it. I will do some rework and refinements to this PR to fix the issues and address your comments.

Is there a reason why is this PR still a draft?

MoeMahhouk · 2025-07-01T15:29:59Z

Thanks for the PR. I left some comments after doing some testing

Thanks for reviewing it. I will do some rework and refinements to this PR to fix the issues and address your comments.

Is there a reason why is this PR still a draft?

Not really, I am waiting for your final feedback to open it for review/merge.
I will open it now for review and if you find anything needed for iteration, we can switch it back again.

chong-he

I have tested this and it is working. Would be great to get some comments from the devs

I have added a label test-reproducible, that will trigger the reproducible builds CI to run. When we have release PR, we can add this label
to run the check and ensure that the binaries are reproducible.

chong-he · 2025-07-02T02:54:57Z

Makefile

+TZ_VAL = UTC
+
+# Default features for lighthouse
+FEATURES ?= gnosis,slasher-lmdb,slasher-mdbx,slasher-redb,jemalloc


I think this breaks the cli-check due to this: chong-he@824a5dc

I think we still support mdbx but it's going to be removed some time? We will wait for the devs to comment on this

what do you suggest to change it into? I am open for any suggestions that suits the purpose

We will wait for a review from @michaelsproul when he has the time

jemalloc is now the default

Suggested change

FEATURES ?= gnosis,slasher-lmdb,slasher-mdbx,slasher-redb,jemalloc

FEATURES ?= gnosis,slasher-lmdb,slasher-mdbx,slasher-redb

For simplicity, we could probably use the CROSS_FEATURES that's already defined in the Makefile? or is there something in CROSS_FEATURES that make the reproducible invalid?

chong-he · 2025-07-02T03:20:17Z

.github/workflows/release-reproducible.yml

+jobs:
+  extract-version:
+    name: extract version
+    runs-on: ubuntu-latest


Should we stick to ubuntu-22.04 here? The docker.yml is also running on this now:

lighthouse/.github/workflows/docker.yml

Line 29 in 999b045

runs-on: ubuntu-22.04

I somehow recall that we usually go with the previous version of OS for maximal compatibility

I have no strong opinion about this. I will defer this decision to you

For consistency, let's use Ubuntu 22.04 here to be the same with docker.yml file

chong-he · 2025-07-02T03:22:53Z

.github/workflows/reproducible-build.yml

+
+  build-aarch64:
+    name: test reproducible builds (aarch64)
+    runs-on: ubuntu-24.04-arm


Similar comment as above, wondering should we use ubuntu-22.04-arm instead?

same as my answer above

MoeMahhouk · 2025-07-09T09:49:39Z

@chong-he , I just wanted to double check if this is waiting on me for anything else so far?

chong-he · 2025-07-09T10:39:00Z

@chong-he , I just wanted to double check if this is waiting on me for anything else so far?

No

The PR is now waiting for other team members to review

mergify · 2025-08-01T12:38:36Z

Hi @MoeMahhouk, this pull request has been closed automatically due to 30 days of inactivity. If you’d like to continue working on it, feel free to reopen at any time.

jimmygchen · 2025-08-06T06:01:20Z

Re-opening this PR as it was awaiting review.

antondlr · 2025-08-06T07:46:09Z

I like this idea but would like it to be less maintenance-intensive.
One thing that immediately stands out:
we should not have granular versions like RUST_IMAGE="rust:1.86-bullseye@sha256:1110399f568f1dbe838e58f15b4162d899cb95f450f5f0ffa739614f3a4c32f1 in the workflow/dockerfile but rather provide them as parameters.

MoeMahhouk · 2025-08-08T18:53:09Z

I like this idea but would like it to be less maintenance-intensive. One thing that immediately stands out: we should not have granular versions like RUST_IMAGE="rust:1.86-bullseye@sha256:1110399f568f1dbe838e58f15b4162d899cb95f450f5f0ffa739614f3a4c32f1 in the workflow/dockerfile but rather provide them as parameters.

good idea, could you provide an example on that suggestion? do you have a reference link or would you like to take that up and refine it to be a parameter instead?

mergify · 2025-09-05T06:03:23Z

Hi @MoeMahhouk, this pull request has been closed automatically due to 30 days of inactivity. If you’d like to continue working on it, feel free to reopen at any time.

MoeMahhouk · 2025-10-06T09:59:02Z

Hey folks, @chong-he @jimmygchen ,this PR got auto-closed due to inactivity, but the changes are current and needs review. Could we reopen this for another pass?

chong-he · 2025-10-06T11:21:46Z

Hey folks, @chong-he @jimmygchen ,this PR got auto-closed due to inactivity, but the changes are current and needs review. Could we reopen this for another pass?

Reopening, could you solve some conflicts in the branch?
I will try to give this another review

MoeMahhouk · 2025-10-06T14:55:32Z

Hey folks, @chong-he @jimmygchen ,this PR got auto-closed due to inactivity, but the changes are current and needs review. Could we reopen this for another pass?

Reopening, could you solve some conflicts in the branch? I will try to give this another review

solved the conflicts and merged. Let me know if you want me to do any other change

chong-he

Look at this a second time and try to suggest some simplifications so that it is easier to maintain and make the code cleaner.

Also, I would prefer to not have the symbol (green tick, cross etc) to keep the code clean.

chong-he · 2025-11-04T03:24:49Z

Makefile

+TZ_VAL = UTC
+
+# Default features for lighthouse
+FEATURES ?= gnosis,slasher-lmdb,slasher-mdbx,slasher-redb,jemalloc


jemalloc is now the default

Suggested change

FEATURES ?= gnosis,slasher-lmdb,slasher-mdbx,slasher-redb,jemalloc

FEATURES ?= gnosis,slasher-lmdb,slasher-mdbx,slasher-redb

chong-he · 2025-11-04T10:03:59Z

.github/workflows/reproducible-build.yml

+          echo "=== Building first Docker image (x86_64) ==="
+          docker build -f Dockerfile.reproducible \
+            --build-arg RUST_TARGET="x86_64-unknown-linux-gnu" \
+            --build-arg RUST_IMAGE="rust:1.88-bullseye@sha256:8e3c421122bf4cd3b2a866af41a4dd52d87ad9e315fd2cb5100e87a7187a9816" \


I think we can remove this line and instead just let it read from Dockerfile.reproducible for the RUST_IMAGE. I have tested it on my local repo and it works, see: https://github.com/chong-he/lighthouse/pull/9/files#diff-587298ff141278ce3be7c54a559f9f31472cc5b384e285e2105b3dee319ba31d

This also applies to all other instances that uses this RUST_IMAGE. Taking the idea from Dockerfile:

lighthouse/Dockerfile

Line 1 in e3ee7fe

FROM rust:1.88.0-bullseye AS builder

We can also remove the sha256 in the RUST_IMAGE in Dockerfile.reproducible. That will make it much easier to maintain.

We also do not need a separate image on x86_64 or arm64 from my understanding, as docker is able to select the image based on the host: https://docs.docker.com/build/building/multi-platform/#difference-between-single-platform-and-multi-platform-images, so that will make things simpler

Suggested change

--build-arg RUST_IMAGE="rust:1.88-bullseye@sha256:8e3c421122bf4cd3b2a866af41a4dd52d87ad9e315fd2cb5100e87a7187a9816" \

chong-he · 2025-11-04T10:10:55Z

.github/workflows/reproducible-build.yml

+            exit 1
+          fi
+
+  summary:


I would like to suggest to remove this part to make things simpler.

I think it is sufficient to have 2 additional CIs for reproducible, one for x86_64, another one for arm64. If both passes then good, if any is failing, then we can look at the CI itself, so this summary job is not necessary imho

chong-he · 2025-11-04T11:37:33Z

.github/workflows/release-reproducible.yml

@@ -0,0 +1,253 @@
+name: release-reproducible


I would suggest to name this as docker-reproducible and also the same for the file name, so make it distinct right from the name

chong-he · 2025-11-04T11:56:07Z

.github/workflows/release-reproducible.yml

+          echo "🔄 Verifying reproducible builds for ${{ matrix.arch }}..."
+
+          # Build first image
+          echo "=== Building first verification image ==="


I know I raised this before to have a reproducible check before publishing the image. I am sorry, thinking about it now, we already have the reproducible-CI checks, and this seems repetitive now. In the interest of making things simpler, should we remove the check? I am trying to make the changes as minimal as possible, so that it is easier to read and maintain. After all, the checks for reproducibility is running every 2 days so if anything, we can catch it earlier.

I am just considering about the consequence - if without the checks and the docker images are pushed, then it could be that the docker images are not reproducible in the worse outcome?

chong-he · 2025-11-04T12:13:58Z

Dockerfile.reproducible

 FROM gcr.io/distroless/cc-debian12:nonroot-6755e21ccd99ddead6edc8106ba03888cbeed41a
 COPY --from=builder /lighthouse /lighthouse
+
+EXPOSE 30303 30303/udp 9001 8545 8546


I am not sure if this is necessary? The ports 8545, 8546, 30303 are for the EL

chong-he · 2025-11-04T12:35:15Z

Makefile

+.PHONY: build-reproducible
+build-reproducible: ## Build the lighthouse binary into `target` directory with reproducible builds
+	SOURCE_DATE_EPOCH=$(SOURCE_DATE) \
+	RUSTFLAGS="${RUST_BUILD_FLAGS} --remap-path-prefix $$(pwd)=." \
+	CARGO_INCREMENTAL=${CARGO_INCREMENTAL_VAL} \
+	LC_ALL=${LOCALE_VAL} \
+	TZ=${TZ_VAL} \
+	cargo build --bin lighthouse --features "$(FEATURES)" --profile "$(PROFILE)" --locked --target $(RUST_TARGET)


Not sure about this. Do we need this, given that we have 1 for x86_64 and 1 for arm64 below?

I see that the build-reproducible-all below also don't call this build-reproducible

MoeMahhouk added 3 commits June 16, 2025 23:03

chore: refactor reproducible builds

9a2d6da

Add release-reproducible CI workflow

10a3b59

Add reproducible-build workflow to catch regressions

a4348b2

MoeMahhouk force-pushed the unstable branch 2 times, most recently from 82dcfef to 238fbaa Compare June 17, 2025 10:39

fixing linting issues

608caab

MoeMahhouk force-pushed the unstable branch from 238fbaa to 608caab Compare June 17, 2025 10:45

This was referenced Jun 17, 2025

feat: Add reproducible Debian package builds and distribution #7617

Draft

Add reproducible Debian package builds and distribution #7618

Open

chong-he added ready-for-review The code is ready for review infra-ci labels Jun 18, 2025

mergify bot added waiting-on-author The reviewer has suggested changes and awaits thier implementation. and removed ready-for-review The code is ready for review labels Jun 23, 2025

chong-he reviewed Jun 24, 2025

View reviewed changes

MoeMahhouk marked this pull request as draft June 24, 2025 15:56

refine the reproducible build and test arm github runner

d93242f

MoeMahhouk force-pushed the unstable branch from 737a442 to d93242f Compare June 24, 2025 21:30

MoeMahhouk added 2 commits June 30, 2025 12:50

make release-reproducible verify reproducibility

febc138

add label trigger for reproducible builds

19f2a0f

MoeMahhouk requested a review from chong-he June 30, 2025 15:55

Merge branch 'unstable' into unstable

966ee01

MoeMahhouk marked this pull request as ready for review July 1, 2025 15:30

chong-he added the test-reproducible for testing reproducible builds label Jul 2, 2025

chong-he reviewed Jul 2, 2025

View reviewed changes

chong-he requested a review from antondlr July 2, 2025 12:37

mergify bot closed this Aug 1, 2025

mergify bot added the stale Stale PRs that have been inactive and is now outdated label Aug 1, 2025

jimmygchen reopened this Aug 6, 2025

jimmygchen removed the stale Stale PRs that have been inactive and is now outdated label Aug 6, 2025

mergify bot closed this Sep 5, 2025

mergify bot added the stale Stale PRs that have been inactive and is now outdated label Sep 5, 2025

chong-he reopened this Oct 6, 2025

Merge branch 'unstable' into unstable

f20ef0c

chong-he and others added 2 commits November 3, 2025 06:47

Merge branch 'unstable' into unstable

5674e5a

update rust version for github actions

bddc4f7

chong-he requested changes Nov 4, 2025

View reviewed changes

chong-he removed the stale Stale PRs that have been inactive and is now outdated label Nov 4, 2025

	FEATURES ?= gnosis,slasher-lmdb,slasher-mdbx,slasher-redb,jemalloc
	FEATURES ?= gnosis,slasher-lmdb,slasher-mdbx,slasher-redb

feat: Add reproducible builds release workflows and push images to DockerHub #7614

Are you sure you want to change the base?

feat: Add reproducible builds release workflows and push images to DockerHub #7614

Conversation

MoeMahhouk commented Jun 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Issue Addressed

Proposed Changes

New GitHub Actions Workflows:

Build Configuration Updates:

Profile Removal:

Additional Info

Uh oh!

cla-assistant bot commented Jun 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cla-assistant bot commented Jun 16, 2025

Uh oh!

chong-he commented Jun 19, 2025

Uh oh!

mergify bot commented Jun 23, 2025

Uh oh!

chong-he left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MoeMahhouk commented Jun 24, 2025

Uh oh!

chong-he commented Jul 1, 2025

Uh oh!

MoeMahhouk commented Jul 1, 2025

Uh oh!

chong-he left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MoeMahhouk commented Jul 9, 2025

Uh oh!

chong-he commented Jul 9, 2025

Uh oh!

mergify bot commented Aug 1, 2025

Uh oh!

jimmygchen commented Aug 6, 2025

Uh oh!

antondlr commented Aug 6, 2025

Uh oh!

MoeMahhouk commented Aug 8, 2025

Uh oh!

mergify bot commented Sep 5, 2025

Uh oh!

MoeMahhouk commented Oct 6, 2025

Uh oh!

chong-he commented Oct 6, 2025

Uh oh!

MoeMahhouk commented Oct 6, 2025

Uh oh!

chong-he left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

MoeMahhouk commented Jun 16, 2025 •

edited

Loading

cla-assistant bot commented Jun 16, 2025 •

edited

Loading

chong-he left a comment •

edited

Loading