[CI] SLSA3 compliance for build

[innobead]

Is your feature request related to a problem? Please describe (👍 if you like this request)

Introduce SLSA3 to the build process for our managed Longhorn components only. External dependencies not built by us are excluded from this process.

Describe the solution you'd like

• Update GitHub actions for release including build provenance and SBOM for each component image.
• By default, it's disabled for community builds and enabled for conditions matched.

Describe alternatives you've considered

Additional context

• https://docs.docker.com/build/metadata/attestations/slsa-provenance/
• https://docs.docker.com/build/metadata/attestations/sbom/
• https://docs.docker.com/build/metadata/attestations/slsa-definitions/

cc @derekbit

[derekbit]

Ref: harvester/harvester#6205

[COLDTURNIP]

Follow up for non-containerized releases (e.g. cli): #1

[COLDTURNIP]

PRs to support provenance in GitHub Workflows:

• feat: add provenance for SLSA3 longhorn/backing-image-manager#501
• feat: add provenance for SLSA3 longhorn/longhorn-engine#1393
• feat(ci): add provenance for SLSA3 longhorn/longhorn-instance-manager#830
• feat(ci): add provenance for SLSA3 longhorn/longhorn-manager#3473
• feat(ci): add provenance for SLSA3 longhorn/longhorn-share-manager#435
• feat(ci): add provenance for SLSA3 longhorn/longhorn-ui#857
• feat(ci): add provenance for SLSA3 longhorn/cli#162

[innobead]

We should make sbom and provenance only for prime images. That said, should remove sbom support for Longhorn images.

In the future, if AppCo integration is ready, we should not need to do anything for SLSA3 here, as it is enabled by default for the images built by AppCo build flow. @derekbit can help check with AppCo team.

Before AppCo is ready, this is a transient task for building prime images and publish them into the prime registry, so paying customers can install Longhorn using these SLSA3 compliant images instead of from longhornio@dockerhub.

[COLDTURNIP]

Adding SBOM and provenance flags to the build process will insert additional layers inside the output container image. It would be necessary to branch build logic for community and enterprise versions. Here's the change proposal:

1. Since the container image & registry are not shareable between community & prime customers, we need a separate deployment chart for the prime user in rancher/longhorn repo.
2. To GHA image build process for each component, build two kinds of container images:
   1. <component_name>:v<versoin> for community releases
   2. <component_name>:v<version> (same name) for prime releases, with SLSA flags. Only triggered for stable build (the build number, z of x.y.z, is greater than 0)
3. To the longhorn/release GHA, collects the build and pushes to corresponded registries
   1. for community release, collect & push the community images to public Docker Hub
   2. for prime release, collect & push the prime images to enterprise registry, and also upload the packed SBOM to the release page under rancher/longhorn

cc @innobead @rebeccazzzz @derekbit

[COLDTURNIP]

Design

• longhorn/release is the entry point to initiate a release

  • creates version PR, tag, changelog to each component repo as usual
  • creates tag also on rancher/longhorn
  • collect artifact both from longhorn/longhorn and rancher/longhorn
  • creates releases
    1. create release on longhorn/longhorn, then wait for image ready as usual
    2. if it is stable release (x.y.z where z > 1), also create release on rancher/longhorn, wait for image ready

• Each component repos are responsible to build the binaries, images, and push to some registries.

  • The executable will be build as usual
  • Extract build & push image steps as parameterized reusable action in every component repo
    • Inputs
      • is_security_build: boolean, false by default, indicate whether to generate SBOM and provenance
      • registry: string, target image registry, e.g. docker.io
      • repo: string, target image repo, e.g. longhornio/longhorn-manager
      • tag: string, image tag, e.g. v1.7.2
    • Outputs
      • container images will be built and pushed as registry/repo:tag
      • if is_security_build is enabled, sbom: true and provenance: mode=max will be set for image creation
    • Trigger
      • when branch build needed, triggers a image build
        • registry is docker.io
        • repo is longhornio/${component}
        • tag is ${branch}-head
      • when tag build needed, triggers a community image build
        • registry is docker.io
        • repo is longhornio/${component}
        • tag is ${tag}
      • in addition to tag build, if it is a stable version, also triggers a prime build
        • is_security_build is true
        • registry is the prime registry (TBD)
        • repo is longhorn/${component} (TBD)
        • tag is ${tag}

• The rancher/longhorn is the place for prime release

  • charts point to prime images are placed under /char
  • longhorn/longhorn's generate-sbom action will be moved here, and triggered when a release is created. It will push sbom.tar.gz to the corresponded rancher/longhorn release

Verification

Since the image build actions for components are triggered by tag creation, we use a vx.y.z-dry-run release tag for end-to-end verification. When the tag contains -dry-run suffix:

• longhorn/release
  Creates tags and branch on each repo.
  No version PR will be sent.
  It won’t expect any image get created.

• For each component repo
  Executables will be generated, but only prints logs instead of image-related steps

• rancher/longhorn
  The generate-sbom action will be triggered, but prints logs instead of generating SBOMs from images

Working Items

1. implement branched action on each component repos, remove SBOM logic from community build, include the -dry-run logic
   • longhornio/longhorn-manager
   • longhornio/longhorn-engine
   • longhornio/longhorn-instance-manager
   • longhornio/longhorn-share-manager
   • longhornio/backing-image-manager
   • longhornio/longhorn-ui
2. clone and refine the charts from longhorn/longhorn to rancher/longhorn
3. move generate-sbom action from longhorn/longhorn to rancher/longhorn
4. add prime and dry-run release logic on longhorn/release
5. verify e2e workflow using vx.y.z-dry-run tag
6. clean up the dry-run tags, branches, releases in each repo
7. remove dry-run logic from each repo

cc @innobead @rebeccazzzz @derekbit

[innobead]

This looks good but I am think complete decoupling between both flow might be even better. Let's discuss this later.

@derekbit let's also move this repo to rancher org, it should just name rancher/longhorn?

[derekbit]

This looks good but I am think complete decoupling between both flow might be even better. Let's discuss this later.

@derekbit let's also move this repo to rancher org, it should just name rancher/longhorn?

Applied and waiting.
https://github.com/rancherlabs/eio/issues/2981

[COLDTURNIP]

ECM project requires building and publishes container image using existed GH workflow rancher/ecm-distro-tools/actions/publish-image, instead of the vanilla docker/build-push-action that we're using currently. That means, to each component, we need to implement image build logic in Makefile both for community and prime build, then build & publish the image like:

jobs:
  build: ...

  build-and-publish-images:
    name: Build and publish image
    needs: build
    if: # when we need to push a build
    permissions:
      contents: read
      id-token: write # required for reading vault secrets and for cosign's use in ecm-distro-tools/publish-image
    ...
    steps:
    - name: Build and publish image
      uses: rancher/ecm-distro-tools/actions/publish-image@master
      with:
        image: longhorn-manager
        tag: ...
        make-target: image-build-push # to implement in Makefile, build image WITHOUT sbom & provenance
        push-to-public: true
        push-to-prime: false
        public-registry: docker.io
        public-repo: longhornio
        public-username: ...
        public-password: ...

  build-and-publish-prime-images:
    name: Build and publish prime image
    needs: build
    if: # when we need to push a prime build
    permissions:
      contents: read
      id-token: write
    ...
    steps:
    - name: Build and publish image
      uses: rancher/ecm-distro-tools/actions/publish-image@master
      with:
        image: longhorn-manager
        tag: ...
        make-target: image-build-push-secure # to implement in Makefile, build image WITH sbom & provenance
        push-to-public: false
        push-to-prime: true
        prime-registry: # secret from PRIME_REGISTRY
        prime-repo: longhorn
        public-username: # secret from PRIME_REGISTRY_USERNAME
        public-password: # secret from PRIME_REGISTRY_PASSWORD

[COLDTURNIP]

Per information from Guidance on achieving SLSA Level 3, it is required to merge the executable binary build logic inside the image creation as an additional layer.

The build process must be self-contained. Ensure that the Dockerfile is responsible for as many of the steps as possible. For example, it must compile the binaries of the project as a Docker layer, instead of copying a pre-compiled binary from disk.

[innobead]

Build time stuff should not be persistent in the final component image, as they are not required for runtime.

[recena]

Well, every OCI image should include the attestation.

[innobead]

Well, every OCI image should include the attestation.

That's interesting for the design (SLSA L3). However, from the build perspective, image layers are the only way to keep that build info. It's a design but also a tradeoff technically.

@COLDTURNIP no worries for my previous comment. It's clarified, go continue your development.