From 454a27405c1fe5a516c647d3c2f969d3c75142a0 Mon Sep 17 00:00:00 2001 From: Matteo Bianchi <37507190+mbianchidev@users.noreply.github.com> Date: Tue, 18 Feb 2025 17:56:47 +0100 Subject: [PATCH 01/26] Create k8s-release-cut.md --- .../handbooks/k8s-release-cut.md | 526 ++++++++++++++++++ 1 file changed, 526 insertions(+) create mode 100644 release-engineering/handbooks/k8s-release-cut.md diff --git a/release-engineering/handbooks/k8s-release-cut.md b/release-engineering/handbooks/k8s-release-cut.md new file mode 100644 index 00000000000..a220fef4f50 --- /dev/null +++ b/release-engineering/handbooks/k8s-release-cut.md @@ -0,0 +1,526 @@ +# Cutting a Kubernetes patch release + +A step by step guide for cutting Kubernetes patch releases. At a high-level: + +- Maintain GitHub release cut issue +- Update tools (~15m) +- Run `krel stage` (~2h) +- Run `krel release` (~15m) +- Run `krel stage --no-mock` (~2h) +- Run `kpromo pr` & merge PR +- Wait for image promo job (~1h) +- Run `krel release --no-mock` (~15m) +- Send announcements (~30m) + + +## Prerequisites + +### Access to GCP + +You must be a member of [k8s-infra-release-editors](https://github.com/kubernetes/k8s.io/blob/main/groups/sig-release/groups.yaml) on GitHub. + +Group membership grants access to [Google Cloud Platform (GCP)](https://console.cloud.google.com/) for issuing `krel` commands that ultimately launch [Cloud Build](https://cloud.google.com/build) jobs. + +> [Example PR](https://github.com/kubernetes/k8s.io/pull/7781) + +### Install latest software (every time) + +Begin by updating to the latest package versions. This helps reduce failure points in the build process. The following packages are needed: + +- golang (Go, used to build OSS tools) +- gcloud (GCP CLI) +- krel (OSS CLI Tool) +- kpromo (OSS CLI Tool) +- [optional] schedule-builder (OSS CLI Tool) + +#### Download or update `Go` to the latest available stable version: + +[go.dev/dl](https://go.dev/dl/) + +``` +# to install +brew install go + +# to update (latest) +brew upgrade go +``` + +[optional] if on a Linux-based system using [zsh](https://www.zsh.org/) (Mac, Linux, Windows WSL) add to `~/.zshrc` PATH: + +``` +export GOPATH=$(go env GOPATH) +export PATH="$HOME/go/bin:${PATH}" +#reload your .zshrc +source ~/.zshrc +``` + +#### Download or update the `gcloud` CLI to interact with GCP. + +``` +# to install +brew install --cask google-cloud-sdk + +# to update +gcloud components update --quiet +``` + +Ensure your GCP credentials are active. + +``` +gcloud auth login +``` + +You will be prompted to log in via your default browser. + +> [!TIP] +Ensure that the email you are using to login is the same added in the [Access to GCP](#Access-to-GCP) step. + +#### Download or update `krel` + +Krel ([docs](https://github.com/kubernetes/release/blob/master/docs/krel/README.md)). + +First time: + +``` +git clone git@github.com:kubernetes/release.git +``` + +Every time: + +``` +cd release + +# to update +git checkout master +git pull origin master +``` + +Compile `krel` using the script in the repo: + +``` +./compile-release-tools krel +``` + +Validate with: + +``` +krel version +``` + + +#### Download or update the latest `kpromo` tool + +Run the following command ([source](https://github.com/kubernetes-sigs/promo-tools/blob/main/docs/promotion-pull-requests.md#preparing-environment)): + +``` +go install sigs.k8s.io/promo-tools/v4/cmd/kpromo@latest +``` + +Validate with: + +``` +kpromo version +``` + +Output should look like this: + +``` +_ __ ____ ____ ___ __ __ ___ +| |/ / | _ \ | _ \ / _ \ | \/ | / _ \ +| ' / | |_) | | |_) | | | | | | |\/| | | | | | +| . \ | __/ | _ < | |_| | | | | | | |_| | +|_|\_\ |_| |_| \_\ \___/ |_| |_| \___/ +kpromo: Kubernetes project artifact promoter + +GitVersion: v3.4.4 +GitCommit: unknown +GitTreeState: unknown +BuildDate: unknown +GoVersion: go1.19.1 +Compiler: gc +Platform: darwin/arm64 +``` + +#### Download schedule-builder + +Only when cutting a patch release (not a pre-release like alpha, beta, etc.). + +``` +go install k8s.io/release/cmd/schedule-builder@latest +``` + +### Configure GitHub Personal Access Token + +Get a GitHub API token (one off or expiration based) + +Navigate to `Settings > Developer Settings > Personal Access Token` to generate a new token. For the scopes, select: + +- repo +- gist +- notifications +- user + +Set an appropriate expiration date then click on `[GENERATE TOKEN]` and copy it. + +> NOTE / TODO: Look into using fine grained tokens: https://stackoverflow.com/questions/78216547/minimum-permissions-for-a-github-access-token-to-clone-push-and-pull-from-repo. Doing the bare minimum seemed to not work. + +Run this command in your shell to export the token and making it available for `krel` and `kpromo`. They should be short lived, no need to store. + +``` +export GITHUB_TOKEN=xyz +``` + +## 1. Release cut issue + +First step before cutting the release is to open an [issue](https://github.com/kubernetes/sig-release/issues/new?template=cut-release.md). + +An example issue you can follow can be found [here](https://github.com/kubernetes/sig-release/issues/2719). + +The issue contains a series of steps to follow and at some point you will be asked to input the `krel version` command output: + +``` +GitVersion: v0.13.0 +GitCommit: 2c2e73f4e15c85beba135e4513308bffa79abc64 +GitTreeState: clean +BuildDate: 2022-09-14T08:01:01 +GoVersion: go1.19.1 +Compiler: gc +Platform: darwin/arm64 +``` + +> [!WARNING] +Always recompile `krel` before cutting a release. + +## 2. Create a thread on the `#release-management` Slack channel + +Helpful templates, each one has a "completed" response too: + +``` +:thread: Release Cut v1.xx.yy-alpha|beta|rc-z (GH Issue) + +:hourglass_flowing_sand: Mock Stage (logs) +-> :white_check_mark: + +:hourglass_flowing_sand: Mock Release (logs) +-> :white_check_mark: + +:hourglass_flowing_sand: No-mock Stage (logs) +-> :white_check_mark: + +https://github.com/kubernetes/k8s.io/pull/xxxx + +Image Promotion PR (link) +cc: @release-managers +-> Image promotion is :white_check_mark: + +:hourglass_flowing_sand: No-mock Release (logs) +-> :white_check_mark: +``` + +## 3. Generate testgrid screenshots + +Generate a comment with the `krel testgridshot` command as follows: + +``` +# defaults to master / main: +krel testgridshot + +# specific branch: +krel testgridshot --branch 1.xx +``` + +Post the comment generated in the **GitHub issue**. + +This comment contains a current snapshot of the test results for the target branch, if there is any error you should stop the release process and inform #release-management on Slack. Before proceeding the failing tests need to be either fixed or marked as non-release blocking. + +## 4. Check publishing-bot status + +Check the health of the publishing-bot by navigating to [this issue](https://github.com/kubernetes/kubernetes/issues/56876) if it's closed, the bot is healthy and you can keep going. + +If the issue is open you must stop the release process and inform #release-management on Slack. + +## 5. Mock stage and Mock release + +Mock stages and releases are non-destructive and can be reran upon failure. To begin the mock stage, run the following `krel stage` command (replace the stage with the appropriate "type"). + +Run the following command from within the release repo directory. + +``` +# if cutting any unreleased version +krel stage --type alpha --branch master + +krel stage --type beta --branch master + +# or specify a branched version +krel stage --type official --branch release-1.xx + +krel stage --type rc --branch release-1.xx +``` + +> [!WARNING] +If you encounter an error that looks like this +`Step #4: level=fatal msg="checking out obs project: checking out obs project: command /usr/local/bin/osc checkout isv:kubernetes:core:prerelease:v1.33:build did not succeed: Server returned an error: HTTP Error 404: Not Found\nProject not found: isv:kubernetes:core:prerelease:v1.33:build\n"` +The OBS project has not been created, contact @release-managers on Slack. + +> [!CAUTION] +If you encounter this error +"FATA submit stage job: verifying repository state: local HEAD (d2eb4818bbaf2820128f21c1895308dad4cfc6de) is not equal to latest remote commit (90469b0184a2e3d1b31209bf8fed413273f92a17)" you need to pull the release repo [as follows](#Download-or-update-krel-docs) + +The `krel` tool outputs a link to the logs. + +This takes about **2 hours**. Once passed, move on to the "release" command using a specific build-version string (output from the first stage). + +> [!NOTE] +At this point you should start updating the Slack ([thread](#Create-a-thread-on-release-management)) and the release-cut GitHub [issue](#Release-cut-issue) + +``` +# take the output of the previous command from the logs and run the command +# It should look like this: +krel release --type=alpha|beta|official|release --branch=release-1.xx --build-version=v1.xx.yy-alpha|beta|rc-z+ +``` + +If you are releasing an `alpha.1` you will have a command output that has a `build-version` parameter value containing `alpha.0`, same goes for `alpha.2` having `alpha.1` and so on. This is expected, you can proceed with executing the krel release command. + +> [!NOTE] +Remember to update the Slack ([thread](#Create-a-thread-on-release-management)) and the release-cut GitHub [issue](#Release-cut-issue) after this step + +## 5. No-mock stage + +The following stages, called no-mock, create real artifacts that can be promoted for general use. The process should be near identical to the `no-mock` stages ran prior. + +Run the no-mock staging: + +``` +# if cutting an unreleased version +krel stage --type alpha --branch master --nomock + +krel stage --type rc --branch release-1.xx --nomock + +# or specify a branch version +krel stage --type official --branch release-1.xx --nomock +``` + +> [!NOTE] +Remember to update the Slack ([thread](#Create-a-thread-on-release-management)) and the release-cut GitHub [issue](#Release-cut-issue) after this no-mock stage step + +This also takes about **2 hours**. Once passed, move on to the `kpromo` command + +``` +# take the output of the previous command from the logs and copy this command somewhere, do NOT execute this. + +# It should look like this: +krel release --type=alpha|beta|official|release --branch=release-1.xx --build-version=v1.xx.yy-alpha|beta|rc-z+ +``` + +> [!CAUTION] +Do not run the command yet, just copy it somewhere and wait for the image promo to happen first (PR merged + prow job completed). + +## 6. Image promotion + +Promote the images (`alpha`, `beta` or `rc`) using `kpromo`. The tool will automatically create / open a PR with the images and tags properly promoted. + +### Which image should be promoted? + +The images that need to be promoted depend on the release you're cutting: + +- **Alpha or Beta release:** just promote the images for the release you're cutting (e.g. v1.20.0-beta.1) +- **The first RC (e.g. v1.20.0-rc.0):** promote the images for the RC and for the next minor alpha release (e.g. v1.21.0-alpha.0) +- **The subsequent RCs (e.g. v1.20.0-rc.1):** promote the images for the RC you're cutting (e.g. v1.20.0-rc.0) +- **A stable release (e.g. v1.20.0):** promote the images for the release you're cutting and for the RC of the next patch release (e.g. v1.20.1-rc.0) + +Some examples can be found below: + +``` +kpromo pr --fork=jimangel --tag="v1.29.0-alpha.1" + +# RC +kpromo pr --fork=jimangel --tag="v1.32.0-rc.1" +#or +kpromo pr --fork=jimangel --tag="v1.29.0-rc.0" --tag="v1.30.0-alpha.0" +``` + +### Run kpromo (opens the PR) + +> [!IMPORTANT] +If promoting the first release candidate (rc), include the next release alpha as follows. + +``` +kpromo pr --fork= --tag="v1.xx.yy-rc-z" --tag="v1.xx+1.yy-alpha-0" +``` + +> [!TIP] +Multiple tags can be specified with the `kpromo` pr command + +This is the standard command to be run ([source](https://github.com/kubernetes-sigs/promo-tools/blob/main/docs/promotion-pull-requests.md#promoting-images)): + +``` +# --interactive=true asks confirmation before every step, highly recommended + +kpromo pr --fork= --tag="v1.xx.yy-alpha|beta|rc-z" --interactive=true +``` + + > [!WARNING] +`FATA growing manifest with image filter [] and tag [1.29.0-alpha.1]: no images survived filtering; double-check your --filter_* flag(s) for typos` +means you forgot the letter "v" in the tag parameter. + +Once run the command its output should look like this: + +``` +INFO Successfully created PR #xxxx +INFO Successfully created PR: https://github.com/kubernetes/k8s.io/pull/xxxx +``` + +### Merge PR + +Wait for the generated PR to be merged (after approval and LGTM). +You might wanna ping @release-managers on Slack to speed this process up. + +**The PR is not blocked by anything but approvals / review.** + +> [Example PR](https://github.com/kubernetes/k8s.io/pull/3024). + +### Wait on image promotion job + +!! DO NOT CONTINUE **RELEASE** (UNTIL FINISHED) + +> [!CAUTION] +**After** the Pull Request is merged and **before** starting the official nomock release step, watch [the following Prow Job](https://prow.k8s.io/?job=post-k8sio-image-promo) and wait for it to succeed. + +If the job fails, look into the root cause and message the Slack thread. In many cases a rerun is fine (such as 429 too many requests). + +To Rerun, click the refresh icon button shown below: + +![image](https://hackmd.io/_uploads/Bk6UOW5Kyg.png) + + +> [!NOTE] +Remember to update the Slack ([thread](#Create-a-thread-on-release-management)) and the release-cut GitHub [issue](#Release-cut-issue) after the image promotion step + +## 7. No-mock release + +You should have copied the no mock release command output from the nomock stage previously, now you can run the release: + +``` +krel release --nomock --build-version=v1.xx.yy-alpha|beta|rc-z+ +``` + +> [!NOTE] +Remember to update the Slack ([thread](#Create-a-thread-on-release-management)) and the release-cut GitHub [issue](#Release-cut-issue) after this no-mock release step. + +## 8. Notify public dev Google group mailinglist + +> **NOTE from February 2025: Using manual is the preferred method given current issues with Sendgrid.** + +### Manually create release HTML announcements + +Sometimes you might need to manually send the announcement, in which case you can run this command: + +``` +krel announce send -p --tag v1.xx.yy-alpha|beta|rc-z --name "First Last" --email "your-email@gmail.com" > ~/Downloads/announce.html +``` + +And then send the email from your account as follows: + +``` +# SUBJECT: +# Kubernetes v1.xx.yy-alpha|beta|rc-z is live! + +# ... +# SEND TO: +dev@kubernetes.io, kubernetes-announce@googlegroups.com +``` + +#### Legacy Sendgrid method: + +> [!TIP] +username == email` & `password == normalx2`. + +``` +cd ~/release +export SENDGRID_API_KEY= + +krel announce send --tag v1.xx.yy-alpha|beta|rc-z --name "First Last" --email "your-email@gmail.com" --nomock +``` + +Samples: + +``` +# for 1.22.14, 1.23.11, 1.24.5 and 1.25.1 +krel announce send --tag v1.24.14 --name "Jim Angel" --email "jameswangel@gmail.com" --nomock +krel announce send --tag v1.25.10 --name "Jim Angel" --email "jameswangel@gmail.com" --nomock +krel announce send --tag v1.26.5 --name "Jim Angel" --email "jameswangel@gmail.com" --nomock +krel announce send --tag v1.27.2 --name "Jim Angel" --email "jameswangel@gmail.com" --nomock +``` + +> [!WARNING] +You could run in a reached max recipients quota, in such case communicate with #release-management. + +> [!TIP] +Kubernetes-announce might require permissions to post, check you have them or ask your release-manager buddy to post the message for you. + +## 9. Notify stakeholders + +Post this message in release-management: + +``` +:kubernetes: Kubernetes v1.xx.yy is live! (shoutout to @xx & @Byfor helping cut the release) +https://groups.google.com/a/kubernetes.io/g/dev/c/fbaBcFvZFMo +``` + +Examples + +``` +:kubernetes: Kubernetes v1.23.11 is live! (shoutout to @arnaud & @Benjamin Kazemi for helping cut the release) +https://groups.google.com/a/kubernetes.io/g/dev/c/E14i6e2ndNc + +:kubernetes: Kubernetes v1.24.5 is live! (shoutout to @arnaud & @Benjamin Kazemi for helping cut the release) +https://groups.google.com/a/kubernetes.io/g/dev/c/WSQwY55HeeI + +:kubernetes: Kubernetes v1.25.1 is live! (shoutout to @arnaud & @Benjamin Kazemi for helping cut the release) +https://groups.google.com/a/kubernetes.io/g/dev/c/R5vB-jFqnDw +``` + +Copy / paste link to #release-management notification post. + +> [!NOTE] +Remember to provide one last final update to the Slack ([thread](#Create-a-thread-on-release-management)) and the release-cut GitHub [issue](#Release-cut-issue) after the announcement step + +To valorize the release stats required in the release cut issue, run this command: + +``` +# date should be the previous release date +krel history --branch master --date-from yyyy-mm-dd + +krel history --branch release-1.xx --date-from yyyy-mm-dd +``` + +## 10. Update schedule on k/website + +> [!NOTE] +**Only for patch releases** + +You need to update the release schedule on the official Kubernetes website through schedule-builder, here is how: + +``` +# update k/website repo +git checkout main +git fetch upstream +git pull upstream main +git reset --hard upstream/main +rm -rf api-ref-generator +git submodule deinit -f api-ref-generator +git submodule update --init --recursive + + +# from in the k/website repo +git checkout -b schedule-updates-nov-2024 +schedule-builder -uc ./data/releases/schedule.yaml -e ./data/releases/eol.yaml +git add . +git commit -m "updating release schedule" +git push -u origin schedule-updates-oct-2024 +``` + +## Cleanup + +Close out related issues, open new issues if you find any, and ask Google admins to revoke your personal access to GCP if it is no longer required. +You might want to delete your GitHub access token too. From 0ff990a72c3eddbf4fa444475094104cdb512684 Mon Sep 17 00:00:00 2001 From: Matteo Bianchi <37507190+mbianchidev@users.noreply.github.com> Date: Tue, 18 Feb 2025 18:12:14 +0100 Subject: [PATCH 02/26] Update k8s-release-cut.md --- release-engineering/handbooks/k8s-release-cut.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/release-engineering/handbooks/k8s-release-cut.md b/release-engineering/handbooks/k8s-release-cut.md index a220fef4f50..f96a18ed7ed 100644 --- a/release-engineering/handbooks/k8s-release-cut.md +++ b/release-engineering/handbooks/k8s-release-cut.md @@ -388,10 +388,18 @@ You might wanna ping @release-managers on Slack to speed this process up. If the job fails, look into the root cause and message the Slack thread. In many cases a rerun is fine (such as 429 too many requests). +To check the logs, click on the spyglass eye icon. + To Rerun, click the refresh icon button shown below: ![image](https://hackmd.io/_uploads/Bk6UOW5Kyg.png) +The UI can be funky at times, if you click rerun and it forces you to log in - you might need to click it again. +In case it doesn't it should show a pop up: +- if it says it didn’t run, retry +- if it says it ran wait 30s and check again (takes a minute) + +If the job is stuck in "triggered" (empty clock yellow icon) //TODO > [!NOTE] Remember to update the Slack ([thread](#Create-a-thread-on-release-management)) and the release-cut GitHub [issue](#Release-cut-issue) after the image promotion step From ee41dbe6025ca8f8522239aab81479c65485012c Mon Sep 17 00:00:00 2001 From: Matteo Bianchi <37507190+mbianchidev@users.noreply.github.com> Date: Tue, 18 Feb 2025 18:18:34 +0100 Subject: [PATCH 03/26] Update k8s-release-cut.md --- release-engineering/handbooks/k8s-release-cut.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/release-engineering/handbooks/k8s-release-cut.md b/release-engineering/handbooks/k8s-release-cut.md index f96a18ed7ed..a8a39b5798e 100644 --- a/release-engineering/handbooks/k8s-release-cut.md +++ b/release-engineering/handbooks/k8s-release-cut.md @@ -392,13 +392,16 @@ To check the logs, click on the spyglass eye icon. To Rerun, click the refresh icon button shown below: -![image](https://hackmd.io/_uploads/Bk6UOW5Kyg.png) +image The UI can be funky at times, if you click rerun and it forces you to log in - you might need to click it again. In case it doesn't it should show a pop up: - if it says it didn’t run, retry - if it says it ran wait 30s and check again (takes a minute) +You might notice different jobs running, be sure to check out the hash from the merge commit that has been generated by prow after your image promo PR. +That is the hash of the job you want to check and rerun if it fails. + If the job is stuck in "triggered" (empty clock yellow icon) //TODO > [!NOTE] From b8ec74dee5df78dadc5e2a8108182db808573d5c Mon Sep 17 00:00:00 2001 From: Matteo Bianchi <37507190+mbianchidev@users.noreply.github.com> Date: Tue, 18 Feb 2025 18:21:07 +0100 Subject: [PATCH 04/26] Update k8s-release-cut.md --- release-engineering/handbooks/k8s-release-cut.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/release-engineering/handbooks/k8s-release-cut.md b/release-engineering/handbooks/k8s-release-cut.md index a8a39b5798e..21cadb830f6 100644 --- a/release-engineering/handbooks/k8s-release-cut.md +++ b/release-engineering/handbooks/k8s-release-cut.md @@ -4,11 +4,11 @@ A step by step guide for cutting Kubernetes patch releases. At a high-level: - Maintain GitHub release cut issue - Update tools (~15m) -- Run `krel stage` (~2h) +- Run `krel stage` (~1h 15m - up to 2h) - Run `krel release` (~15m) -- Run `krel stage --no-mock` (~2h) +- Run `krel stage --no-mock` (~1h 15m - up to 2h) - Run `kpromo pr` & merge PR -- Wait for image promo job (~1h) +- Wait for image promo prow job (~1h) - Run `krel release --no-mock` (~15m) - Send announcements (~30m) From 72bdc1760028faec5fb779565ca69b767f4386bb Mon Sep 17 00:00:00 2001 From: Matteo Bianchi <37507190+mbianchidev@users.noreply.github.com> Date: Wed, 19 Feb 2025 12:04:19 +0100 Subject: [PATCH 05/26] Update k8s-release-cut.md --- release-engineering/handbooks/k8s-release-cut.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/release-engineering/handbooks/k8s-release-cut.md b/release-engineering/handbooks/k8s-release-cut.md index 21cadb830f6..69a324a7f7f 100644 --- a/release-engineering/handbooks/k8s-release-cut.md +++ b/release-engineering/handbooks/k8s-release-cut.md @@ -388,7 +388,7 @@ You might wanna ping @release-managers on Slack to speed this process up. If the job fails, look into the root cause and message the Slack thread. In many cases a rerun is fine (such as 429 too many requests). -To check the logs, click on the spyglass eye icon. +To check the logs after failure, click on the spyglass eye icon, otherwise you can look at the POD logs when the job is running by clicking on the paper icon. To Rerun, click the refresh icon button shown below: @@ -402,7 +402,7 @@ In case it doesn't it should show a pop up: You might notice different jobs running, be sure to check out the hash from the merge commit that has been generated by prow after your image promo PR. That is the hash of the job you want to check and rerun if it fails. -If the job is stuck in "triggered" (empty clock yellow icon) //TODO +If the job is stuck in "triggered" (empty clock yellow icon), especially after a rerun, it's normal and expected. Please wait and do not touch it. > [!NOTE] Remember to update the Slack ([thread](#Create-a-thread-on-release-management)) and the release-cut GitHub [issue](#Release-cut-issue) after the image promotion step From 4f27b6621c37a88cf365b8ddf981436c8a90daf1 Mon Sep 17 00:00:00 2001 From: Matteo Bianchi <37507190+mbianchidev@users.noreply.github.com> Date: Fri, 14 Mar 2025 15:01:30 +0100 Subject: [PATCH 06/26] Include Green Release Signal prereq Update k8s-release-cut.md --- release-engineering/handbooks/k8s-release-cut.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/release-engineering/handbooks/k8s-release-cut.md b/release-engineering/handbooks/k8s-release-cut.md index 69a324a7f7f..09a62250838 100644 --- a/release-engineering/handbooks/k8s-release-cut.md +++ b/release-engineering/handbooks/k8s-release-cut.md @@ -15,6 +15,13 @@ A step by step guide for cutting Kubernetes patch releases. At a high-level: ## Prerequisites +### Green Release Signal + +On the same day of the release, a green signal must've been given in the #release-management Slack channel. If in doubt, double check with the current Release Signal Team Lead. +You can find the complete list of release signal team members at this link (substitute `1.xx` with the current release version): + +`https://github.com/kubernetes/sig-release/blob/master/releases/release-1.xx/release-team.md` + ### Access to GCP You must be a member of [k8s-infra-release-editors](https://github.com/kubernetes/k8s.io/blob/main/groups/sig-release/groups.yaml) on GitHub. From c8a97fe96007a81f2ddce6b3ce3298dc32f9df26 Mon Sep 17 00:00:00 2001 From: Matteo Bianchi <37507190+mbianchidev@users.noreply.github.com> Date: Fri, 14 Mar 2025 15:02:44 +0100 Subject: [PATCH 07/26] Added green release signal checklist item Update cut-release.md --- .github/ISSUE_TEMPLATE/cut-release.md | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/ISSUE_TEMPLATE/cut-release.md b/.github/ISSUE_TEMPLATE/cut-release.md index 3b15216fdc4..02733b21bd0 100644 --- a/.github/ISSUE_TEMPLATE/cut-release.md +++ b/.github/ISSUE_TEMPLATE/cut-release.md @@ -72,6 +72,7 @@ Help? Ring @release-managers on slack! ## Release Steps +- [ ] Ensure a green signal has been given by Release Signal **on the day of the cut** - [ ] Create a thread on #release-management: - [ ] Screenshot unhealthy release branch testgrid boards - [ ] Check the health of the publishing-bot From 41bf32df7ebcaf84f243a7948727904ff28f2855 Mon Sep 17 00:00:00 2001 From: Matteo Bianchi <37507190+mbianchidev@users.noreply.github.com> Date: Fri, 21 Mar 2025 01:00:14 +0100 Subject: [PATCH 08/26] Added email groups mention Update k8s-release-cut.md --- release-engineering/handbooks/k8s-release-cut.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/release-engineering/handbooks/k8s-release-cut.md b/release-engineering/handbooks/k8s-release-cut.md index 09a62250838..05cb4852899 100644 --- a/release-engineering/handbooks/k8s-release-cut.md +++ b/release-engineering/handbooks/k8s-release-cut.md @@ -427,7 +427,11 @@ Remember to update the Slack ([thread](#Create-a-thread-on-release-management)) ## 8. Notify public dev Google group mailinglist -> **NOTE from February 2025: Using manual is the preferred method given current issues with Sendgrid.** +Ensure that you are a moderator of dev@kubernetes.io so you can send messages without passing by the moderation queue, and also an admin of [kubernetes-announce](https://groups.google.com/g/kubernetes-announce). +In case you are not part of these groups as moderator/admin, ask to be added in #release-management. +Ideally this is a task that should be performed during onboarding and not on the day of the cut. + +> **NOTE as of February 2025: Using manual is the preferred method given the issues with Sendgrid.** ### Manually create release HTML announcements From 5d954da9742f409ebd12d7a33be3d184f7554af2 Mon Sep 17 00:00:00 2001 From: Matteo Bianchi <37507190+mbianchidev@users.noreply.github.com> Date: Tue, 8 Apr 2025 19:18:55 +0200 Subject: [PATCH 09/26] Made branch creation and rc.0 process more readable Update branch-manager.md --- .../role-handbooks/branch-manager.md | 75 +++++++++++-------- 1 file changed, 45 insertions(+), 30 deletions(-) diff --git a/release-engineering/role-handbooks/branch-manager.md b/release-engineering/role-handbooks/branch-manager.md index 30321c84f03..44e0594255c 100644 --- a/release-engineering/role-handbooks/branch-manager.md +++ b/release-engineering/role-handbooks/branch-manager.md @@ -35,9 +35,8 @@ - [Update kubekins-e2e variants](#update-kubekins-e2e-variants) - [Cut next alpha](#cut-next-alpha) - [Branch Management](#branch-management) - - [Branch Creation](#branch-creation) - - [During the release creation](#during-the-release-creation) - - [After the release creation](#after-the-release-creation) + - [Branch Creation](#release-branch-creation) + - [Post release branch creation tasks](#post-release-branch-creation-tasks) - [Update test-infra configurations](#update-test-infra-configurations) - [Update milestone appliers](#update-milestone-appliers) - [Update milestone requirements](#update-milestone-requirements) @@ -515,7 +514,7 @@ Set the `K8S_RELEASE` marker for the current release variant to `stable-x.y` whi #### Cut next alpha -Recall that an alpha.0 of the next minor release was created during [branch creation](#branch-creation). +Recall that an alpha.0 of the next minor release was created during [release branch creation](#release-branch-creation). The previously created alpha.0 is now several commits behind `master`. As an example, see the [comparison between the `v1.18.0-alpha.0` (after 1.17 branch creation) and `v1.18.0-alpha.1` (after 1.17.0 release) tags](https://github.com/kubernetes/kubernetes/compare/v1.18.0-alpha.0...v1.18.0-alpha.1). @@ -542,61 +541,75 @@ In case the release branch is missing - we will end up with an issue we had for This section discusses the methods in managing commits on the `release-x.y` branch. -### Branch Creation +### Update test-infra configurations -During a `rc.0` release our release tooling creates a new release branch named `release-x.y`, where `x` and `y` are the major and minor versions of the next release, respectively. +Before getting started, Branch Managers should: -Behind the scenes `krel` is doing a git branch create and git push. The branch is created in the staging phase and is pushed to the repository in the release phase. +- Fork the [test-infra repository](https://github.com/kubernetes/test-infra) +- Clone their fork of `kubernetes/test-infra`: -`prow`’s [`branchprotector`](https://git.k8s.io/test-infra/prow/cmd/branchprotector/README.md) runs every hour at 54 minutes past the hour and automatically adds [branch protection](https://help.github.com/articles/about-protected-branches/) to any new branch in the `kubernetes/kubernetes` repo. +```shell +git clone git@github.com:/test-infra.git +``` -New release branch creation (for example: `release-1.18`) also automatically triggers an alpha.0 build for the subsequent release (for example: [`v1.19.0-alpha.0`](https://github.com/kubernetes/kubernetes/releases/tag/v1.19.0-alpha.0)). +- (optional) [Install Bazel](https://docs.bazel.build/versions/master/install.html) or run Bazel inside a container + - Running Bazel in a container is recommended over installing Bazel locally, as Bazel has many dependencies -This means that the staging step will take about twice as long, as it will stage both versions `v1.18.0-rc.0` and `v1.19.0-alpha.0`. The release step will also be extended, but not substantially longer in time. +### Release Branch Creation -#### During the release creation +> [!IMPORTANT] +> The branch is created in the nomock staging phase and is pushed to the repository during the nomock release phase. -Before the nomock release job is started, run through the following tasks, **_putting an explicit hold_** on any PRs (to be removed once the release branch has been created): +During a `rc.0` release our release tooling creates a new release branch named `release-x.0`, where `x` is the major version of the next release. -- [Update test-infra configurations](#update-test-infra-configurations) - - [Update milestone appliers](#update-milestone-appliers) - - [Update milestone requirements](#update-milestone-requirements) - - [Update e2e variants](#update-e2e-variants) +Behind the scenes `krel` is executing a `git branch create` command and `git push`. -#### After the release creation +At the same time Prow’s [`branchprotector`](https://git.k8s.io/test-infra/prow/cmd/branchprotector/README.md) runs every hour at 54 minutes past the hour and automatically adds [branch protection](https://help.github.com/articles/about-protected-branches/) to any new branch in the `kubernetes/kubernetes` repo, including the newly created one. +No need to manually create the branch protection rule. -Once the new `release-x.y` branch is created, the [release jobs and dashboards should be generated and merged](#generate-release-branch-jobs). +New release branch creation (for example: `release-1.18`) also automatically triggers an alpha.0 build for the subsequent release (for example: [`v1.19.0-alpha.0`](https://github.com/kubernetes/kubernetes/releases/tag/v1.19.0-alpha.0)). -### Update test-infra configurations +> [!NOTE] +This means that the staging step will take about twice as long, as it will stage both versions `v1.18.0-rc.0` and `v1.19.0-alpha.0`. +The release step will also be extended, but not substantially longer in time. -Before getting started, Branch Managers should: +#### Post release branch creation tasks -- Fork the [test-infra repository](https://github.com/kubernetes/test-infra) -- Clone their fork of `kubernetes/test-infra`: +During the mock stage runs and before the nomock release job is started, run through the following tasks. -```shell -git clone git@github.com:/test-infra.git -``` +> [!WARNING] +Rembember to put a `/hold` on all the PRs, they have to be lifted only once the nomock release phase is done and the branch is created. -- [Install Bazel](https://docs.bazel.build/versions/master/install.html) or run Bazel inside a container - - Running Bazel in a container is recommended over installing Bazel locally, as Bazel has many dependencies +- [Update test-infra configurations](#update-test-infra-configurations) + - [Update milestone appliers](#update-milestone-appliers) + - [Update milestone requirements](#update-milestone-requirements) (this should have been already done as part of the Code Freeze routine) + - [Update e2e variants](#update-e2e-variants) + +After the rc.0 cut is done and you announced it, proceed with the following: + +- [Generate release branch jobs](#generate-release-branch-jobs) +- [Add a new variant for the kube-cross image](#add-a-new-variant-for-the-kube-cross-image) +- [Update publishing-bot rules](#update-publishing-bot-rules) + +Once the new `release-x.0` branch is created, the [release jobs and dashboards should be generated and merged](#generate-release-branch-jobs). #### Update milestone appliers -The [milestone applier plugin](https://git.k8s.io/test-infra/prow/plugins/milestoneapplier/milestoneapplier.go) automatically applies a GitHub milestone to pull requests after they have merged. +The [milestone applier plugin](https://github.com/kubernetes-sigs/prow/blob/main/pkg/plugins/milestoneapplier/milestoneapplier.go) automatically applies a GitHub milestone to pull requests after they have merged. This only applies to repos that have the milestone applier configured and for pull requests that do not already have a milestone. Update the `milestoneapplier` plugin configs for `kubernetes/kubernetes`: -- Remove configs for the unsupported release branches, if present +- Remove configs for the unsupported release branches, if present - _this is typically the oldest release in the list_ - Add config for the current release branch Here's an [example PR](https://github.com/kubernetes/test-infra/pull/20075). #### Update milestone requirements -If the [code freeze](#code-freeze) was enabled before creating the release branch, the milestone requirements wouldn't include the newest release branch. +> [!NOTE] +If the [code freeze](#code-freeze) was enabled before creating the release branch, there's nothing to do, as the milestone requirements would include the newest release branch already. Find the query config for `kubernetes/kubernetes` (in [config.yaml][config.yaml] file) with the code freeze enabled and add the newest release release branch. @@ -654,6 +667,8 @@ Here's an [example PR](https://github.com/kubernetes/kubernetes/pull/100616), bu [sig-release-x.y-blocking]: https://testgrid.k8s.io/sig-release-1.17-blocking +--- + ### Configure Merge Automation Between the [Code Freeze](#code-freeze) and lifting Code Freeze ([Code Thaw](#code-thaw)) period, merging new code is restricted. The main focus is on fixing existing code and getting green test builds on Testgrid. Preventing new code is implemented by config changes for [tide]. The `master` and current release cycle branch (`release-x.y`) are the only branches affected during this period. From 7fe8d09369f350e34aedf1d80ea7b525cb3adecd Mon Sep 17 00:00:00 2001 From: Matteo Bianchi <37507190+mbianchidev@users.noreply.github.com> Date: Wed, 9 Apr 2025 00:02:31 +0200 Subject: [PATCH 10/26] Update k8s-release-cut.md --- release-engineering/handbooks/k8s-release-cut.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/release-engineering/handbooks/k8s-release-cut.md b/release-engineering/handbooks/k8s-release-cut.md index 05cb4852899..f5099c063be 100644 --- a/release-engineering/handbooks/k8s-release-cut.md +++ b/release-engineering/handbooks/k8s-release-cut.md @@ -331,12 +331,15 @@ Promote the images (`alpha`, `beta` or `rc`) using `kpromo`. The tool will autom The images that need to be promoted depend on the release you're cutting: - **Alpha or Beta release:** just promote the images for the release you're cutting (e.g. v1.20.0-beta.1) -- **The first RC (e.g. v1.20.0-rc.0):** promote the images for the RC and for the next minor alpha release (e.g. v1.21.0-alpha.0) +- **The first RC (e.g. v1.20.0-rc.0):** promote the images for the RC and for the next minor alpha release (e.g. v1.21.0-alpha.0) in two separate kpromo commands/PRs - **The subsequent RCs (e.g. v1.20.0-rc.1):** promote the images for the RC you're cutting (e.g. v1.20.0-rc.0) - **A stable release (e.g. v1.20.0):** promote the images for the release you're cutting and for the RC of the next patch release (e.g. v1.20.1-rc.0) Some examples can be found below: +> [!WARNING] +Even if it's possible to open a single PR for multiple promos it is not recommended due to the extreme amount of resources used by the double promotion job. Promo one image at a time. + ``` kpromo pr --fork=jimangel --tag="v1.29.0-alpha.1" From 6965be7cb31e78ef763903802457a928980e7ad1 Mon Sep 17 00:00:00 2001 From: Matteo Bianchi <37507190+mbianchidev@users.noreply.github.com> Date: Wed, 9 Apr 2025 18:21:54 +0200 Subject: [PATCH 11/26] Add clarification on release branch job creation Update branch-manager.md --- release-engineering/role-handbooks/branch-manager.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/release-engineering/role-handbooks/branch-manager.md b/release-engineering/role-handbooks/branch-manager.md index 44e0594255c..a27da42ae2c 100644 --- a/release-engineering/role-handbooks/branch-manager.md +++ b/release-engineering/role-handbooks/branch-manager.md @@ -641,9 +641,12 @@ Create a PR with this change and wait for it to be merged ([example PR](https:// #### Generate release branch jobs +> [!CAUTION] To minimize drift with tooling changes, this section has been moved to [kubernetes/test-infra][release-branch-job-creation]. +See [kubernetes/test-infra][release-branch-job-creation#release-branch-jobs] for the exact steps to follow. + #### Add a new variant for the kube-cross image Once we have cut the branch, it is a good time to start producing a new kube-cross image. To do this, we first need to add the variant to kube-cross [variants.yaml](https://github.com/kubernetes/release/blob/master/images/build/cross/variants.yaml) file and update the [dependencies.yaml](https://github.com/kubernetes/release/blob/master/dependencies.yaml) file. Here is an [example PR](https://github.com/kubernetes/release/pull/2344/files). From 2199edd430e146f3ea3cb7019903d5ad2d91b038 Mon Sep 17 00:00:00 2001 From: Matteo Bianchi <37507190+mbianchidev@users.noreply.github.com> Date: Tue, 15 Apr 2025 18:40:34 +0200 Subject: [PATCH 12/26] Update k8s-release-cut.md --- release-engineering/handbooks/k8s-release-cut.md | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/release-engineering/handbooks/k8s-release-cut.md b/release-engineering/handbooks/k8s-release-cut.md index f5099c063be..63b228b4928 100644 --- a/release-engineering/handbooks/k8s-release-cut.md +++ b/release-engineering/handbooks/k8s-release-cut.md @@ -291,7 +291,7 @@ If you are releasing an `alpha.1` you will have a command output that has a `bui > [!NOTE] Remember to update the Slack ([thread](#Create-a-thread-on-release-management)) and the release-cut GitHub [issue](#Release-cut-issue) after this step -## 5. No-mock stage +## 6. No-mock stage The following stages, called no-mock, create real artifacts that can be promoted for general use. The process should be near identical to the `no-mock` stages ran prior. @@ -322,7 +322,7 @@ krel release --type=alpha|beta|official|release --branch=release-1.xx --build-ve > [!CAUTION] Do not run the command yet, just copy it somewhere and wait for the image promo to happen first (PR merged + prow job completed). -## 6. Image promotion +## 7. Image promotion Promote the images (`alpha`, `beta` or `rc`) using `kpromo`. The tool will automatically create / open a PR with the images and tags properly promoted. @@ -417,7 +417,7 @@ If the job is stuck in "triggered" (empty clock yellow icon), especially after a > [!NOTE] Remember to update the Slack ([thread](#Create-a-thread-on-release-management)) and the release-cut GitHub [issue](#Release-cut-issue) after the image promotion step -## 7. No-mock release +## 8. No-mock release You should have copied the no mock release command output from the nomock stage previously, now you can run the release: @@ -428,7 +428,7 @@ krel release --nomock --build-version=v1.xx.yy-alpha|beta|rc-z+ > [!NOTE] Remember to update the Slack ([thread](#Create-a-thread-on-release-management)) and the release-cut GitHub [issue](#Release-cut-issue) after this no-mock release step. -## 8. Notify public dev Google group mailinglist +## 9. Notify public dev Google group mailinglist Ensure that you are a moderator of dev@kubernetes.io so you can send messages without passing by the moderation queue, and also an admin of [kubernetes-announce](https://groups.google.com/g/kubernetes-announce). In case you are not part of these groups as moderator/admin, ask to be added in #release-management. @@ -483,8 +483,6 @@ You could run in a reached max recipients quota, in such case communicate with # > [!TIP] Kubernetes-announce might require permissions to post, check you have them or ask your release-manager buddy to post the message for you. -## 9. Notify stakeholders - Post this message in release-management: ``` @@ -522,7 +520,7 @@ krel history --branch release-1.xx --date-from yyyy-mm-dd ## 10. Update schedule on k/website > [!NOTE] -**Only for patch releases** +**Only for patch releases, 1.x.y not for alpha, beta, rc, etc. ** You need to update the release schedule on the official Kubernetes website through schedule-builder, here is how: From 30b01ce0da4278ceb14e33dee9c3ee7d8743aac8 Mon Sep 17 00:00:00 2001 From: Matteo Bianchi <37507190+mbianchidev@users.noreply.github.com> Date: Thu, 17 Apr 2025 13:16:14 +0200 Subject: [PATCH 13/26] Link update Update branch-manager.md --- release-engineering/role-handbooks/branch-manager.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/release-engineering/role-handbooks/branch-manager.md b/release-engineering/role-handbooks/branch-manager.md index a27da42ae2c..b1f90236b21 100644 --- a/release-engineering/role-handbooks/branch-manager.md +++ b/release-engineering/role-handbooks/branch-manager.md @@ -645,7 +645,7 @@ Create a PR with this change and wait for it to be merged ([example PR](https:// To minimize drift with tooling changes, this section has been moved to [kubernetes/test-infra][release-branch-job-creation]. -See [kubernetes/test-infra][release-branch-job-creation#release-branch-jobs] for the exact steps to follow. +See (kubernetes/test-infra)[release-branch-job-creation#release-branch-jobs] for the exact steps to follow. #### Add a new variant for the kube-cross image From b6bba606d57308a90a9b6174c058ae7a8f05dea9 Mon Sep 17 00:00:00 2001 From: Matteo Bianchi <37507190+mbianchidev@users.noreply.github.com> Date: Thu, 17 Apr 2025 13:22:21 +0200 Subject: [PATCH 14/26] Update branch-manager.md --- release-engineering/role-handbooks/branch-manager.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/release-engineering/role-handbooks/branch-manager.md b/release-engineering/role-handbooks/branch-manager.md index b1f90236b21..662593893e5 100644 --- a/release-engineering/role-handbooks/branch-manager.md +++ b/release-engineering/role-handbooks/branch-manager.md @@ -508,6 +508,9 @@ The following are some ways to determine if the release process was successful: ### Post-release Activities +> [!IMPORTANT] +> These activities are needed after `rc.0` releases only + #### Update kubekins-e2e variants Set the `K8S_RELEASE` marker for the current release variant to `stable-x.y` which was currently `latest-x.y` in the [`variants.yaml` file for `kubekins-e2e`](https://github.com/kubernetes/test-infra/blob/fa43d4a7a6c88c0dedd0db83b250cec485b60736/images/kubekins-e2e/variants.yaml). ([reference PR review comment](https://github.com/kubernetes/test-infra/pull/13870#discussion_r313628808)) From bc04d5bf436ab12ddd26d9771d0163f08abda9f4 Mon Sep 17 00:00:00 2001 From: Matteo Bianchi <37507190+mbianchidev@users.noreply.github.com> Date: Thu, 17 Apr 2025 13:25:05 +0200 Subject: [PATCH 15/26] Update branch-manager.md --- release-engineering/role-handbooks/branch-manager.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/release-engineering/role-handbooks/branch-manager.md b/release-engineering/role-handbooks/branch-manager.md index 662593893e5..359e011855f 100644 --- a/release-engineering/role-handbooks/branch-manager.md +++ b/release-engineering/role-handbooks/branch-manager.md @@ -538,7 +538,7 @@ Example `1.18.0-alpha.1` release issue template: https://github.com/kubernetes/s Reach out to SIG Scalability to ensure a new branch is cut in [sigs.k8s.io/perf-tests](https://github.com/kubernetes/perf-tests/) repository. -In case the release branch is missing - we will end up with an issue we had for 1.30: https://github.com/kubernetes/kubernetes/issues/124119 +In case the release branch is missing - we will end up with an issue we had for 1.30, see [#124119](https://github.com/kubernetes/kubernetes/issues/124119) ## Branch Management @@ -648,7 +648,7 @@ Create a PR with this change and wait for it to be merged ([example PR](https:// To minimize drift with tooling changes, this section has been moved to [kubernetes/test-infra][release-branch-job-creation]. -See (kubernetes/test-infra)[release-branch-job-creation#release-branch-jobs] for the exact steps to follow. +See [kubernetes/test-infra](release-branch-job-creation#release-branch-jobs) for the exact steps to follow. #### Add a new variant for the kube-cross image From 1cbc6800181c4ca825fe635661c0b9d04bcc40f1 Mon Sep 17 00:00:00 2001 From: Matteo Bianchi <37507190+mbianchidev@users.noreply.github.com> Date: Tue, 22 Apr 2025 17:48:23 +0200 Subject: [PATCH 16/26] Removed embargo time (not enforced anymore for official release) --- release-engineering/role-handbooks/branch-manager.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/release-engineering/role-handbooks/branch-manager.md b/release-engineering/role-handbooks/branch-manager.md index 359e011855f..eefdfe6ef55 100644 --- a/release-engineering/role-handbooks/branch-manager.md +++ b/release-engineering/role-handbooks/branch-manager.md @@ -400,9 +400,7 @@ In addition to `v1.18.n` this will also build and stage the subsequent patch's `rc.0`, in this example `v1.18.(n+1)-rc.0`. Similar to [creating a new branch](#branch-creation), the staging step will take about twice as long, the release step will also take a couple of minutes more. -However, there is an embargo policy which requires the nomock release publication happens after 4 pm Pacific (see [Release Team Lead Handbook](https://github.com/kubernetes/sig-release/tree/master/release-team/role-handbooks/release-team-lead#week-12) for details), make sure to coordinates with other roles and follow the timeline. - -To better prepare and see what to expect, this is a sequence of events that took place on past [official release days](https://docs.google.com/document/d/1K0B91lgeEiJTbT602VloA5arb6AkaTif-MLryaHmlYc/edit?usp=sharing). +To better prepare and see what to expect, this is a sequence of events that took place on past official release days [1.14-1.16](https://docs.google.com/document/d/1K0B91lgeEiJTbT602VloA5arb6AkaTif-MLryaHmlYc/edit?usp=sharing), and the most recent [1.32](https://kubernetes.slack.com/archives/CJH2GBF7Y/p1733928000598509). #### Security fixes From 280d4f001cc0cb897bb048833bff0724b06d1935 Mon Sep 17 00:00:00 2001 From: Matteo Bianchi <37507190+mbianchidev@users.noreply.github.com> Date: Tue, 22 Apr 2025 17:51:11 +0200 Subject: [PATCH 17/26] Update k8s-release-cut.md --- release-engineering/handbooks/k8s-release-cut.md | 1 + 1 file changed, 1 insertion(+) diff --git a/release-engineering/handbooks/k8s-release-cut.md b/release-engineering/handbooks/k8s-release-cut.md index 63b228b4928..bc9fb0be17d 100644 --- a/release-engineering/handbooks/k8s-release-cut.md +++ b/release-engineering/handbooks/k8s-release-cut.md @@ -397,6 +397,7 @@ You might wanna ping @release-managers on Slack to speed this process up. **After** the Pull Request is merged and **before** starting the official nomock release step, watch [the following Prow Job](https://prow.k8s.io/?job=post-k8sio-image-promo) and wait for it to succeed. If the job fails, look into the root cause and message the Slack thread. In many cases a rerun is fine (such as 429 too many requests). +A resolution to this problem is being discussed [here](https://docs.google.com/document/d/1S5NlwOdtXKEWXvdkGRvVTsNip6niyhb52FSNk8-uJiQ/edit?resourcekey=0-IRBLdEtrNfK9e7z1w4y9ng&tab=t.0). To check the logs after failure, click on the spyglass eye icon, otherwise you can look at the POD logs when the job is running by clicking on the paper icon. From dfe96c86508755db87bbf16efd6da5b642344647 Mon Sep 17 00:00:00 2001 From: Matteo Bianchi <37507190+mbianchidev@users.noreply.github.com> Date: Wed, 23 Apr 2025 20:51:56 +0200 Subject: [PATCH 18/26] Update k8s-release-cut.md --- release-engineering/handbooks/k8s-release-cut.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/release-engineering/handbooks/k8s-release-cut.md b/release-engineering/handbooks/k8s-release-cut.md index bc9fb0be17d..580badfdc79 100644 --- a/release-engineering/handbooks/k8s-release-cut.md +++ b/release-engineering/handbooks/k8s-release-cut.md @@ -516,6 +516,9 @@ To valorize the release stats required in the release cut issue, run this comman krel history --branch master --date-from yyyy-mm-dd krel history --branch release-1.xx --date-from yyyy-mm-dd + +#example +krel history --branch release-1.33 --date-from 2025-04-23 ``` ## 10. Update schedule on k/website From e06c13be409f81baf02f84f3daca0d80de109ac7 Mon Sep 17 00:00:00 2001 From: Matteo Bianchi <37507190+mbianchidev@users.noreply.github.com> Date: Wed, 23 Apr 2025 22:45:09 +0200 Subject: [PATCH 19/26] Update k8s-release-cut.md --- .../handbooks/k8s-release-cut.md | 23 +++++++------------ 1 file changed, 8 insertions(+), 15 deletions(-) diff --git a/release-engineering/handbooks/k8s-release-cut.md b/release-engineering/handbooks/k8s-release-cut.md index 580badfdc79..24b41dadc14 100644 --- a/release-engineering/handbooks/k8s-release-cut.md +++ b/release-engineering/handbooks/k8s-release-cut.md @@ -248,7 +248,7 @@ If the issue is open you must stop the release process and inform #release-manag ## 5. Mock stage and Mock release -Mock stages and releases are non-destructive and can be reran upon failure. To begin the mock stage, run the following `krel stage` command (replace the stage with the appropriate "type"). +Mock stages and mock releases are non-destructive and can be reran upon failure. To begin the mock stage, run the following `krel stage` command (replace the stage with the appropriate "type"). Run the following command from within the release repo directory. @@ -310,17 +310,10 @@ krel stage --type official --branch release-1.xx --nomock > [!NOTE] Remember to update the Slack ([thread](#Create-a-thread-on-release-management)) and the release-cut GitHub [issue](#Release-cut-issue) after this no-mock stage step -This also takes about **2 hours**. Once passed, move on to the `kpromo` command - -``` -# take the output of the previous command from the logs and copy this command somewhere, do NOT execute this. - -# It should look like this: -krel release --type=alpha|beta|official|release --branch=release-1.xx --build-version=v1.xx.yy-alpha|beta|rc-z+ -``` +This also takes about **2 hours**. Once passed, move on to the `kpromo` command. > [!CAUTION] -Do not run the command yet, just copy it somewhere and wait for the image promo to happen first (PR merged + prow job completed). +Do not run the release command yet, just copy it somewhere and wait for the image promo to happen first (which means the image promo PR gets merged and the prow job has to be completed). ## 7. Image promotion @@ -332,8 +325,8 @@ The images that need to be promoted depend on the release you're cutting: - **Alpha or Beta release:** just promote the images for the release you're cutting (e.g. v1.20.0-beta.1) - **The first RC (e.g. v1.20.0-rc.0):** promote the images for the RC and for the next minor alpha release (e.g. v1.21.0-alpha.0) in two separate kpromo commands/PRs -- **The subsequent RCs (e.g. v1.20.0-rc.1):** promote the images for the RC you're cutting (e.g. v1.20.0-rc.0) -- **A stable release (e.g. v1.20.0):** promote the images for the release you're cutting and for the RC of the next patch release (e.g. v1.20.1-rc.0) +- **The subsequent RCs (e.g. v1.20.0-rc.1):** promote the images for the RC you're cutting (e.g. v1.20.0-rc.1) +- **A stable release (e.g. v1.33.0):** promote the images for the release you're cutting (e.g. v1.33.0) Some examples can be found below: @@ -396,7 +389,7 @@ You might wanna ping @release-managers on Slack to speed this process up. > [!CAUTION] **After** the Pull Request is merged and **before** starting the official nomock release step, watch [the following Prow Job](https://prow.k8s.io/?job=post-k8sio-image-promo) and wait for it to succeed. -If the job fails, look into the root cause and message the Slack thread. In many cases a rerun is fine (such as 429 too many requests). +If the job fails, look into the root cause and message the Slack thread. In many cases a rerun is fine (such as 429 too many requests or limit quota exceeded). A resolution to this problem is being discussed [here](https://docs.google.com/document/d/1S5NlwOdtXKEWXvdkGRvVTsNip6niyhb52FSNk8-uJiQ/edit?resourcekey=0-IRBLdEtrNfK9e7z1w4y9ng&tab=t.0). To check the logs after failure, click on the spyglass eye icon, otherwise you can look at the POD logs when the job is running by clicking on the paper icon. @@ -420,10 +413,10 @@ Remember to update the Slack ([thread](#Create-a-thread-on-release-management)) ## 8. No-mock release -You should have copied the no mock release command output from the nomock stage previously, now you can run the release: +You should have copied the nomock release command output from the nomock stage previously run, now you can execute the release command as follows: ``` -krel release --nomock --build-version=v1.xx.yy-alpha|beta|rc-z+ +krel release --type=alpha|beta|official|release --branch=release-1.xx --build-version=v1.xx.yy-alpha|beta|rc-z+ ``` > [!NOTE] From 77e667b2ca84a099ea7ce4b1095fb2c10d055edb Mon Sep 17 00:00:00 2001 From: Matteo Bianchi <37507190+mbianchidev@users.noreply.github.com> Date: Wed, 23 Apr 2025 23:07:47 +0200 Subject: [PATCH 20/26] Update k8s-release-cut.md --- release-engineering/handbooks/k8s-release-cut.md | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/release-engineering/handbooks/k8s-release-cut.md b/release-engineering/handbooks/k8s-release-cut.md index 24b41dadc14..aea0877c2d5 100644 --- a/release-engineering/handbooks/k8s-release-cut.md +++ b/release-engineering/handbooks/k8s-release-cut.md @@ -411,7 +411,7 @@ If the job is stuck in "triggered" (empty clock yellow icon), especially after a > [!NOTE] Remember to update the Slack ([thread](#Create-a-thread-on-release-management)) and the release-cut GitHub [issue](#Release-cut-issue) after the image promotion step -## 8. No-mock release +## 8. No-mock release You should have copied the nomock release command output from the nomock stage previously run, now you can execute the release command as follows: @@ -514,10 +514,18 @@ krel history --branch release-1.xx --date-from yyyy-mm-dd krel history --branch release-1.33 --date-from 2025-04-23 ``` -## 10. Update schedule on k/website +## 10. Post release tasks + +### [RC.0 only] Post rc.0 release tasks + +See [here](post-rc0-release-tasks.md) for the complete list of post rc.0 release tasks. + +It resides in a different document to mainain bite-sized SRE style handbooks. + +### [Patch Releases only] Update schedule on k/website > [!NOTE] -**Only for patch releases, 1.x.y not for alpha, beta, rc, etc. ** +Only for patch releases, 1.x.y not for alpha, beta, rc or official. You need to update the release schedule on the official Kubernetes website through schedule-builder, here is how: From 3a92ec893fd5bb6d3070f73b4867d2df629f1b85 Mon Sep 17 00:00:00 2001 From: Matteo Bianchi <37507190+mbianchidev@users.noreply.github.com> Date: Wed, 23 Apr 2025 23:08:20 +0200 Subject: [PATCH 21/26] Update k8s-release-cut.md --- release-engineering/handbooks/k8s-release-cut.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/release-engineering/handbooks/k8s-release-cut.md b/release-engineering/handbooks/k8s-release-cut.md index aea0877c2d5..01ad7d3993a 100644 --- a/release-engineering/handbooks/k8s-release-cut.md +++ b/release-engineering/handbooks/k8s-release-cut.md @@ -1,4 +1,4 @@ -# Cutting a Kubernetes patch release +# Cutting a Kubernetes release A step by step guide for cutting Kubernetes patch releases. At a high-level: From 5c804694cc634d1167acb484147e052a2d1ebcdc Mon Sep 17 00:00:00 2001 From: Matteo Bianchi <37507190+mbianchidev@users.noreply.github.com> Date: Wed, 23 Apr 2025 23:11:40 +0200 Subject: [PATCH 22/26] Update branch-manager.md --- release-engineering/role-handbooks/branch-manager.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/release-engineering/role-handbooks/branch-manager.md b/release-engineering/role-handbooks/branch-manager.md index eefdfe6ef55..32e2e41a76b 100644 --- a/release-engineering/role-handbooks/branch-manager.md +++ b/release-engineering/role-handbooks/branch-manager.md @@ -777,7 +777,7 @@ Update the `milestoneapplier` plugin configs for the following repos to the **_n Example PRs: -- [1.18](https://github.com/kubernetes/test-infra/pull/16821) +- [1.33](https://github.com/kubernetes/test-infra/pull/34693) ### Branch Fast Forward From f6b537a598cda6a2bcaa39de4a595c7978607185 Mon Sep 17 00:00:00 2001 From: Matteo Bianchi <37507190+mbianchidev@users.noreply.github.com> Date: Wed, 23 Apr 2025 23:14:26 +0200 Subject: [PATCH 23/26] Update k8s-release-cut.md --- release-engineering/handbooks/k8s-release-cut.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/release-engineering/handbooks/k8s-release-cut.md b/release-engineering/handbooks/k8s-release-cut.md index 01ad7d3993a..03539411797 100644 --- a/release-engineering/handbooks/k8s-release-cut.md +++ b/release-engineering/handbooks/k8s-release-cut.md @@ -520,9 +520,14 @@ krel history --branch release-1.33 --date-from 2025-04-23 See [here](post-rc0-release-tasks.md) for the complete list of post rc.0 release tasks. -It resides in a different document to mainain bite-sized SRE style handbooks. +Such list resides in a different document to mainain this one in a bite-sized SRE style format. -### [Patch Releases only] Update schedule on k/website +### [Stable only] Code Thaw + +Code thaw means you need to lift the code freeze, [here](https://github.com/kubernetes/sig-release/blob/master/release-engineering/role-handbooks/branch-manager.md#code-thaw) +are the docs on how to do it, with an example PR. + +### [Patch only] Update schedule on k/website > [!NOTE] Only for patch releases, 1.x.y not for alpha, beta, rc or official. From 0940dac688ea6f2d1b6caefd719c407649b71e61 Mon Sep 17 00:00:00 2001 From: Matteo Bianchi <37507190+mbianchidev@users.noreply.github.com> Date: Wed, 23 Apr 2025 23:15:39 +0200 Subject: [PATCH 24/26] Update k8s-release-cut.md --- release-engineering/handbooks/k8s-release-cut.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/release-engineering/handbooks/k8s-release-cut.md b/release-engineering/handbooks/k8s-release-cut.md index 03539411797..b2513a0e431 100644 --- a/release-engineering/handbooks/k8s-release-cut.md +++ b/release-engineering/handbooks/k8s-release-cut.md @@ -366,7 +366,7 @@ kpromo pr --fork= --tag="v1.xx.yy-alpha|beta|rc-z" --interactive= `FATA growing manifest with image filter [] and tag [1.29.0-alpha.1]: no images survived filtering; double-check your --filter_* flag(s) for typos` means you forgot the letter "v" in the tag parameter. -Once run the command its output should look like this: +Once you've run the command its output should look like this: ``` INFO Successfully created PR #xxxx From c656fa8293b2b57980dc3f4f31131ae30d062866 Mon Sep 17 00:00:00 2001 From: Matteo Bianchi <37507190+mbianchidev@users.noreply.github.com> Date: Thu, 24 Apr 2025 14:53:30 +0200 Subject: [PATCH 25/26] Update k8s-release-cut.md --- .../handbooks/k8s-release-cut.md | 55 +++---------------- 1 file changed, 8 insertions(+), 47 deletions(-) diff --git a/release-engineering/handbooks/k8s-release-cut.md b/release-engineering/handbooks/k8s-release-cut.md index b2513a0e431..d282f6948b2 100644 --- a/release-engineering/handbooks/k8s-release-cut.md +++ b/release-engineering/handbooks/k8s-release-cut.md @@ -317,50 +317,7 @@ Do not run the release command yet, just copy it somewhere and wait for the imag ## 7. Image promotion -Promote the images (`alpha`, `beta` or `rc`) using `kpromo`. The tool will automatically create / open a PR with the images and tags properly promoted. - -### Which image should be promoted? - -The images that need to be promoted depend on the release you're cutting: - -- **Alpha or Beta release:** just promote the images for the release you're cutting (e.g. v1.20.0-beta.1) -- **The first RC (e.g. v1.20.0-rc.0):** promote the images for the RC and for the next minor alpha release (e.g. v1.21.0-alpha.0) in two separate kpromo commands/PRs -- **The subsequent RCs (e.g. v1.20.0-rc.1):** promote the images for the RC you're cutting (e.g. v1.20.0-rc.1) -- **A stable release (e.g. v1.33.0):** promote the images for the release you're cutting (e.g. v1.33.0) - -Some examples can be found below: - -> [!WARNING] -Even if it's possible to open a single PR for multiple promos it is not recommended due to the extreme amount of resources used by the double promotion job. Promo one image at a time. - -``` -kpromo pr --fork=jimangel --tag="v1.29.0-alpha.1" - -# RC -kpromo pr --fork=jimangel --tag="v1.32.0-rc.1" -#or -kpromo pr --fork=jimangel --tag="v1.29.0-rc.0" --tag="v1.30.0-alpha.0" -``` - -### Run kpromo (opens the PR) - -> [!IMPORTANT] -If promoting the first release candidate (rc), include the next release alpha as follows. - -``` -kpromo pr --fork= --tag="v1.xx.yy-rc-z" --tag="v1.xx+1.yy-alpha-0" -``` - -> [!TIP] -Multiple tags can be specified with the `kpromo` pr command - -This is the standard command to be run ([source](https://github.com/kubernetes-sigs/promo-tools/blob/main/docs/promotion-pull-requests.md#promoting-images)): - -``` -# --interactive=true asks confirmation before every step, highly recommended - -kpromo pr --fork= --tag="v1.xx.yy-alpha|beta|rc-z" --interactive=true -``` +Complete docs can be found [here](https://github.com/kubernetes-sigs/promo-tools/blob/main/docs/promotion-pull-requests.md) > [!WARNING] `FATA growing manifest with image filter [] and tag [1.29.0-alpha.1]: no images survived filtering; double-check your --filter_* flag(s) for typos` @@ -373,12 +330,16 @@ INFO Successfully created PR #xxxx INFO Successfully created PR: https://github.com/kubernetes/k8s.io/pull/xxxx ``` -### Merge PR +### Merge promo PR Wait for the generated PR to be merged (after approval and LGTM). You might wanna ping @release-managers on Slack to speed this process up. -**The PR is not blocked by anything but approvals / review.** +**The PR has no dependencies outside of approvals / review.** + +> [!CAUTION] +In case a blocking test goes red during the release cut, you should keep the PR held and reach a consesus with @release-managers that the promo can continue. +e.g. if a test went red for infra flakyness "Node not ready" it's probably ok to continue, but it's always better to double check. > [Example PR](https://github.com/kubernetes/k8s.io/pull/3024). @@ -555,5 +516,5 @@ git push -u origin schedule-updates-oct-2024 ## Cleanup -Close out related issues, open new issues if you find any, and ask Google admins to revoke your personal access to GCP if it is no longer required. +Close the release cut issue and possibly related issues, open new issues if you find any docs or feature missing, and ask Google admins to revoke your personal access to GCP if it is no longer required. You might want to delete your GitHub access token too. From c67954a8266d6d7e2ff0f335c57d1239e687161c Mon Sep 17 00:00:00 2001 From: Matteo Bianchi <37507190+mbianchidev@users.noreply.github.com> Date: Thu, 24 Apr 2025 15:00:20 +0200 Subject: [PATCH 26/26] Update cut-release.md --- .github/ISSUE_TEMPLATE/cut-release.md | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/.github/ISSUE_TEMPLATE/cut-release.md b/.github/ISSUE_TEMPLATE/cut-release.md index 02733b21bd0..7367da64895 100644 --- a/.github/ISSUE_TEMPLATE/cut-release.md +++ b/.github/ISSUE_TEMPLATE/cut-release.md @@ -4,7 +4,7 @@ about: Create a tracking issue for a release cut title: Cut v1.x.y-{alpha,beta,rc}.z release labels: sig/release, area/release-eng --- -## Scheduled to happen: +## Scheduled to happen: _Note for v1.x.0 releases: Having this issue in open state will stop the periodic run of [`krel fast-forward`](https://testgrid.k8s.io/sig-release-releng-blocking#git-repo-kubernetes-fast-forward) @@ -81,13 +81,14 @@ Help? Ring @release-managers on slack! - [ ] Release - NoMock Run - [ ] Stage - - [ ] Image Promotion: + - [ ] Image Promotion PR: - [ ] Release -- [ ] Notify #release-management: -- [ ] Send notification: -- [ ] Update [`schedule.yaml` file](https://github.com/kubernetes/website/blob/main/data/releases/schedule.yaml) with the latest release using [`schedule-builder`](https://github.com/kubernetes/release/tree/master/cmd/schedule-builder) -- [ ] Collect metrics and add them to the `Release steps` table below - +- [ ] Send email notification: +- [ ] Notify #release-management on Slack: +- [ ] Update [`schedule.yaml` file](https://github.com/kubernetes/website/blob/main/data/releases/schedule.yaml) with the latest release using [`schedule-builder`](https://github.com/kubernetes/release/tree/master/cmd/schedule-builder) (_patch releases only_) +- [ ] Collect metrics and add them to the `Release steps` table below through `krel history --branch release-1.xx --date-from yyyy-mm-dd` (current date) + + ## Release Tools Version