[1.32] Signal Subteam Lead Progress

[drewhagen]

✅ Kubernetes 1.32 is released!

⚠️ The following is a work in progress and somewhat experimental, to create an issue to track my work similar to the Release Lead. Follows the Release Signal Handbook that details the team's responsibilities.

Onboarding

• Select shadows that will be on the team

  • Created Slack group chat with the shadows and gave them the grand welcome 🎉

• Coordinate with release lead to make teams.yaml updates

• Add the lead and (more experienced) shadows as milestone maintainers in the teams.yaml.

• Update the release-signal team in the teams.yaml which grants access to the ci signal project board and the bug triage project board.

  • drewhagen added
  • awaiting merge of this PR: Adding Release Signal Shadows org#5161
  • awaiting merge of this PR: update groups.yaml for Release Signal shadows k8s.io#7310
  • awaiting merge of this PR: add release signal shadows to the release doc #2633
    (📜 Note: should add the groups and Release Team md updates to the SIG Release Signal handbook as it only mentions teams.yaml)

• Setup CI-Signal Project Board

  • Copy prior project
  • Add v1.32 to K8s Release
  • Update the filter

• Setup Bug Triage Project Board

  • Set up periodic syncing PR against Test Infra
    • Placed hold on this PR until Sept 9th (first day of cycle)
    • Check when merged
  • Update bug triage board name here for the current cycle 1.32
  • Update the filter to milestone:v1.32
  • Edit the Responsible field with the new 1.32 Release Signal team members
    • Cleaned up values from 1.31
  • Set Status issue to Pending inclusion for all open issues and PRs targeting the current release cycle

• Double check that selected shadows are members of the K8s org (see Release Signal handbook)

  • Release Signal 1.32 members already appear to be apart of K8s org

• Send out the lettucemeet on general Shadow Orientation

  • Verify that all shadows signed up for orientation

• Work on a Release Signal slide deck for team Orientation, similar to Release Docs one

  • Almost ready to present
  • Clarify team objectives and responsibilities
  • Scheduled this for Sept 24th (9-10am), all shadows are available to attend.
  • Introduce how to use our toolset of GitHub, Slack, Testgrid, Triage, Spyglass, Prow and Tide

• Vyom: Create a one-pager for the release signal handbook #2612 ( Give a comprehensive TL;DR of handbook)

  • Drew: Review and give feedback
  • Try to get this merged in

• Supporting the Shadows: Figure out a way to open up my calendar for shadows to schedule 1:1s similar to what Laura did for v1.27
  - Going to use Lettuce Meet for this.
  - Also sent out another Lettuce Meet to find a good time for optional "Working Office Hours" for shadows to jump in and we can work on some things as a group.

• Finish onboarding meeting with shadows to go through the slide deck, demo the tools and emphasize the handbook.

• Create Assignment Excel Sheet - Example (see Release Signal handbook for Task Assignee and Main Coordinator)

  • Added all of the 1.32 shadows to the Excel Sheet

Early Release (~week 0-5, Sept. 3rd - Oct. 11th)**

📆 Release cycle begins Monday Sept 9th.
⚠️ Drew will be OOO from Sept 11th - 14th. Vyom agrees to be main coordinator of the test grid during this time.

• Monitor master-blocking and master-informing dashboards daily and ensure all failures and flakes are tracked via open issues. See Opening Issues for how to write an effective issue.
  • Start this work
• Start pinging stale or long-running issues / PRs so that an early assessment can be made if these should be moved out of the current release cycle's milestone.
  • Start this work with CI Signal issues/PR
  • Start this work with Bug Triage board issues/PR
    • Assign Bug Triage issues to everyone on the team
• Fill the Assignment Excel Sheet
  • The team is filling it out
• Begin maintaining CI Signal Project Board
• Assign new milestone labels to the open issues from previous release
  • Commented that 1.32 Release has started and encouraged some movement
  • Assign a member of the Release Signal team, and have that member follow up on the issue with the owners
• Updating/Extending scripts/actions that populate the Bug Triage and CI Signal project board with relevant issues/PRs.
• Look into the automated processes? (builds, scripts, actions?)
• Familiarize myself with major enhancements and fixes planned by each SIG for this release so that you can have context in advance of when you will need to identify incoming bugs as being associated with a work focus in the current release.

Delegate No/Go Signal for Alpha Release Cuts

Following guidance in the handbook and assignments in the spreadsheet, see that the assigned member of Release Signal gives the Go/No Go signal to Branch Management.

• 1.32.0-alpha.1 released | Week 4 - Tuesday 1st October 2024
  • One week in advance, start to remind folks about this deadline
  • Was delayed due to failures with SIG Storage - but was resolved.
• 1.32.0-alpha.2 released | Week 6 - Tuesday 15th October 2024
  - [x ] One week in advance, start to remind folks about this deadline
• 1.32.0-alpha.3 released | Week 8 - Tuesday 29th October 2024

Mid-Release Cycle (~week 6-9, Oct 12th - Nov 7th)

📆⚠️ Code Freeze is 02:00 UTC Friday 8th November 2024 / 19:00 PDT Thursday 7th November 2024

• Monitor the master-blocking, master-informing, release-x.y-blocking and release-x.y-informing dashboards on a daily basis.
• Closely work with SIGs/WGs and test owners to fix failing and flaky tests.
• Ensure that issues and PRs targeting the release have the correct milestone and priority labels.
  • Linked issues and PRs tracked in the current milestone should have priority and kind labels.
• Ensure that priority/critical-urgent issues and PRs are not blocked.
  • If such a priority/critical-urgent issue/PR is found, ping the owner and owner SIGs and alert the release team leads.
• Ping all issues and PRs to remind contributors and SIG leads about the code freeze approaching in a month or so.
  • Ping issues at least two times to ensure deadlines are communicated correctly, usually once this stage starts and a couple of days before the code freeze starts.
• Prioritize pinging issues/PRs that haven't been updated for a longer time (see handbook)

Delegate No/Go Signal for Beta Release Cut

Following guidance in the handbook and assignments in the spreadsheet, see that the assigned member of Release Signal gives the Go/No Go signal to Branch Management.

• [ x] 1.32.0-beta.0 released (Tuesday Nov 5th 2024)

Preparing for Code Freeze (before Thursday Nov 7th)

• Monitor new issues/PRs targeting the milestone, removing those that are not part of the release cycle.
• Watch for existing issues/PRs and ensure the stability of the codebase before Code Freeze.
• Use queries to monitor PRs and ensure they are appropriately categorized (approved, lgtm, on-hold, etc.):
  • PRs in the merge pool: is:pr is:open milestone:v1.28 label:approved label:lgtm -label:do-not-merge/hold
  • PRs on hold: is:pr is:open milestone:v1.28 label:approved label:lgtm label:do-not-merge/hold
  • PRs with lgtm but no approval: is:pr is:open milestone:v1.28 -label:approved label:lgtm
  • PRs with approval but no lgtm: is:pr is:open milestone:v1.28 label:approved -label:lgtm

Day of Code Freeze (on Thursday Nov 7th)

• Help contributors get approval and necessary labels for their PRs.
• Ensure no items are removed from the milestone until the next day after Code Freeze.
• Prioritize PRs with approved and lgtm labels in the milestone after Code Freeze.

Day After Code Freeze (Friday Nov 8th)

• Work with the Release Lead team in handling Code Freeze exception requests, adjusting issues accordingly.
• Remove non-approved issues/PRs out of the milestone.
• Communicate in a friendly manner with issue/PR authors when changing/removing milestones to maintain transparency.
• Coordinate with the Release Team Lead before removing issues/PRs from the milestone.
• Move issues/PRs to the next milestone using /milestone v1.33 if they are planned for the next cycle or had recent activity.
• Clear issues/PRs with no activity using the /milestone clear command.
• Coordinate with the SIG and Release Team Lead for critical/urgent issues to determine if they should stay in the release or be deprioritized.

📆 KubeCon NA and Contrib Summit (Monday Nov 11 - Friday Nov 15th)

• Enjoy the conference 😄 Drew will be attending

One Week After Code Freeze (Thursday Nov 14th)

• Depending on testgrid state, merge queue length, and Release Team Lead’s decision, allow some extra time for PRs to get merged.
• Continue monitoring PRs using the same queries. (see the handbook)
• Prioritize kind/bug PRs but allow other PRs if there is no risk of destabilizing the release (requires an exception).

Week 1 of Code Freeze (Thursday Nov 7th) until Code Thaw (Wednesday Dec 11th)

• Continue to monitor and remove issues/PRs from the milestone if they lack activity or confirmation.
• Confirm with the Release Team Lead before removing issues/PRs from the milestone.
• For issues/PRs with recent activity or next milestone confirmation, move them using /milestone v1.xx.
• For inactive issues/PRs, clear them using /milestone clear.
• Coordinate with SIGs and the Release Team Lead for critical/urgent issues to adjust their priority if necessary.

Delegate No/Go Signal for Release Candidate Cuts

Following guidance in the handbook and assignments in the spreadsheet, see that the assigned member of Release Signal gives the Go/No Go signal to Branch Management.

• 1.32.0-rc.0 released (Tuesday Nov 26th 2024)
• 1.32.0-rc.1 released (Tuesday Dec 3rd 2024)

Code Thaw (One Week Before Release Target Date?? December 3rd?)

• Ensure major breakage bugs are fixed immediately.
• Ensure any pending release-blocking PRs are approved and merged.
• Remove non-essential items from the release.
• Ensure tests on informing and blocking dashboards are green.
• Communicate with issue owners and SIG leads to get updates on outstanding items within hours (consider time zones).
• Monitor commits to master and release branches for unexpected changes and triage for destabilizing risk.
• For unexpected commits, ensure proper exception justification, tests, and documentation are in place.

Delegate No/Go Signal for Release Cut

Following guidance in the handbook, see that a member of Release Signal signals to Branch Management.

• 1.32.0 released (Wednesday Dec 11th 2024)

[k8s-ci-robot]

@drewhagen: The label(s) /label sig-release cannot be applied. These labels are supported: api-review, tide/merge-method-merge, tide/merge-method-rebase, tide/merge-method-squash, team/katacoda, refactor. Is this label configured under labels -> additional_labels or labels -> restricted_labels in plugin.yaml?

In response to this:

/label sig-release

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[drewhagen]

/sig release

[drewhagen]

/assign

[Vyom-Yadav]

Nice job on creating this checklist.

Mid-Release Cycle (~week 6-9, Oct 12th - Nov 7th)
📆⚠️ Code Freeze is 02:00 UTC Friday 8th November 2024 / 19:00 PDT Thursday 7th November 2024

Monitor the master-blocking, master-informing, release-x.y-blocking and release-x.y-informing dashboards on a daily basis.

This is wrongly documented in the handbook (which I'll update alongside creating a one-pager).
The release-x.y boards are set alongside the rc-0 cut. Usually, it takes 2-3 days, sometimes even a week, to get those boards set up and stabilized.

Often, some job misconfiguration/some other bugs can be patched by the Release Signal Lead. So I'll include this and monitor/help with setting up release-x.y boards after rc-0 cut to the list.

I've assigned myself to create a one-pager.

[drewhagen]

@Vyom-Yadav Are there any GitHub Actions or other automated code that I should be aware of?

I'm curious to gain more knowledge to cover this if needed:

Updating/Extending scripts/actions that populate the Bug Triage and CI Signal project board with relevant issues/PRs.

[drewhagen]

Kubernetes 1.32 has been released :)

⚠️ My thoughts addressing the experiment of using this checklist for Release Signal Lead:

This high level checklist felt useful at times and like additional overhead at others. It was useful to organize and work through tasks across the release. In the onboarding section, this was more useful when there were a lot of very specific action items by the lead that needed to be done.

In the second half of the release, I found myself not using/updating this checklist very much because:

1. It felt like some of the tasks were vague/general descriptions of work needed rather than actionable items.
2. Many of the tasks were team tasks rather than lead tasks and were easily delegated.
3. This checklist felt like unnecessary overhead over doing certain tasks.

On the other hand, I acknowledge the purpose of having a checklist like this is for succinct transparency and visibility on how this subteam is operating throughout the cycle. If this is something we want to take more commitment to in future releases, that makes sense. Perhaps there is an easier tool than a GitHub checklist, but I acknowledge this aligns with the Release Lead's own checklist.

/closing this issue