Skip to content

Commit 588283d

Browse files
committed
Update the TPP reboot checklist.
Haven't had to use it for a long time, as TPP haven't requested a reboot for a while. Removed some obsolete instructions, and also updated to account for a reboot we initiate rather than TPP.
1 parent d94ee09 commit 588283d

1 file changed

Lines changed: 15 additions & 14 deletions

File tree

.github/ISSUE_TEMPLATE/tpp-reboot.md

Lines changed: 15 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -3,19 +3,22 @@ name: TPP reboot
33
about: Handling
44
title: "[TPP reboot]"
55
labels: ''
6-
assignees: pipeline-team
76

87
---
98

10-
Checklist for TPP reboot, to be created when TPP inform us of a maintenance window:
9+
Checklist for TPP reboot
10+
1111

1212

1313

1414
Preparation:
1515

16-
- [ ] bring window start forward 10min, and extend window close by 30min, as TPP can be a bit optimistic about how long they need
17-
- [ ] schedule [maintenance window](https://support.freshstatus.io/en/support/solutions/articles/50000001851-how-to-create-a-planned-maintenance-incident-) in [freshstatus](https://bennettoxford.freshstatus.io/admin/incidents/public)
18-
- [ ] communicate advance notice of window to users on #opensafely and #opensafely-users Slack channels
16+
- [ ] If TPP have inform us of a maintenance window, bring window start
17+
forward 10min, and extend window close by 30min, as TPP can be a bit
18+
optimistic about how long they need. If we are rebooting for other reasons,
19+
decide on window.
20+
- [ ] communicate advance notice of window to users on #opensafely and
21+
#opensafely-users Slack channels
1922
- [ ] set Slack reminder for window start
2023

2124
When window starts:
@@ -24,20 +27,18 @@ When window starts:
2427

2528
See the [preparing for reboot](https://github.com/opensafely-core/backend-server/blob/main/playbook.md#preparing-for-reboot) section of the playbook for how to do this.
2629

27-
These commands can also be scheduled in advance if the maintenance window is out of hours, see the [planned maintenance](https://github.com/opensafely-core/backend-server/blob/main/playbook.md#planned-maintenance) section for more details.
30+
Perform reboot
31+
- [ ] if we are doing the reboot, then run `sudo reboot`. If TPP are stoping the VM, let them do it.
2832

29-
When window closes:
33+
When window closes or the reboot has finished:
3034
- [ ] Check the VM is up and you can SSH in
31-
- If not chase TPP via email for an update.
35+
- If TPP initatied, chase TPP via email for an update.
3236
- It is likely they have not finished maintenance yet, if so you may need to extend window and communicate to users.
37+
- If we initiated, we have limited options to debug failure, so we'll need to contact TPP to help diagnose.
3338

3439
When the VM is back up:
3540
- [ ] Is docker up? run `docker ps` on the VM
3641
- [ ] Is job-runner up? [check status](https://jobs.opensafely.org/status/), or `sudo systemctl status jobrunner`
37-
38-
If Level 4 was rebooted:
39-
- [ ] [manually start release hatch on the level 4 VM](https://bennettinstitute-team-manual.pages.dev/tech-group/playbooks/opensafely-tpp-level-4/#release-hatch)
40-
- Note: once we've got the releases UI this will not be necessary
41-
42+
- [ ] If there were jobs that got stopped as part of shutdown, check they are running again
4243
Finally
43-
- [ ] close window on freshstatus if still active.
44+
- [ ] notifiy users that maintenance is completed.

0 commit comments

Comments
 (0)