You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Haven't had to use it for a long time, as TPP haven't requested a reboot
for a while.
Removed some obsolete instructions, and also updated to account for
a reboot we initiate rather than TPP.
Copy file name to clipboardExpand all lines: .github/ISSUE_TEMPLATE/tpp-reboot.md
+15-14Lines changed: 15 additions & 14 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -3,19 +3,22 @@ name: TPP reboot
3
3
about: Handling
4
4
title: "[TPP reboot]"
5
5
labels: ''
6
-
assignees: pipeline-team
7
6
8
7
---
9
8
10
-
Checklist for TPP reboot, to be created when TPP inform us of a maintenance window:
9
+
Checklist for TPP reboot
10
+
11
11
12
12
13
13
14
14
Preparation:
15
15
16
-
-[ ] bring window start forward 10min, and extend window close by 30min, as TPP can be a bit optimistic about how long they need
17
-
-[ ] schedule [maintenance window](https://support.freshstatus.io/en/support/solutions/articles/50000001851-how-to-create-a-planned-maintenance-incident-) in [freshstatus](https://bennettoxford.freshstatus.io/admin/incidents/public)
18
-
-[ ] communicate advance notice of window to users on #opensafely and #opensafely-users Slack channels
16
+
-[ ] If TPP have inform us of a maintenance window, bring window start
17
+
forward 10min, and extend window close by 30min, as TPP can be a bit
18
+
optimistic about how long they need. If we are rebooting for other reasons,
19
+
decide on window.
20
+
-[ ] communicate advance notice of window to users on #opensafely and
21
+
#opensafely-users Slack channels
19
22
-[ ] set Slack reminder for window start
20
23
21
24
When window starts:
@@ -24,20 +27,18 @@ When window starts:
24
27
25
28
See the [preparing for reboot](https://github.com/opensafely-core/backend-server/blob/main/playbook.md#preparing-for-reboot) section of the playbook for how to do this.
26
29
27
-
These commands can also be scheduled in advance if the maintenance window is out of hours, see the [planned maintenance](https://github.com/opensafely-core/backend-server/blob/main/playbook.md#planned-maintenance) section for more details.
30
+
Perform reboot
31
+
-[ ] if we are doing the reboot, then run `sudo reboot`. If TPP are stoping the VM, let them do it.
28
32
29
-
When window closes:
33
+
When window closes or the reboot has finished:
30
34
-[ ] Check the VM is up and you can SSH in
31
-
- If not chase TPP via email for an update.
35
+
- If TPP initatied, chase TPP via email for an update.
32
36
- It is likely they have not finished maintenance yet, if so you may need to extend window and communicate to users.
37
+
- If we initiated, we have limited options to debug failure, so we'll need to contact TPP to help diagnose.
33
38
34
39
When the VM is back up:
35
40
-[ ] Is docker up? run `docker ps` on the VM
36
41
-[ ] Is job-runner up? [check status](https://jobs.opensafely.org/status/), or `sudo systemctl status jobrunner`
37
-
38
-
If Level 4 was rebooted:
39
-
-[ ][manually start release hatch on the level 4 VM](https://bennettinstitute-team-manual.pages.dev/tech-group/playbooks/opensafely-tpp-level-4/#release-hatch)
40
-
- Note: once we've got the releases UI this will not be necessary
41
-
42
+
-[ ] If there were jobs that got stopped as part of shutdown, check they are running again
0 commit comments