-
Notifications
You must be signed in to change notification settings - Fork 37
mac maintenance
These are the manual steps to maintain the macs. Until we fully automate setup and maintenance, we need to ssh into each mac to do maintenance.
The list of machines is here. Please keep the statuses updated on that page.
For each of these, consider quarantining the machine before performing maintenance.
On puppetized fx and tb prod boxes, scriptworker is running out of /builds/scriptworker and the poller is running out of /builds/scriptworker/poller. They share a single virtualenv in /builds/scriptworker/virtualenv.
Scriptworker-scripts clones are in /builds/scriptworker/virtualenv/src/*, for iscript, mozbuild, notarization-poller, and scriptworker-client.
Config files are at /builds/scriptworker/script_config.yaml, /builds/scriptworker/scriptworker.yaml, and /builds/scriptworker/poller/poller.yaml.
Certs are in /builds/scriptworker/certs.
# scriptworker + iscript/poller python dependencies
sudo -u cltbld -i
cd /builds/scriptworker
. virtualenv/bin/activate
pip install ...
# scriptworker-scripts python modules
sudo -u cltbld -i
cd /builds/scriptworker/virtualenv/src/iscript
git fetch -a origin
git checkout REVISION_OR_BRANCH
# Repeat for ../mozbuild, ../notarization-poller, ../scriptworker-client
# restart scriptworker / iscript
sudo -u root -i
launchctl stop org.mozilla.scriptworker && launchctl start org.mozilla.scriptworker
# restart notarization_poller
sudo -u root -i
launchctl stop org.mozilla.notarization_poller && launchctl start org.mozilla.notarization_poller
As of this writing 2020.09.16, this is only the TB boxes, and only because :dhouse is poking at tb2.
Scriptworker is installed in /builds/scriptworker. Only some non-puppetized machines have pollers, but those are in /builds/scriptworker/poller. The virtualenv is in /builds/scriptworker.
The scriptworker-scripts clone is in /builds/scriptworker/scriptworker-scripts.
Config files are at /builds/scriptworker/script_config.yaml, /builds/scriptworker/scriptworker.yaml, and /builds/scriptworker/poller/poller.yaml if the poller is installed.
Certs are in /builds/scriptworker/ and /home/cltbld.
# scriptworker + iscript/poller python dependencies
sudo -u cltbld -i
cd /builds/scriptworker
. bin/activate
pip install ...
# scriptworker-scripts python modules
sudo -u cltbld -i
cd /builds/scriptworker/scriptworker-scripts
git fetch -a origin
git checkout REVISION_OR_BRANCH
# Or just `git pull` to get the latest, if we're currently on origin/master
# restart scriptworker / iscript
# look for SCREEN in processes.
ps -ef | grep -i screen
# Kill that process if it doesn't belong to you.
sudo kill PID
# start scriptworker in screen.
screen
sudo -u cltbld -i
cd /builds/scriptworker
bin/scriptworker script_config.yaml
# restart notarization_poller
screen
sudo -u poller -i
cd /builds/scriptworker/poller
../bin/notarization_poller poller.yaml
As of this writing 2020.09.16, puppet is broken on dep boxes, so both dep boxes are manually set up and configured.
Scriptworker is running out of /builds/dep1 (as depbld1), /builds/dep2 (as depbld2), and /builds/tb-dep (as tbbld).
Scriptworker-scripts clones are in scriptworker-scripts in each of the above working directories.
Config files are in script_config.yaml and scriptworker.yaml in each of the above working directories.
There is no notarization for dep, so there are no notarization pollers.
# scriptworker + iscript python dependencies
sudo -u depbld1 -i
cd /builds/dep1
. bin/activate
pip install ...
# Repeat for depbld2 + /builds/dep2, and tbbld + /builds/tb-dep
# scriptworker-scripts python modules
sudo -u depbld1 -i
cd /builds/dep1/scriptworker-scripts
git fetch -a origin
git checkout REVISION_OR_BRANCH
# Or just `git pull` to get the latest, if we're currently on origin/master
# Repeat for depbld2 + /builds/dep2/scriptworker-scripts
# and tbbld + /builds/tb-dep/scriptworker-scripts
# restart scriptworker / iscript
# look for SCREEN in processes.
ps -ef | grep -i screen
# Kill that process if it doesn't belong to you.
sudo kill PID
# start scriptworker in screen.
screen
sudo -u depbld1 -i
cd /builds/dep1
bin/scriptworker scriptworker.yaml
# Repeat for depbld2 + /builds/dep2, and tbbld + /builds/tb-dep
Wipe secrets if someone may gain access to the box: when moving the machine or retiring the machine.
When wiping secrets, you want to nuke ed25519_privkey (prod only), *.keychain, and widevine*.crt. Use rm -P to overwrite them multiple times.