semi-auto Noble upgrade fails on trying to start securedrop_rqworkser.service

[zenmonkeykstop]

Description

Noble migration fails to complete, due to an error on startup of securedrop services.

Steps to Reproduce

• install 2.11.1 on Focal
• upgrade to 2.12.0-rc2 on Focal and on Admin workstation
• run ./securedrop-admin --force noble_migration

Expected Behavior

Migration completes successfully, with app and mon upgraded to Noble and interfaces available

Actual Behavior

Migration times out with an "Timeout when waiting for search string "finished":"Done"" error,

On viewing logs, timeout is due to rq_worker.service failing to start repeatedly with following errors:

Mar 04 16:20:19 app-prod systemd[1]: securedrop_shredder.service: Consumed 1.197s CPU time.
Mar 04 16:20:25 app-prod systemd[1]: securedrop_rqworker.service: Scheduled restart job, restart counter is at 28.
Mar 04 16:20:25 app-prod systemd[1]: securedrop_rqrequeue.service: Scheduled restart job, restart counter is at 28.
Mar 04 16:20:25 app-prod systemd[1]: Starting securedrop_rqrequeue.service - SecureDrop rqrequeue process...
Mar 04 16:20:25 app-prod systemd[1]: Starting securedrop_rqworker.service - SecureDrop rq worker...
Mar 04 16:20:25 app-prod systemd[1]: Started securedrop_rqrequeue.service - SecureDrop rqrequeue process.
Mar 04 16:20:25 app-prod systemd[1]: Started securedrop_rqworker.service - SecureDrop rq worker.
Mar 04 16:20:25 app-prod python[2769]: Traceback (most recent call last):
Mar 04 16:20:25 app-prod python[2769]:   File "/var/www/securedrop/scripts/rqrequeue", line 12, in <module>
Mar 04 16:20:25 app-prod python[2769]:     from worker import requeue_interrupted_jobs
Mar 04 16:20:25 app-prod python[2769]:   File "/var/www/securedrop/worker.py", line 6, in <module>
Mar 04 16:20:25 app-prod python[2769]:     from rq.exceptions import InvalidJobOperation, NoSuchJobError
Mar 04 16:20:25 app-prod python[2769]:   File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/rq/__init__.py", line 8, in <module>
Mar 04 16:20:25 app-prod python[2769]:     from .job import cancel_job, get_current_job, requeue_job, Retry
Mar 04 16:20:25 app-prod python[2769]:   File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/rq/job.py", line 14, in <module>
Mar 04 16:20:25 app-prod python[2769]:     from distutils.version import StrictVersion
Mar 04 16:20:25 app-prod python[2769]: ModuleNotFoundError: No module named 'distutils'
Mar 04 16:20:25 app-prod rqworker[2770]: Traceback (most recent call last):
Mar 04 16:20:25 app-prod rqworker[2770]:   File "/opt/venvs/securedrop-app-code/bin/rqworker", line 5, in <module>
Mar 04 16:20:25 app-prod rqworker[2770]:     from rq.cli import worker
Mar 04 16:20:25 app-prod rqworker[2770]:   File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/rq/__init__.py", line 8, in <module>
Mar 04 16:20:25 app-prod rqworker[2770]:     from .job import cancel_job, get_current_job, requeue_job, Retry
Mar 04 16:20:25 app-prod rqworker[2770]:   File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/rq/job.py", line 14, in <module>
Mar 04 16:20:25 app-prod rqworker[2770]:     from distutils.version import StrictVersion
Mar 04 16:20:25 app-prod rqworker[2770]: ModuleNotFoundError: No module named 'distutils'
Mar 04 16:20:25 app-prod systemd[1]: securedrop_rqrequeue.service: Main process exited, code=exited, status=1/FAILURE
Mar 04 16:20:25 app-prod systemd[1]: securedrop_rqrequeue.service: Failed with result 'exit-code'.
Mar 04 16:20:25 app-prod systemd[1]: securedrop_rqworker.service: Main process exited, code=exited, status=1/FAILURE
Mar 04 16:20:25 app-prod systemd[1]: securedrop_rqworker.service: Failed with result 'exit-code'.

Starting securedrop virtualenv, can confirm distutils module is not available.

Please provide screenshots where appropriate.

Comments

Suggestions to fix, any other relevant information.

[zenmonkeykstop]

This looks to be an issue with test plan instructions - the apt-test repo needs to be explicitly specified for the upgrade script to use it, otherwise securedrop packages remain stuck on focal versions. Retesting, but likely not a release blocker.