Migrate celery settings to new ones

[stsewd]

Celery 6.x will drop support for some deprecated settings, and it will require to explicitly set CELERY as the prefix.

https://docs.celeryq.dev/en/stable/userguide/configuration.html#new-lowercase-settings

They do provide a tool for renaming the settings, but we should also check for settings set from the ops repos.

https://docs.celeryq.dev/en/stable/history/whatsnew-4.0.html#latentcall-django-admonition

[humitos]

We are already using CELERY_ prefix in all our settings, so it seems we only need to update how we load them at https://github.com/readthedocs/readthedocs.org/blob/main/readthedocs/worker.py#L18

diff --git a/readthedocs/worker.py b/readthedocs/worker.py
index 527ee509b..8a2de7dac 100644
--- a/readthedocs/worker.py
+++ b/readthedocs/worker.py
@@ -15,7 +15,7 @@ def create_application():
     )
 
     application = Celery(settings.CELERY_APP_NAME)
-    application.config_from_object("django.conf:settings")
+    application.config_from_object("django.conf:settings", namespace='CELERY')
     application.autodiscover_tasks(None)
 
     # A step to initialize django-structlog

We need to do the same on commercial.

I put this in Q4, but I'm not sure when Celery 6.x is going to be released. Maybe we can already make this change, tho.

[stsewd]

We are using the CELERYD prefix for some settings