-
Notifications
You must be signed in to change notification settings - Fork 122
Description
In exporter.py:293, the exporter calls self.celery_task_runtime.labels(**labels).observe(task.runtime) without a null check.
if task.runtime is not None:
self.celery_task_runtime.labels(**labels).observe(task.runtime)
When a task event arrives with runtime=None, the prometheus_client library attempts self._value += None in values.py:20, raising:
TypeError: unsupported operand type(s) for +=: 'float' and 'NoneType'
In Celery, task.runtime is populated from the task-succeeded event. It can be None in these cases:
task-failed or task-retried events — runtime is only set on success; failure/retry events don't include it
task-revoked events — a revoked task never ran to completion, so no runtime is recorded
Worker crash mid-task — if the worker dies before emitting a success event, a synthetic event may be reconstructed without runtime
task-sent / task-received events — early lifecycle events never have a runtime since the task hasn't executed yet
The most likely trigger here is the exporter processing non-success events (failed, revoked, or retried tasks) that it shouldn't be calling observe() on, or it's listening to all event types and not filtering to task-succeeded before recording runtime.