Skip to content

crash on task events with null runtime value #377

@kgk

Description

@kgk

In exporter.py:293, the exporter calls self.celery_task_runtime.labels(**labels).observe(task.runtime) without a null check.

if task.runtime is not None:  
    self.celery_task_runtime.labels(**labels).observe(task.runtime)  

When a task event arrives with runtime=None, the prometheus_client library attempts self._value += None in values.py:20, raising:
TypeError: unsupported operand type(s) for +=: 'float' and 'NoneType'

In Celery, task.runtime is populated from the task-succeeded event. It can be None in these cases:

task-failed or task-retried events — runtime is only set on success; failure/retry events don't include it
task-revoked events — a revoked task never ran to completion, so no runtime is recorded
Worker crash mid-task — if the worker dies before emitting a success event, a synthetic event may be reconstructed without runtime
task-sent / task-received events — early lifecycle events never have a runtime since the task hasn't executed yet

The most likely trigger here is the exporter processing non-success events (failed, revoked, or retried tasks) that it shouldn't be calling observe() on, or it's listening to all event types and not filtering to task-succeeded before recording runtime.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions