Skip to content

Cannot be used from within Celery #17

@danielcohenlive

Description

@danielcohenlive

See the stacktrace below

11:40:18 celery_general.1 | [2018-10-18 15:40:18,210: ERROR/ForkPoolWorker-2] Task xxx.apps.marketo.tasks.update_marketingdata_lead.update_marketingdata_lead[9f1605d0-bfe2-4549-8cf2-91abcdc8fd7b] raised unexpected: AssertionError('daemonic processes are not allowed to have children',)
11:40:18 celery_general.1 | Traceback (most recent call last):
11:40:18 celery_general.1 |   File "venv/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task
11:40:18 celery_general.1 |     R = retval = fun(*args, **kwargs)
11:40:18 celery_general.1 |   File "venv/lib/python2.7/site-packages/celery/app/trace.py", line 622, in __protected_call__
11:40:18 celery_general.1 |     return self.run(*args, **kwargs)
11:40:18 celery_general.1 |   File "xxxx/apps/marketo/tasks/update_marketingdata_lead.py", line 81, in update_marketingdata_lead
11:40:18 celery_general.1 |     _enqueue_lead_data(email, to_update, user_id)
11:40:18 celery_general.1 |   File "xxxx/apps/marketo/tasks/update_marketingdata_lead.py", line 93, in _enqueue_lead_data
11:40:18 celery_general.1 |     get_lead_producer().put(json.dumps(lead_data))
11:40:18 celery_general.1 |   File "xxxx/apps/xxxx_kinesis/__init__.py", line 39, in get_lead_producer
11:40:18 celery_general.1 |     buffer_time=5 * 60,
11:40:18 celery_general.1 |   File "venv/lib/python2.7/site-packages/kinesis/producer.py", line 142, in __init__
11:40:18 celery_general.1 |     max_size=max_size, boto3_session=boto3_session)
11:40:18 celery_general.1 |   File "venv/lib/python2.7/site-packages/kinesis/producer.py", line 78, in __init__
11:40:18 celery_general.1 |     self.start()
11:40:18 celery_general.1 |   File "venv/lib/python2.7/site-packages/offspring/process.py", line 55, in start
11:40:18 celery_general.1 |     self.process.start()
11:40:18 celery_general.1 |   File "/usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 124, in start
11:40:18 celery_general.1 |     'daemonic processes are not allowed to have children'
11:40:18 celery_general.1 | AssertionError: daemonic processes are not allowed to have children

My understanding is that using billiard instead of multiprocess will solve this issue

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions