Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Celery support for asynchronous execution of Rulesets and Rules #6

Open
miraculixx opened this issue Nov 20, 2014 · 0 comments
Open
Labels

Comments

@miraculixx
Copy link
Owner

This story adds a celery task to asynchronously execute Rulesets.

Expected behaviour

  1. Assume celery does not execute with the same backend as the calling process. That is, calling the task must only accept serialized input and/or keys for e.g. rulesets
  2. Deployment of rulesets to the celery instance shall be done using pip installable eggs (remember a ruleset is the collection of rules that either point to a module are have a yaml or JSON input).
  3. There is a management command to deploy rulesets, using graceful restart of the celery instance. the command must take a ruleset name, build the egg, then deploy it.

Implementation notes

Ruleset execution

Ruleset execution is by means of calling the respective celery task. Can we have one named celery task for each ruleset? => can track each ruleset execution by celery's log, easy management with flower etc. If not, regress to a simple ruleset celery task that takes the ruleset's slug and the rule context, executes it, returns and stores its results.

Rule export

see #13

Rule deyployment

Two options for deployment.

  1. using a simple celery instance. gracefully shutdown the celery worker, pip install the new ruleset/version, restart the worker
  2. using docker. Have a docker image with python, django, celery. then each time we want to redeploy the tasks, gracefully shut down the worker, rebuild the docker image, shut down the docker image, start it again

priority

Since 2 is the same as 1 plus docker, do 1 first. We can do 2 in a later story.

restart worker instance

As for stopping and restarting celery instances, use flower

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant