Celery auto reload on ANY changes

Celery --autoreload doesn't work and it is deprecated.

Since you are using django, you can write a management command for that. Django has autoreload utility which is used by runserver to restart WSGI server when code changes.

The same functionality can be used to reload celery workers. Create a seperate management command called celery. Write a function to kill existing worker and start a new worker. Now hook this function to autoreload as follows.

import shleximport subprocessfrom django.core.management.base import BaseCommandfrom django.utils import autoreloaddef restart_celery():    cmd = 'pkill celery'    subprocess.call(shlex.split(cmd))    cmd = 'celery worker -l info -A foo'    subprocess.call(shlex.split(cmd))class Command(BaseCommand):    def handle(self, *args, **options):        print('Starting celery worker with autoreload...')        # For Django>=2.2        autoreload.run_with_reloader(restart_celery)         # For django<2.1        # autoreload.main(restart_celery)

Now you can run celery worker with python manage.py celery which will autoreload when codebase changes.

This is only for development purposes and do not use it in production. Code taken from my other answer here.

You can manually include additional modules with -I|--include. Combine this with GNU tools like find and awk and you'll be able to find all .py files and include them.

$ celery -A app worker --autoreload --include=$(find . -name "*.py" -type f | awk '{sub("\./",""); gsub("/", "."); sub(".py",""); print}' ORS=',' | sed 's/.$//')

Lets explain it:

find . -name "*.py" -type f

find searches recursively for all files containing .py. The output looks something like this:

./app.py./some_package/foopy./some_package/bar.py

Then:

awk '{sub("\./",""); gsub("/", "."); sub(".py",""); print}' ORS=','

This line takes output of find as input and removes all occurences of ./. Then it replaces all / with a .. The last sub() removes replaces .py with an empty string. ORS replaces all newlines with ,. This outputs:

app,some_package.foo,some_package.bar,

The last command, sed removes the last ,.

So the command that is being executed looks like:

$ celery -A app worker --autoreload --include=app,some_package.foo,some_package.bar

If you have a virtualenv inside your source you can exclude it by adding -path .path_to_your_env -prune -o:

$ celery -A app worker --autoreload --include=$(find . -path .path_to_your_env -prune -o -name "*.py" -type f | awk '{sub("\./",""); gsub("/", "."); sub(".py",""); print}' ORS=',' | sed 's/.$//')

You can use watchmedo

pip install watchdog

Start celery worker indirectly via watchmedo

watchmedo auto-restart --directory=./ --pattern=*.py --recursive -- celery worker --app=worker.app --concurrency=1 --loglevel=INFO

More detailed