Using Celery on processes and gevent in tasks at the same time

I believe the recommended way to start the task is as follows.

python manage.py celery worker -P gevent --loglevel=INFO

Gevent needs to be patched as early as possible.

You can run celery with multiple threads containing multiple greenlets like this:

$ celery multi start 4 -P gevent -l info -c:1-4 1000

From my weird experience, Celery Beat can't work properly with workers with gevent pool (scheduled tasks are blocked and wait forever), unless you activate gevent monkey patching for Beat process.

However, celery beat doesn't support --pool=gevent or -P gevent option. The proper way to inject gevent monkey patching is to use a curstomized celery binary, such as:

#!/usr/bin/env python# -*- coding: utf-8 -*-from gevent import monkeymonkey.patch_all()import reimport sysfrom celery.__main__ import mainif __name__ == '__main__':    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])    sys.exit(main())

Save it as celery-gevent, and run Beat service as follows:

celery-gevent beat --app=proj.celery:app --loader=djcelery.loaders.DjangoLoader -f /var/log/celery/beat.log -l INFO --workdir=/my/proj --pidfile=/var/run/celery/beat.pid

In proj.celery you should also patch Django connection to avoid DatabaseError:

from __future__ import absolute_importimport os# Set the Django settings module for the 'celery' programos.environ.setdefault('DJANGO_SETTINGS_MODULE', 'proj.settings')import django# Load Django model definitions, etcdjango.setup()from django.db import connection# Allow thread sharing to ensure that Django database connection# works properly with gevent.connection.allow_thread_sharing = Truefrom django.conf import settingsfrom celery import Celeryapp = Celery('proj')# Using a string here means the worker will not have to# pickle the object when using Windows.app.config_from_object('django.conf:settings')app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)

(The above example works for Python 2.7.10, Celery 3.1.18, Django 1.8.2 and gevent 1.0.2)