Celery: how to limit number of tasks in queue and stop feeding when full?
You can set rabbitmq x-max-length in queue predeclare using kombu
example :
import timefrom celery import Celeryfrom kombu import Queue, Exchangeclass Config(object): BROKER_URL = "amqp://guest@localhost//" CELERY_QUEUES = ( Queue( 'important', exchange=Exchange('important'), routing_key="important", queue_arguments={'x-max-length': 10} ), )app = Celery('tasks')app.config_from_object(Config)@app.task(queue='important')def process_data(x): passor using Policies
rabbitmqctl set_policy Ten "^one-meg$" '{"max-length-bytes":1000000}' --apply-to queues