Multithreading within a Celery Worker Multithreading within a Celery Worker multithreading multithreading

Multithreading within a Celery Worker


Using the eventlet library, you can patch the standard libraries for making them asynchronous.

First import the async urllib2:

from eventlet.green import urllib2

So you will get the url body with:

def fetch(url):    body = urllib2.urlopen(url).read()    return body

See more eventlet examples here.


I would create two tasks, one for downloading the data and the other for processing it once it is downloaded. This way you could scale the two tasks independently. See: Routing, Chains.