Need help understanding Comet in Python (with Django)

You could use Socket.IO. There are gevent and tornado handlers for it. See my blog post on gevent-socketio with Django here: http://codysoyland.com/2011/feb/6/evented-django-part-one-socketio-and-gevent/

I feel your pain, having had to go through the same research over the past few months. I haven't had time to deal with proper documentation yet but I have a working example of using Django with socket.io and tornadio at http://bitbucket.org/virtualcommons/vcweb - I was hoping to set up direct communication from the Django server-side to the tornadio server process using queues (i.e., logic in a django view pushes a message onto a queue that then gets handled by tornadio which pushes a json encoded version of that message out to all interested subscribers) but haven't implemented that part fully yet. The way I've currently gotten it set up involves:

1. An external tornado (tornadio) server, running on another port, accepting socket.io requests and working with Django models. The only writes this server process makes to the database are the chat messages that need to be stored. It has full access to all Django models, etc., and all real-time interactions need to go directly through this server process.
2. Django template pages that require real-time access include the socket.io javascript and establish direct connections to the tornadio server

I looked into orbited, hookbox, and gevent but decided to go with socket.io + tornado as it seemed to allow me the cleanest javascript + python code. I could be wrong about that though, having just started to learn Python/Django over the past year.

Redis is relevant as a persistence layer that also supports native publish/subscribe. So instead of a situation where you are polling the db looking for new messages, you can subscribe to a channel, and have messages pushed out to you.

I found a working example of the type of system you describe. The magic happens in the socketio view:

def socketio(request):    """The socket.io view."""    io = request.environ['socketio']    redis_sub = redis_client().pubsub()    user = username(request.user)    # Subscribe to incoming pubsub messages from redis.    def subscriber(io):        redis_sub.subscribe(room_channel())        redis_client().publish(room_channel(), user + ' connected.')        while io.connected():            for message in redis_sub.listen():                if message['type'] == 'message':                    io.send(message['data'])    greenlet = Greenlet.spawn(subscriber, io)    # Listen to incoming messages from client.    while io.connected():        message = io.recv()        if message:            redis_client().publish(room_channel(), user + ': ' + message[0])    # Disconnected. Publish disconnect message and kill subscriber greenlet.    redis_client().publish(room_channel(), user + ' disconnected')    greenlet.throw(Greenlet.GreenletExit)    return HttpResponse()

Take the view step-by-step:

1. Set up socket.io, get a redis client and the current user
2. Use Gevent to register a "subscriber" - this takes incoming messages from Redis and forwards them on to the client browser.
3. Run a "publisher" which takes messages from socket.io (from the user's browser) and pushes them into Redis
4. Repeat until the socket disconnects

The Redis Cookbook gives a little more detail on the Redis side, as well as discussing how you can persist messages.

Regarding the rest of your question: Twisted is an event-based networking library, it could be considered an alternative to Gevent in this application. It's powerful and difficult to debug in my experience.

Celery is a "distributed task queue" - basically, it lets you spread units of work out across multiple machines. The "distributed" angle means some sort of transport is required between the machines. Celery supports several types of transport, including RabbitMQ (and Redis too).

In the context of your example, Celery would only be appropriate if you had to do some sort of costly processing on each message like scanning for profanity or something. Even still, something would have to initiate the Celery task, so there would need to be some code listening for the socket.io callback.

(Just in case you weren't totally confused, Celery itself can be made to use Gevent as its underlying concurrency library.)

Hope that helps!