Choosing and deploying a comet server

I would recommend looking into Twisted, their twisted.web server, and the comet work done on top of it at Divmod. They can handle far more concurrent connections than traditional thread or process based servers, which is exactly what you need for something like this. And, yes, I've architected systems using Twisted for COMET stuff, while using other things for the more front-facing web applications beside it. It works out well with each part doing what it does best.

I need to do something very similar. I found this, but haven't had the time to look at it properly yet:

django_evserver is simple http server for Django applications. It's based on libevent library. The main advantage of django_evserver is that it provides methods of preempting django views. You can render a site in several chunks, each of them can be handled by different django view.

Using this idea it's possible to use django_evserver as comet server for django applications.

http://code.google.com/p/django-evserver/

One option is Netty, client-server socket framework based on Java NIO from JBoss. For a comparison and discussion see here. It reportedly handles 100000 simultaneous open connections on a quad-core server.