Gunicorn, Django, Gevent: Spawned threads are blocking

One way to run a task in the background is to fork the parent process. Unlike with Gevent, it won't block -- you're running two completely separate processes. This is slower than starting another (very cheap) greenlet, but in this case it's a good trade off.

Your process splits into two parts, the parent and the child. In the parent, return a response to Gunicorn just like in normal code.

In the child, do your long-running processing. At the end, clean up by doing a specialized version of exit. Here's some code that does processing and sends emails:

    if os.fork():        return JsonResponse({}) # async parent: return HTTP 200    # child: do stuff, exit quietly    ret = do_tag_notify(        event, emails=emails, photo_names=photo_names,        )    logging.info('do_tag_notify/async result={0}'.format(ret))    os._exit(0)             # pylint: disable=W0212    logging.error("async child didn't _exit correctly") # never happens

Be careful with this. If there's an exception thrown in the child, even a syntax error or unused variable, you'll never know about it! The parent with its logging is already gone. Be verbose with logging, and don't do too much.

Using fork is a useful tool -- have fun!

It would appear no one here gave an actual to your question.

Is the blocking intended or is our setup wrong?

There is something wrong with your setup. SQL queries are almost entirely I/O bound and should not be blocking any greenlets. You are either using a SQL/ORM library that is not gevent-friendly, or something else in your code is causing the blocking. You should not need to use multiprocessing for this kind of task.

Unless you are explicitly doing a join on the greenlets, then the server response should not be blocking.

I have settled for using a synchronous (standard) worker and making use of the multiprocessing library. This seems to be the easiest solution for now.

I have also implemented a global pool abusing a memcached cache providing locks so only two tasks can run.