Python Celery versus Threading Library for running async requests [closed] Python Celery versus Threading Library for running async requests [closed] python python

Python Celery versus Threading Library for running async requests [closed]


The Python interpreter is resolutely single threaded due to the (in)famous global interpreter lock aka (GIL). Thus threading in Python only provides parallelism when computation and IO can happen simultaneously. Compute bound tasks will see little benefit from threading in the Python threading model, at least under CPython 2 or 3.

On the other hand, those limitations don't apply to multiprocessing, which is what you'll be doing with a queuing system like celery. You can run several worker instances of Python which can execute simultaneously on multicore machines, or multiple machines for that matter.

If I understand your scenario - an interaction on a web site kicks off a long-running job - queuing is almost certainly the way to go. You get true parallelism and the easy option to move processing to other machines.