Python threads and queue example

Threads do not exit normally in this code (they are indeed blocked when the queue is empty). The program doesn't wait for them because they're daemon threads.

The program doesn't exit immediately and doesn't block forever because of q.join and q.task_done calls.

The count of unfinished tasks goes up whenever an item is added to the queue. The count goes down whenever a consumer thread calls task_done() to indicate that the item was retrieved and all work on it is complete. When the count of unfinished tasks drops to zero, join() unblocks, and the program exists without waiting for daemon threads.

I have had the same problem. When all threads completed, I saw "sleeping threads" in process list (use top -H -p <pid> where <pid> is process id from ps aux | grep python with your script).

I solved this problem by replacing "infinite loop" while True to while not q.empty():.

It fixed the problem with "sleeping threads".

def worker():    while not q.empty():        item = q.get()        do_work(item)        q.task_done()q = Queue()for i in range(num_worker_threads):     t = Thread(target=worker)     t.daemon = True     t.start()for item in source():    q.put(item)q.join()       # block until all tasks are done