The workers in ThreadPoolExecutor is not really daemon
Suddenly... I found why. According to much more source code of ThreadPoolExecutor
:
# Workers are created as daemon threads. This is done to allow the interpreter# to exit when there are still idle threads in a ThreadPoolExecutor's thread# pool (i.e. shutdown() was not called). However, allowing workers to die with# the interpreter has two undesirable properties:# - The workers would still be running during interpreter shutdown,# meaning that they would fail in unpredictable ways.# - The workers could be killed while evaluating a work item, which could# be bad if the callable being evaluated has external side-effects e.g.# writing to a file.## To work around this problem, an exit handler is installed which tells the# workers to exit when their work queues are empty and then waits until the# threads finish._threads_queues = weakref.WeakKeyDictionary()_shutdown = Falsedef _python_exit(): global _shutdown _shutdown = True items = list(_threads_queues.items()) for t, q in items: q.put(None) for t, q in items: t.join()atexit.register(_python_exit)
There is an exit handler which will join all unfinished worker...
Here's the way to avoid this problem. Bad design can be beaten by another bad design. People write daemon=True
only if they really know that the worker won't damage any objects or files.
In my case, I created TreadPoolExecutor
with a single worker and after a single submit
I just deleted the newly created thread from the queue so the interpreter won't wait till this thread stops on its own. Notice that worker threads are created after submit
, not after the initialization of TreadPoolExecutor
.
import concurrent.futures.threadfrom concurrent.futures import ThreadPoolExecutor...executor = ThreadPoolExecutor(max_workers=1)future = executor.submit(lambda: self._exec_file(args))del concurrent.futures.thread._threads_queues[list(executor._threads)[0]]
It works in Python 3.8 but may not work in 3.9+ since this code is accessing private variables.
See the working piece of code on github