Nesting concurrent.futures.ThreadPoolExecutor Nesting concurrent.futures.ThreadPoolExecutor python python

Nesting concurrent.futures.ThreadPoolExecutor


There is absolutely no issue with spawning threads from other threads. Your case is no different.

Sooner or later though, the overhead of spawning threads will be quite high, and spawning more threads will actually cause your software to slow down.

I highly suggest using a library like asyncio which beautifully handles tasks asynchronously. It does so by using one thread with non-blocking io. The results will probably be even faster than with normal threads, as the overhead is much less significant.

If you do not wish to use asyncio, why not create another pool executor inside main, and pass it on to the outer() function? This way, instead of 25 (5x5) threads, you will have a maximum of 10 (2x5) which is much more reasonable?

You cannot pass the same main() executor which calls outer() to outer() as it might cause a deadlock (by each outer() waiting for another outer() to finish before they can schedule inner()).