Python: execute cat subprocess in parallel

You don't need neither multiprocessing nor threading to run subprocesses in parallel e.g.:

#!/usr/bin/env pythonfrom subprocess import Popen# run commands in parallelprocesses = [Popen("echo {i:d}; sleep 2; echo {i:d}".format(i=i), shell=True)             for i in range(5)]# collect statusesexitcodes = [p.wait() for p in processes]

it runs 5 shell commands simultaneously. Note: neither threads nor multiprocessing module are used here. There is no point to add ampersand & to the shell commands: Popen doesn't wait for the command to complete. You need to call .wait() explicitly.

It is convenient but it is not necessary to use threads to collect output from subprocesses:

#!/usr/bin/env pythonfrom multiprocessing.dummy import Pool # thread poolfrom subprocess import Popen, PIPE, STDOUT# run commands in parallelprocesses = [Popen("echo {i:d}; sleep 2; echo {i:d}".format(i=i), shell=True,                   stdin=PIPE, stdout=PIPE, stderr=STDOUT, close_fds=True)             for i in range(5)]# collect output in paralleldef get_lines(process):    return process.communicate()[0].splitlines()outputs = Pool(len(processes)).map(get_lines, processes)

Related: Python threading multiple bash subprocesses?.

Here's code example that gets output from several subprocesses concurrently in the same thread:

#!/usr/bin/env python3import asyncioimport sysfrom asyncio.subprocess import PIPE, STDOUT@asyncio.coroutinedef get_lines(shell_command):    p = yield from asyncio.create_subprocess_shell(shell_command,            stdin=PIPE, stdout=PIPE, stderr=STDOUT)    return (yield from p.communicate())[0].splitlines()if sys.platform.startswith('win'):    loop = asyncio.ProactorEventLoop() # for subprocess' pipes on Windows    asyncio.set_event_loop(loop)else:    loop = asyncio.get_event_loop()# get commands output in parallelcoros = [get_lines('"{e}" -c "print({i:d}); import time; time.sleep({i:d})"'                    .format(i=i, e=sys.executable)) for i in range(5)]print(loop.run_until_complete(asyncio.gather(*coros)))loop.close()

Another approach (rather than the other suggestion of putting shell processes in the background) is to use multithreading.

The run method that you have would then do something like this:

thread.start_new_thread ( myFuncThatDoesZGrep)

To collect results, you can do something like this:

class MyThread(threading.Thread):   def run(self):       self.finished = False       # Your code to run the command here.       blahBlah()       # When finished....       self.finished = True       self.results = []

Run the thread as stated above in the link on multithreading. When your thread object has myThread.finished == True, then you can collect the results via myThread.results.