Displaying subprocess output to stdout and redirecting it

To save subprocess' stdout to a variable for further processing and to display it while the child process is running as it arrives:

#!/usr/bin/env python3from io import StringIOfrom subprocess import Popen, PIPEwith Popen('/path/to/script', stdout=PIPE, bufsize=1,           universal_newlines=True) as p, StringIO() as buf:    for line in p.stdout:        print(line, end='')        buf.write(line)    output = buf.getvalue()rc = p.returncode

To save both subprocess's stdout and stderr is more complex because you should consume both streams concurrently to avoid a deadlock:

stdout_buf, stderr_buf = StringIO(), StringIO()rc =  teed_call('/path/to/script', stdout=stdout_buf, stderr=stderr_buf,                universal_newlines=True)output = stdout_buf.getvalue()...

where teed_call() is define here.

Update: here's a simpler asyncio version.

^{Old version:}

Here's a single-threaded solution based on child_process.py example from tulip:

import asyncioimport sysfrom asyncio.subprocess import PIPE@asyncio.coroutinedef read_and_display(*cmd):    """Read cmd's stdout, stderr while displaying them as they arrive."""    # start process    process = yield from asyncio.create_subprocess_exec(*cmd,            stdout=PIPE, stderr=PIPE)    # read child's stdout/stderr concurrently    stdout, stderr = [], [] # stderr, stdout buffers    tasks = {        asyncio.Task(process.stdout.readline()): (            stdout, process.stdout, sys.stdout.buffer),        asyncio.Task(process.stderr.readline()): (            stderr, process.stderr, sys.stderr.buffer)}    while tasks:        done, pending = yield from asyncio.wait(tasks,                return_when=asyncio.FIRST_COMPLETED)        assert done        for future in done:            buf, stream, display = tasks.pop(future)            line = future.result()            if line: # not EOF                buf.append(line)    # save for later                display.write(line) # display in terminal                # schedule to read the next line                tasks[asyncio.Task(stream.readline())] = buf, stream, display    # wait for the process to exit    rc = yield from process.wait()    return rc, b''.join(stdout), b''.join(stderr)

The script runs '/path/to/script command and reads line by line both its stdout&stderr concurrently. The lines are printed to parent's stdout/stderr correspondingly and saved as bytestrings for future processing. To run the read_and_display() coroutine, we need an event loop:

import osif os.name == 'nt':    loop = asyncio.ProactorEventLoop() # for subprocess' pipes on Windows    asyncio.set_event_loop(loop)else:    loop = asyncio.get_event_loop()try:    rc, *output = loop.run_until_complete(read_and_display("/path/to/script"))    if rc:        sys.exit("child failed with '{}' exit code".format(rc))finally:    loop.close()

p.communicate() waits for the subprocess to complete and then returns its entire output at once.

Have you tried something like this instead, where you read the subprocess output line-by-line?

p = subprocess.Popen('/path/to/script', stdout=subprocess.PIPE, stderr=subprocess.PIPE)for line in p.stdout:  # do something with this individual line  print line

The Popen.communicate doc clearly states:

Note: The data read is buffered in memory, so do not use this method if the data size is large or unlimited.

https://docs.python.org/2/library/subprocess.html#subprocess.Popen.communicate

So if you need realtime output, you need to use something like this:

stream_p = subprocess.Popen('/path/to/script', stdout=subprocess.PIPE, stderr=subprocess.PIPE)while stream_line in stream_p:    #Parse it the way you want    print stream_line