catching stdout in realtime from subprocess

Some rules of thumb for subprocess.

• Never use shell=True. It needlessly invokes an extra shell process to call your program.
• When calling processes, arguments are passed around as lists. sys.argv in python is a list, and so is argv in C. So you pass a list to Popen to call subprocesses, not a string.
• Don't redirect stderr to a PIPE when you're not reading it.
• Don't redirect stdin when you're not writing to it.

Example:

import subprocess, time, os, syscmd = ["rsync.exe", "-vaz", "-P", "source/" ,"dest/"]p = subprocess.Popen(cmd,                     stdout=subprocess.PIPE,                     stderr=subprocess.STDOUT)for line in iter(p.stdout.readline, b''):    print(">>> " + line.rstrip())

That said, it is probable that rsync buffers its output when it detects that it is connected to a pipe instead of a terminal. This is the default behavior - when connected to a pipe, programs must explicitly flush stdout for realtime results, otherwise standard C library will buffer.

To test for that, try running this instead:

cmd = [sys.executable, 'test_out.py']

and create a test_out.py file with the contents:

import sysimport timeprint ("Hello")sys.stdout.flush()time.sleep(10)print ("World")

Executing that subprocess should give you "Hello" and wait 10 seconds before giving "World". If that happens with the python code above and not with rsync, that means rsync itself is buffering output, so you are out of luck.

A solution would be to connect direct to a pty, using something like pexpect.

I know this is an old topic, but there is a solution now. Call the rsync with option --outbuf=L. Example:

cmd=['rsync', '-arzv','--backup','--outbuf=L','source/','dest']p = subprocess.Popen(cmd,                     stdout=subprocess.PIPE)for line in iter(p.stdout.readline, b''):    print '>>> {}'.format(line.rstrip())

Depending on the use case, you might also want to disable the buffering in the subprocess itself.

If the subprocess will be a Python process, you could do this before the call:

os.environ["PYTHONUNBUFFERED"] = "1"

Or alternatively pass this in the env argument to Popen.

Otherwise, if you are on Linux/Unix, you can use the stdbuf tool. E.g. like:

cmd = ["stdbuf", "-oL"] + cmd

See also here about stdbuf or other options.