read subprocess stdout line by line

It's been a long time since I last worked with Python, but I think the problem is with the statement for line in proc.stdout, which reads the entire input before iterating over it. The solution is to use readline() instead:

#filters outputimport subprocessproc = subprocess.Popen(['python','fake_utility.py'],stdout=subprocess.PIPE)while True:  line = proc.stdout.readline()  if not line:    break  #the real code does filtering here  print "test:", line.rstrip()

Of course you still have to deal with the subprocess' buffering.

Note: according to the documentation the solution with an iterator should be equivalent to using readline(), except for the read-ahead buffer, but (or exactly because of this) the proposed change did produce different results for me (Python 2.5 on Windows XP).

Bit late to the party, but was surprised not to see what I think is the simplest solution here:

import ioimport subprocessproc = subprocess.Popen(["prog", "arg"], stdout=subprocess.PIPE)for line in io.TextIOWrapper(proc.stdout, encoding="utf-8"):  # or another encoding    # do something with line

(This requires Python 3.)

Indeed, if you sorted out the iterator then buffering could now be your problem. You could tell the python in the sub-process not to buffer its output.

proc = subprocess.Popen(['python','fake_utility.py'],stdout=subprocess.PIPE)

becomes

proc = subprocess.Popen(['python','-u', 'fake_utility.py'],stdout=subprocess.PIPE)

I have needed this when calling python from within python.