A non-blocking read on a subprocess.PIPE in Python
fcntl, select, asyncproc won't help in this case.
A reliable way to read a stream without blocking regardless of operating system is to use Queue.get_nowait():
import sysfrom subprocess import PIPE, Popenfrom threading import Threadtry: from queue import Queue, Emptyexcept ImportError: from Queue import Queue, Empty # python 2.xON_POSIX = 'posix' in sys.builtin_module_namesdef enqueue_output(out, queue): for line in iter(out.readline, b''): queue.put(line) out.close()p = Popen(['myprogram.exe'], stdout=PIPE, bufsize=1, close_fds=ON_POSIX)q = Queue()t = Thread(target=enqueue_output, args=(p.stdout, q))t.daemon = True # thread dies with the programt.start()# ... do other things here# read line without blockingtry: line = q.get_nowait() # or q.get(timeout=.1)except Empty: print('no output yet')else: # got line # ... do something with line
I have often had a similar problem; Python programs I write frequently need to have the ability to execute some primary functionality while simultaneously accepting user input from the command line (stdin). Simply putting the user input handling functionality in another thread doesn't solve the problem because readline() blocks and has no timeout. If the primary functionality is complete and there is no longer any need to wait for further user input I typically want my program to exit, but it can't because readline() is still blocking in the other thread waiting for a line. A solution I have found to this problem is to make stdin a non-blocking file using the fcntl module:
import fcntlimport osimport sys# make stdin a non-blocking filefd = sys.stdin.fileno()fl = fcntl.fcntl(fd, fcntl.F_GETFL)fcntl.fcntl(fd, fcntl.F_SETFL, fl | os.O_NONBLOCK)# user input handling threadwhile mainThreadIsRunning: try: input = sys.stdin.readline() except: continue handleInput(input)In my opinion this is a bit cleaner than using the select or signal modules to solve this problem but then again it only works on UNIX...
Python 3.4 introduces new provisional API for asynchronous IO -- asyncio module.
The approach is similar to twisted-based answer by @Bryan Ward -- define a protocol and its methods are called as soon as data is ready:
#!/usr/bin/env python3import asyncioimport osclass SubprocessProtocol(asyncio.SubprocessProtocol): def pipe_data_received(self, fd, data): if fd == 1: # got stdout data (bytes) print(data) def connection_lost(self, exc): loop.stop() # end loop.run_forever()if os.name == 'nt': loop = asyncio.ProactorEventLoop() # for subprocess' pipes on Windows asyncio.set_event_loop(loop)else: loop = asyncio.get_event_loop()try: loop.run_until_complete(loop.subprocess_exec(SubprocessProtocol, "myprogram.exe", "arg1", "arg2")) loop.run_forever()finally: loop.close()There is a high-level interface asyncio.create_subprocess_exec() that returns Process objects that allows to read a line asynchroniosly using StreamReader.readline() coroutine (with async/await Python 3.5+ syntax):
#!/usr/bin/env python3.5import asyncioimport localeimport sysfrom asyncio.subprocess import PIPEfrom contextlib import closingasync def readline_and_kill(*args): # start child process process = await asyncio.create_subprocess_exec(*args, stdout=PIPE) # read line (sequence of bytes ending with b'\n') asynchronously async for line in process.stdout: print("got line:", line.decode(locale.getpreferredencoding(False))) break process.kill() return await process.wait() # wait for the child process to exitif sys.platform == "win32": loop = asyncio.ProactorEventLoop() asyncio.set_event_loop(loop)else: loop = asyncio.get_event_loop()with closing(loop): sys.exit(loop.run_until_complete(readline_and_kill( "myprogram.exe", "arg1", "arg2")))readline_and_kill() performs the following tasks:
- start subprocess, redirect its stdout to a pipe
- read a line from subprocess' stdout asynchronously
- kill subprocess
- wait for it to exit
Each step could be limited by timeout seconds if necessary.