subprocess's Popen closes stdout/stderr filedescriptors used in another thread when Popen errors

I would like to answer your questions with:

1. Yes.
2. You shouldn't have to.
3. No.

The error occurs indeed in Python 2.7.4 as well.

I think this is a bug in the library code. If you add a lock in your program and make sure that the two calls to subprocess.Popen are executed atomically, the error does not occur.

@@ -1,32 +1,40 @@ import time import threading import subprocess+lock = threading.Lock()+ def subprocesscall():+    lock.acquire()     p = subprocess.Popen(         ['ls', '-l'],         stdin=subprocess.PIPE,         stdout=subprocess.PIPE,         stderr=subprocess.PIPE,         )+    lock.release()     time.sleep(2) # simulate the Popen call takes some time to complete.     out, err = p.communicate()     print 'succeeding command in thread:', threading.current_thread().ident def failingsubprocesscall():     try:+        lock.acquire()         p = subprocess.Popen(             ['thiscommandsurelydoesnotexist'],             stdin=subprocess.PIPE,             stdout=subprocess.PIPE,             stderr=subprocess.PIPE,             )     except Exception as e:         print 'failing command:', e, 'in thread:', threading.current_thread().ident+    finally:+        lock.release()+ print 'main thread is:', threading.current_thread().ident subprocesscall_thread = threading.Thread(target=subprocesscall) subprocesscall_thread.start() failingsubprocesscall() subprocesscall_thread.join()

This means that it is most probably due to some data race in the implementation of Popen. I will risk a guess: the bug may be in the implementation of pipe_cloexec, called by _get_handles, which (in 2.7.4) is:

def pipe_cloexec(self):    """Create a pipe with FDs set CLOEXEC."""    # Pipes' FDs are set CLOEXEC by default because we don't want them    # to be inherited by other subprocesses: the CLOEXEC flag is removed    # from the child's FDs by _dup2(), between fork() and exec().    # This is not atomic: we would need the pipe2() syscall for that.    r, w = os.pipe()    self._set_cloexec_flag(r)    self._set_cloexec_flag(w)    return r, w

and the comment warns explicitly about it not being atomic... This definitely causes a data race but, without experimentation, I don't know if it's what causes the problem.

Other solution, in the case that you are not handling the files that were opened (e.g when building an API).

I found a workaround to the problem by doing windll API calls, to mark all already opened file descriptors as "not inheritable". This is somewhat a hack, and the Q&A is available here:

Howto: workaround of close_fds=True and redirect stdout/stderr on windows

It will bypass the Python 2.7 bug.

Other solution would be to use Python 3.4+ :) It has been fixed