PyInstaller-built Windows EXE fails with multiprocessing
To add on to nikola's answer...
*nix (Linux, Mac OS X, etc.) does NOT require any changes for PyInstaller to work. (This includes both --onedir and --onefile options.) If you only intend to support *nix systems, no need to worry about any of this.
However, if you are planning on supporting Windows, you will need to add some code, depending on which option you pick: --onedir or --onefile.
If you plan to use --onedir, all you will need to add is a special method call:
if __name__ == '__main__': # On Windows calling this function is necessary. multiprocessing.freeze_support()According to the documentation, this call must be made immediately after if __name__ == '__main__':, or else it will not work. (It is strongly suggested that you have these two lines in your main module.)
In reality, however, you can afford to do a check before the call, and things will still work:
if __name__ == '__main__': if sys.platform.startswith('win'): # On Windows calling this function is necessary. multiprocessing.freeze_support()However, calling multiprocessing.freeze_support() is possible on other platforms and situations as well - running it only impacts freezing support on Windows. If you're a bytecode nut, you'll notice that the if statement adds some bytecode, and makes potential savings from using an if statement negligible. Therefore, you should just stick to a simple multiprocessing.freeze_support() call immediately after if __name__ == '__main__':.
If you plan to use --onefile, you will need to add nikola's code:
import multiprocessing.forkingimport osimport sysclass _Popen(multiprocessing.forking.Popen): def __init__(self, *args, **kw): if hasattr(sys, 'frozen'): # We have to set original _MEIPASS2 value from sys._MEIPASS # to get --onefile mode working. os.putenv('_MEIPASS2', sys._MEIPASS) try: super(_Popen, self).__init__(*args, **kw) finally: if hasattr(sys, 'frozen'): # On some platforms (e.g. AIX) 'os.unsetenv()' is not # available. In those cases we cannot delete the variable # but only set it to the empty string. The bootloader # can handle this case. if hasattr(os, 'unsetenv'): os.unsetenv('_MEIPASS2') else: os.putenv('_MEIPASS2', '')class Process(multiprocessing.Process): _Popen = _Popen# ...if __name__ == '__main__': # On Windows calling this function is necessary. multiprocessing.freeze_support() # Use your new Process class instead of multiprocessing.ProcessYou can combine the above with the rest of his code, or the following:
class SendeventProcess(Process): def __init__(self, resultQueue): self.resultQueue = resultQueue multiprocessing.Process.__init__(self) self.start() def run(self): print 'SendeventProcess' self.resultQueue.put((1, 2)) print 'SendeventProcess'if __name__ == '__main__': # On Windows calling this function is necessary. multiprocessing.freeze_support() print 'main' resultQueue = multiprocessing.Queue() SendeventProcess(resultQueue) print 'main'I got the code from here, PyInstaller's new site for the multiprocessing recipe. (They seem to have shut down their Trac based site.)
Note that they have a minor error with their code for --onefile multiprocessing support. They add os.sep to their _MEIPASS2 environment variable. (Line: os.putenv('_MEIPASS2', sys._MEIPASS + os.sep)) This breaks things:
File "<string>", line 1 sys.path.append(r"C:\Users\Albert\AppData\Local\Temp\_MEI14122\") ^SyntaxError: EOL while scanning string literal
The code I provided above is the same, without the os.sep. Removing the os.sep fixes this issue and allows multiprocessing to work using the --onefile configuration.
In summary:
Enabling --onedir multiprocessing support on Windows (does NOT work with --onefile on Windows, but otherwise safe on all platforms/configurations):
if __name__ == '__main__': # On Windows calling this function is necessary. multiprocessing.freeze_support()Enabling --onefile multiprocessing support on Windows (safe on all platforms/configurations, compatible with --onedir):
import multiprocessing.forkingimport osimport sysclass _Popen(multiprocessing.forking.Popen): def __init__(self, *args, **kw): if hasattr(sys, 'frozen'): # We have to set original _MEIPASS2 value from sys._MEIPASS # to get --onefile mode working. os.putenv('_MEIPASS2', sys._MEIPASS) try: super(_Popen, self).__init__(*args, **kw) finally: if hasattr(sys, 'frozen'): # On some platforms (e.g. AIX) 'os.unsetenv()' is not # available. In those cases we cannot delete the variable # but only set it to the empty string. The bootloader # can handle this case. if hasattr(os, 'unsetenv'): os.unsetenv('_MEIPASS2') else: os.putenv('_MEIPASS2', '')class Process(multiprocessing.Process): _Popen = _Popen# ...if __name__ == '__main__': # On Windows calling this function is necessary. multiprocessing.freeze_support() # Use your new Process class instead of multiprocessing.Process
Answering my own questions after finding this PyInstaller ticket:
Apparently all we have to do is provide a Process (and _Popen) class as shown below, and use it instead of multiprocessing.Process. I've corrected and simplified the class to work on Windows only, *ix systems might need different code.
For the sake of completeness, here's the adapted sample from the above question:
import multiprocessingfrom Queue import Emptyclass _Popen(multiprocessing.forking.Popen): def __init__(self, *args, **kw): if hasattr(sys, 'frozen'): os.putenv('_MEIPASS2', sys._MEIPASS) try: super(_Popen, self).__init__(*args, **kw) finally: if hasattr(sys, 'frozen'): os.unsetenv('_MEIPASS2')class Process(multiprocessing.Process): _Popen = _Popendef _start(): while True: try: command = queue.get_nowait() # ... and some more code to actually interpret commands except Empty: time.sleep(0.015)def start(): process = Process(target=_start, args=args) process.start() return process