Parfor for Python Parfor for Python python python

Parfor for Python


The one built-in to python would be multiprocessing docs are here. I always use multiprocessing.Pool with as many workers as processors. Then whenever I need to do a for-loop like structure I use Pool.imap

As long as the body of your function does not depend on any previous iteration then you should have near linear speed-up. This also requires that your inputs and outputs are pickle-able but this is pretty easy to ensure for standard types.

UPDATE:Some code for your updated function just to show how easy it is:

from multiprocessing import Poolfrom itertools import productoutput = np.zeros((N,N))pool = Pool() #defaults to number of available CPU'schunksize = 20 #this may take some guessing ... take a look at the docs to decidefor ind, res in enumerate(pool.imap(Fun, product(xrange(N), xrange(N))), chunksize):    output.flat[ind] = res


There are many Python frameworks for parallel computing. The one I happen to like most is IPython, but I don't know too much about any of the others. In IPython, one analogue to parfor would be client.MultiEngineClient.map() or some of the other constructs in the documentation on quick and easy parallelism.


Jupyter Notebook

To see an example consider you want to write the equivalence of this Matlab code on in Python

matlabpool open 4parfor n=0:9   for i=1:10000       for j=1:10000           s=j*i          end   end   nenddisp('done')

The way one may write this in python particularly in jupyter notebook. You have to create a function in the working directory (I called it FunForParFor.py) which has the following

def func(n):    for i in range(10000):        for j in range(10000):            s=j*i    print(n)

Then I go to my Jupyter notebook and write the following code

import multiprocessing  import FunForParForif __name__ == '__main__':    pool = multiprocessing.Pool(processes=4)    pool.map(FunForParFor.func, range(10))    pool.close()    pool.join()       print('done')

This has worked for me! I just wanted to share it here to give you a particular example.