Multiprocessing incompatible with NumPy [duplicate]

(First Post sorry if it is not well formulated or alligned)

You can stop Numpy to use multithreading by seting the MKL_NUM_THREADS to 1

Under debian I used:

export MKL_NUM_THREADS=1

Source from related stackoverflow post: Python: How do you stop numpy from multithreading?

Result:

user@pc:~/tmp$ python multi.pyCostlyFunc : elapsed time 3.847009 sCostlyFunc : elapsed time 3.253226 sCostlyFunc : elapsed time 3.415734 sElapsed time (serial) :  10.5163660049CostlyFunc : elapsed time 4.218424 sCostlyFunc : elapsed time 5.252429 sCostlyFunc : elapsed time 4.862513 sElapsed time (multiporcessing) :  9.11713695526user@pc:~/tmp$ export MKL_NUM_THREADS=1user@pc:~/tmp$ python multi.pyCostlyFunc : elapsed time 3.014677 sCostlyFunc : elapsed time 3.102548 sCostlyFunc : elapsed time 3.060915 sElapsed time (serial) :  9.17840886116CostlyFunc : elapsed time 3.720322 sCostlyFunc : elapsed time 3.950583 sCostlyFunc : elapsed time 3.656165 sElapsed time (multiporcessing) :  7.399310112

I am not sure if that helps because I guess eventually you want numpy to run in parallel maybe try to adjust the number of threads for numpy to your machine.

From the comments on your question, have a look at that link@Ophion no, but I have flagged it as a duplicate of Why does multiprocessing use only a single core after I import numpy? – ali_m Aug 22 at 9:06

I would check to see if you are using an optimized version of BLAS. I have found that some generic installs of numpy do not deliver and optimized version of this lib. From my install you can note that is points to libf77blas.so, libcblas.so, libatlas.so.

Here are the instructions to build an optimized version of BLAS: http://docs.scipy.org/doc/numpy/user/install.html

From with in python:

import numpy.core._dotblas

>>> numpy.core._dotblas.__file__## output:'PYTHONHOME/lib/python2.7/site-packages/numpy/core/_dotblas.so'

From your terminal:

$ ldd 'PYTHONHOME/lib/python2.7/site-packages/numpy/core/_dotblas.so'linux-vdso.so.1 =>  (0x00007fff241ff000)libf77blas.so => /opt/arch/intel/lib/libf77blas.so (0x00007f6050647000)libcblas.so => /opt/arch/intel/lib/libcblas.so (0x00007f6050429000)libatlas.so => /opt/arch/intel/lib/libatlas.so (0x00007f604fbf1000)libpython2.7.so.1.0 => 'PYTHONHOME/lib/libpython2.7.so.1.0 (0x00007f604f817000)libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f604f5f9000)libc.so.6 => /lib64/libc.so.6 (0x00007f604f266000)libgfortran.so.3 => /usr/lib64/libgfortran.so.3 (0x00007f604ef74000)libm.so.6 => /lib64/libm.so.6 (0x00007f604ecef000)libdl.so.2 => /lib64/libdl.so.2 (0x00007f604eaeb000)libutil.so.1 => /lib64/libutil.so.1 (0x00007f604e8e8000)

/lib64/ld-linux-x86-64.so.2 (0x0000003c75e00000)