python child process crashes on numpy dot if PySide is imported

this is a general issue with some BLAS libraries used by numpy for dot.

Apple Accelerate and OpenBlas built with GNU Openmp are known to not be safe to use on both sides of a fork (the parent and the child process multiprocessing create). They will deadlock.

This cannot be fixed by numpy but there are three workarounds:

• use netlib BLAS, ATLAS or git master OpenBlas based on pthreads (2.8.0 does not work)
• use python 3.4 and its new multiprocessing spawn or forkserver start methods
• use threading instead of multiprocessing, numpy releases the gil for most expensive operations so you can archive decent threading speedups on typical desktop machines

I believe this to be an issue with the multiprocessing module.

Try using the following instead.

import numpy as npimport PySide    def hang():        import multiprocessing.dummy as multiprocessing        pool = multiprocessing.Pool(processes = 1)        pool.map(f, [None])    def f(ignore):        print('before dot..')        np.dot(np.zeros((128, 1)), np.zeros((1, 32)))        print('after dot.')    if __name__ == "__main__":        hang()        print('success!')

I ran into this exact problem. There was a deadlock when the child process used numpy.dot. But it ran when I reduced the size of the matrix. So instead of a dot product on a matrix with 156000 floats, I performed 3 dot products of 52000 each and concatenated the result. I'm not sure of what the max limit is and whether it depends on the number of child processes, available memory or any other factors. But if the largest matrix that does not deadlock can be identified by trial and error, then the following code should help.

def get_batch(X, update_iter, batchsize):    curr_ptr = update_iter*batchsize    if X.shape[0] - curr_ptr <= batchsize :        X_i = X[curr_ptr:, :]    else:        X_i = X[curr_ptr:curr_ptr+batchsize, :]    return X_idef batch_dot(X, w, batchsize):    y = np.zeros((1,))    num_batches = X.shape[0]/batchsize    if X.shape[0]%batchsize != 0:        num_batches += 1    for batch_iter in range(0, num_batches):        X_batch = get_batch(X, batch_iter, batchsize)        y_batch = X_batch.dot(w)        y = np.hstack((y, y_batch))    return y[1:]