Numpy Memory Error on Linux Server but not Mac

My best guess are:

1. The Mac has a swap that allows more allocated memory than the RAM you see.
2. The Mac does not realise that the array does not fit in memory until the memory is actually used. So the array actually does not fit in memory but you will not know it until you use that memory.

I base my first guess in the fact that in 64 bit your array will take 500000*10000*8= 40GB of RAM 20GB in 32 bit, and therefore the array does not fit in the memory you have. There may be a swap to account for the missing memory.

I base my second guess in this link, where it is explained that np.zeros will not allocate actually in memory the zeros until that memory is accessed for the first time. I have tested in my linux (Ubuntu) computer that np.zeros works with increasing arrays until I reach my RAM limit. Then I get a memory error even if it does not actually allocate the memory.

Once you create the matrix (increase the size enough to make it clear the memory usage):

a = np.zeros((50,10))

You can check the actual memory required by storing a zero in each cell of the matrix:

a[:,:] = 0.0

Or forcing an operation so the memory is accessed and therefore allocated:

a = a + a

Keep track of the memory usage of the computer while performing this check to understand when the memory is allocated.