What's the maximum size of a numpy array?

You're trying to create an array with 2.7 billion entries. If you're running 64-bit numpy, at 8 bytes per entry, that would be 20 GB in all.

So almost certainly you just ran out of memory on your machine. There is no general maximum array size in numpy.

A ValueError indicates the size is too big for allocation, not that there is not enough memory. On my laptop, using 64bits python, I can allocate it if I reduce the number of bits:

In [16]: a=np.arange(2708000000)---------------------------------------------------------------------------MemoryError                               Traceback (most recent call last)<ipython-input-16-aaa1699e97c5> in <module>()----> 1 a=np.arange(2708000000)MemoryError: # Note I don't get a ValueErrorIn [17]: a = np.arange(2708000000, dtype=np.int8)In [18]: a.nbytesOut[18]: 2708000000In [19]: a.nbytes * 1e-6Out[19]: 2708.0

In your case, arange uses int64 bits, which means it is 16 times more, or around 43 GB. a 32 bits process can only access around 4 GB of memory.

The underlying reason is the size of the pointers used to access data and how many numbers you can represent with those bits:

In [26]: np.iinfo(np.int32)Out[26]: iinfo(min=-2147483648, max=2147483647, dtype=int32)In [27]: np.iinfo(np.int64)Out[27]: iinfo(min=-9223372036854775808, max=9223372036854775807, dtype=int64)

Note that I can replicate your ValueError if I try to create an absurdly large array:

In [29]: a = np.arange(1e350)---------------------------------------------------------------------------ValueError                                Traceback (most recent call last)<ipython-input-29-230a6916f777> in <module>()----> 1 a = np.arange(1e350)ValueError: Maximum allowed size exceeded

If your machine has a lot of memory, as you said, it will be 64 bits, so you should install Python 64 bits to be able to access it. On the other hand, for such big datasets, you should consider the possibility of using out of core computations.

I was able to create an array with a size of 6Billion that ate up 45GB of memory. By default, numpy created the array with a dtype of float64. By dropping the precision, I was able to save a lot of memory.

np.arange(6000000000,dtype=np.dtype('f8'))np.arange(6000000000,dtype=np.dtype('f4'))#etc...

default == float64

• np.float64 -- 45.7GB

• np.float32 -- 22.9GB

• np.int8 -- 5.7GB

Obviously a 8bit integer cant store a value of 6B. I'm sure a max size exists at some point but I suspect it's FAR past anything possible in 2016. Interestingly, "Python Blaze" allows you to create numpy arrays on disk. I recall playing with it some time ago and creating an extremely large array that took up 1TB of disk.