Why does copying a >= 16 GB Numpy array set all its elements to 0?
This is just a guess. I don't have any evidence supporting the following claims at the moment but my guess is that this is a simple overflow problem:
>>> np.arange(2 ** 31 - 1).size2147483647Which just happens to be the largest int32 value:
>>> np.iinfo(np.int32)iinfo(min=-2147483648, max=2147483647, dtype=int32)So when you actually have an array with a size of 2147483648 (2**31) and use an int32 this would overflow and give an actual negative value. Then there is probably something like this inside the numpy.ndarray.copy method:
for (i = 0 ; i < size ; i ++) { newarray[i] = oldarray[i]}But given that the size is now negative the loop wouldn't execute because 0 > -2147483648.
That the new array is actually initialized with zeros is strange because it wouldn't make sense to actually put zeros before one copies the array (but it could be something like in this question).
Again: That's just guessing at this point but it would match the behaviour.