Unable to load a previously dumped pickle file of large size in Python
How about save & load the numpy array by numpy.save() & np.load()?
You can save the pickled list and the numpy array to the same file:
import numpy as npimport cPickledata = np.random.rand(50000000)f = open('foo.pck', 'wb')cPickle.dump([1,2,3], f, protocol=2)np.save(f, data)f.close()to read the data:
import cPickleimport numpy as npf= open('foo.pck', 'rb')v = cPickle.load(f)data = np.load(f)print data.shape, data