Is it possible to map a discontiuous data on disk to an array with python?
I posted another answer because for the example given here numpy.memmap worked:
offset = 0data1 = np.memmap('tmp', dtype='i', mode='r+', order='F', offset=0, shape=(size1))offset += size1*byte_sizedata2 = np.memmap('tmp', dtype='i', mode='r+', order='F', offset=offset, shape=(size2))offset += size1*byte_sizedata3 = np.memmap('tmp', dtype='i', mode='r+', order='F', offset=offset, shape=(size3))for int32 byte_size=32/8, for int16 byte_size=16/8 and so forth...
If the sizes are constant, you can load the data in a 2D array like:
shape = (total_length/size,size)data = np.memmap('tmp', dtype='i', mode='r+', order='F', shape=shape)You can change the memmap object as long as you want. It is even possible to make arrays sharing the same elements. In that case the changes made in one are automatically updated in the other.
Other references: