Dictionary-like efficient storing of scipy/numpy arrays

as JoshAdel already suggested, I would go for HDF5, the simplest way is to use h5py:

http://h5py.alfven.org/

you can attach several attributes to an array with a dictionary like sintax:

dset.attrs["Name"] = "My Dataset"

where dset is your dataset which can be sliced exactly as a numpy array, but in the background it does not load all the array into memory.

I would suggest to use scipy.save and have an dictionnary between the word and the name of the files.

Have you tried just using cPickle to pickle the dictionary directly using:

import cPickleDD = dict()f = open('testfile.pkl','wb')cPickle.dump(DD,f,-1)f.close()

Alternatively, I would just save the vectors in a large multidimensional array using hdf5 or netcdf if necessary since this allows you to open a large array without bringing it all into memory at once and then get slices as needed. You can then associate the words as an additional group in the netcdf4/hdf5 file and use the common indices to quickly associate the appropriate slice from each group, or just name the group the word and then have the data be the vector. You'd have to play around with which is more efficient.

http://netcdf4-python.googlecode.com/svn/trunk/docs/netCDF4-module.html

Pytables also might be a useful storage layer on top of HDF5:

http://www.pytables.org