Import huge data-set from SQL server to HDF5
Try this:
sql_reader = pd.read_sql("select * from table_a", con, chunksize=10**5)hdf_fn = '/path/to/result.h5'hdf_key = 'my_huge_df'store = pd.HDFStore(hdf_fn)cols_to_index = [<LIST OF COLUMNS THAT WE WANT TO INDEX in HDF5 FILE>]for chunk in sql_reader: store.append(hdf_key, chunk, data_columns=cols_to_index, index=False)# index data columns in HDFStorestore.create_table_index(hdf_key, columns=cols_to_index, optlevel=9, kind='full')store.close()