Python PANDAS: Converting from pandas/numpy to dask dataframe/array Python PANDAS: Converting from pandas/numpy to dask dataframe/array numpy numpy

Python PANDAS: Converting from pandas/numpy to dask dataframe/array


Not sure if this is exactly what you are looking for, but I replaced the da.repeat with using np.repeat, along with explicity casting dd_test.index and dd_test['units'] to numpy arrays, and finally adding dd_test['transaction_dt'].astype('M8[us]') to your timedelta calculation.

df_test = pd.read_csv(StringIO(test_data), sep=',')dd_test = dd.from_pandas(df_test, npartitions=3)dd_test['helper'] = 1dd_test = dd_test.loc[np.repeat(np.array(dd_test.index), np.array(dd_test['units']))]dd_test['transaction_dt'] = dd_test['transaction_dt'].astype('M8[us]') + (dd_test.groupby('id')['helper'].cumsum()).astype('timedelta64[D]')dd_test = dd_test.reset_index(drop=True)df_expected = dd_test.compute()