Pandas memoization
Author of jug here: jug works fine. I just tried the following and it works:
from jug import TaskGeneratorimport pandas as pdimport numpy as np@TaskGeneratordef gendata(): return pd.DataFrame(np.arange(343440).reshape((10,-1)))@TaskGeneratordef compute(x): return x.mean()y = compute(gendata())It is not as efficient as it could be as it just uses pickle internally for the DataFrame (although it compresses it on the fly, so it is not horrible in terms of memory use; just slower than it could be).
I would be open to a change which saves these as a special case as jug currently does for numpy arrays: https://github.com/luispedro/jug/blob/master/jug/backends/file_store.py#L102
I use this basic memoization decorator, memoized. http://wiki.python.org/moin/PythonDecoratorLibrary#Memoize
DataFrames are hashable, so it should work fine. Here's an example.
In [2]: func = lambda df: df.apply(np.fft.fft)In [3]: memoized_func = memoized(func)In [4]: df = DataFrame(np.random.randn(1000, 1000))In [5]: %timeit func(df)10 loops, best of 3: 124 ms per loopIn [9]: %timeit memoized_func(df)1000000 loops, best of 3: 1.46 us per loopLooks good to me.