How can I memoize a class instantiation in Python? How can I memoize a class instantiation in Python? python python

How can I memoize a class instantiation in Python?


Let us see two points about your question.

Using memoize

You can use memoization, but you should decorate the class, not the __init__ method. Suppose we have this memoizator:

def get_id_tuple(f, args, kwargs, mark=object()):    """     Some quick'n'dirty way to generate a unique key for an specific call.    """    l = [id(f)]    for arg in args:        l.append(id(arg))    l.append(id(mark))    for k, v in kwargs:        l.append(k)        l.append(id(v))    return tuple(l)_memoized = {}def memoize(f):    """     Some basic memoizer    """    def memoized(*args, **kwargs):        key = get_id_tuple(f, args, kwargs)        if key not in _memoized:            _memoized[key] = f(*args, **kwargs)        return _memoized[key]    return memoized

Now you just need to decorate the class:

@memoizeclass Test(object):    def __init__(self, somevalue):        self.somevalue = somevalue

Let us see a test?

tests = [Test(1), Test(2), Test(3), Test(2), Test(4)]for test in tests:    print test.somevalue, id(test)

The output is below. Note that the same parameters yield the same id of the returned object:

1 30723196602 30723196923 30723197242 30723196924 3072319756

Anyway, I would prefer to create a function to generate the objects and memoize it. Seems cleaner to me, but it may be some irrelevant pet peeve:

class Test(object):    def __init__(self, somevalue):        self.somevalue = somevalue@memoizedef get_test_from_value(somevalue):    return Test(somevalue)

Using __new__:

Or, of course, you can override __new__. Some days ago I posted an answer about the ins, outs and best practices of overriding __new__ that can be helpful. Basically, it says to always pass *args, **kwargs to your __new__ method.

I, for one, would prefer to memoize a function which creates the objects, or even write a specific function which would take care of never recreating a object to the same parameter. Of course, however, this is mostly a opinion of mine, not a rule.


The solution that I ended up using is this:

class memoize(object):    def __init__(self, cls):        self.cls = cls        self.__dict__.update(cls.__dict__)        # This bit allows staticmethods to work as you would expect.        for attr, val in cls.__dict__.items():            if type(val) is staticmethod:                self.__dict__[attr] = val.__func__    def __call__(self, *args):        key = '//'.join(map(str, args))        if key not in self.cls.instances:            self.cls.instances[key] = self.cls(*args)        return self.cls.instances[key]

And then you decorate the class with this, not __init__. Although brandizzi provided me with that key piece of information, his example decorator didn't function as desired.

I found this concept quite subtle, but basically when you're using decorators in Python, you need to understand that the thing that gets decorated (whether it's a method or a class) is actually replaced by the decorator itself. So for example when I'd try to access Photograph.instances or Camera.generate_id() (a staticmethod), I couldn't actually access them because Photograph doesn't actually refer to the original Photograph class, it refers to the memoized function (from brandizzi's example).

To get around this, I had to create a decorator class that actually took all the attributes and static methods from the decorated class and exposed them as it's own. Almost like a subclass, except that the decorator class doesn't know ahead of time what classes it will be decorating, so it has to copy the attributes over after the fact.

The end result is that any instance of the memoize class becomes an almost transparent wrapper around the actual class that it has decorated, with the exception that attempting to instantiate it (but really calling it) will provide you with cached copies when they're available.


The parameters to __new__ also get passed to __init__, so:

def __init__(self, flubid):    ...

You need to accept the flubid argument there, even if you don't use it in __init__

Here is the relevant comment taken from typeobject.c in Python2.7.3

/* You may wonder why object.__new__() only complains about arguments   when object.__init__() is not overridden, and vice versa.   Consider the use cases:   1. When neither is overridden, we want to hear complaints about      excess (i.e., any) arguments, since their presence could      indicate there's a bug.   2. When defining an Immutable type, we are likely to override only      __new__(), since __init__() is called too late to initialize an      Immutable object.  Since __new__() defines the signature for the      type, it would be a pain to have to override __init__() just to      stop it from complaining about excess arguments.   3. When defining a Mutable type, we are likely to override only      __init__().  So here the converse reasoning applies: we don't      want to have to override __new__() just to stop it from      complaining.   4. When __init__() is overridden, and the subclass __init__() calls      object.__init__(), the latter should complain about excess      arguments; ditto for __new__().   Use cases 2 and 3 make it unattractive to unconditionally check for   excess arguments.  The best solution that addresses all four use   cases is as follows: __init__() complains about excess arguments   unless __new__() is overridden and __init__() is not overridden   (IOW, if __init__() is overridden or __new__() is not overridden);   symmetrically, __new__() complains about excess arguments unless   __init__() is overridden and __new__() is not overridden   (IOW, if __new__() is overridden or __init__() is not overridden).   However, for backwards compatibility, this breaks too much code.   Therefore, in 2.6, we'll *warn* about excess arguments when both   methods are overridden; for all other cases we'll use the above   rules.*/