Lazy class factory?

Question

I have a situation where I would like to be able to use a base class to construct objects of derived classes. The specific child class returned is dependent on information that cannot be passed to the constructor/factory method because it isn't available yet. Instead, that information has be downloaded and parsed to determine the child class.

So I'm thinking that I want to lazily initialize my objects, passing only a URL from which it can download the needed info, but wait to actually generate the child object until the program needs it (i.e. on first access).

So when the object is first created, it is an object of the base class. However, when I access it for the first time, I want it to download its information, convert itself to the appropriate derived class, and return the requested information.

How would I do this in python? I'm thinking I want something like a factory method, but with some kind of delayed-initialization feature. Is there a design pattern for this?

Answer 1

That can be done in a couple ways in Python - possibly even without resorting to a metaclass.

If you could just create the class instances until the moment you will need then, it is just a matter of creating a callable - which could be a partial function - that would compute the class and create the instances.

But your text describes you want the class to just "intialize" - including changing its own type - on "first access". That is feasible - but then it requires some wiring in the class special methods - and if by "first access" you mean jsut calling a method or reading an attribute, that is easy - we only have to customize the __getattribute__ method to trigger the initialization mechanism. If, on the other hand, your class will implement the "magic" "dunder" methods of Python, such as __len__ , __getitem__ or __add__ - then "first access" may mean triggering one of these methods on the instance, it is a bit trickier - each of the dunder methods must be wrapped by code that will cause the initialization to happen - as access to these methods does not go through __getattribute__.

As for setting the subclass type itself, it is a matter of setting the __class__ attribute to the correct subclass on the instance. Python allows that if all ancestors of both classes (old and new) have the same __slots__ set - so, even if you use the __slots__ feature, you just can't change that on the subclasses.

So, there are three cases:

1 - Late instantiation

Wrap the class definition itself into a function, pre-loaded with the URL or other data it needs. When the function is called, the new class is computed and instantiated.

from functools import lru_cache

class Base:        
    def __repr__(self):
        return f"Instance of {self.__class__.__name__}"

@lru_cache
def compute_subclass(url):
    # function that eagerly computes the correct subclass
    # given the url .
    # It is strongly suggested that this uses some case
    # of registry, so that classes that where computed once,
    # are readly available when the input parameter is defined.
    # Python's  lru_cache decorator can do that
    ...
    class Derived1(Base):
        def __init__(self, *args, **kwargs):
            self.parameter = kwargs.pop("parameter", None)

    ...
    subclass = Derived1

    return subclass

def prepare(*args, **kwargs):
    def instantiate(url):
        subclass = compute_subclass(url)
        instance = subclass(*args, **kwargs)
        return instance
    return instantiate

And this can be used as:

In [21]: lazy_instance = prepare(parameter=42)                                                                

In [22]: lazy_instance                                                                                        
Out[22]: <function __main__.prepare.<locals>.instantiate(url)>

In [23]: instance = lazy_instance("fetch_from_here")                                                          

In [24]: instance                                                                                             
Out[24]: Instance of Derived1

In [25]: instance.parameter                                                                                   
Out[25]: 42

2 - initialize on attribute/method access - no special __magic__ methods

Trigger the class-computation and initialization in the class __getattribute__ method

from functools import lru_cache
class Base:
    def __init__(self, *args, **kwargs):
        # just annotate intialization parameters that can be later
        # fed into sublasses' init. Also, this can be called
        # more than once (if subclasses call "super"), and it won't hurt

        self._initial_args = args
        self._initial_kwargs = kwargs
        self._initialized = False

    def _initialize(self):
        if not self._initialized:
            subclass = compute_subclass(self._initial_kwargs["url"])
            self.__class__ = subclass
            self.__init__(*self._initial_args, **self._initial_kwargs)
            self._initialized = True

    def __repr__(self):
        return f"Instance of {self.__class__.__name__}"

    def __getattribute__(self, attr):
        if attr.startswith(("_init", "__class__", "__init__")): # return real attribute, no side-effects:
            return object.__getattribute__(self, attr)
        if not self._initialized:
            self._initialize()
        return object.__getattribute__(self, attr)    


@lru_cache
def compute_subclass(url):
    # function that eagerly computes the correct subclass
    # given the url .
    # It is strongly suggested that this uses some case
    # of registry, so that classes that where computed once,
    # are readly available when the input parameter is defined.
    # Python's  lru_cache decorator can do that

    print(f"Fetching initialization data from {url!r}")
    ...
    class Derived1(Base):
        def __init__(self, *args, **kwargs):
            self.parameter = kwargs.pop("parameter", None)

        def method1(self):
            return "alive"

    ...
    subclass = Derived1

    return subclass

And this works seamlessly After the instance is created:

>>> instance = Base(parameter=42, url="this.place")
>>> instance
Instance of Base
>>> instance.parameter
Fetching initialization data from 'this.place'
42
>>> instance
Instance of Derived1
>>> 
>>> instance2 = Base(parameter=23, url="this.place")
>>> instance2.method1()
'alive'

But the parameters needed to compute the subclass lazily have to be passed someway - on this example, I require then to be passed in a "url" parameter to the base class - but even this example can work if the url is not available at this time. Before using the instance you can update the url by doing instance._initial_kwargs["url"] = "i.got.it.now".

Also, for the demo, I had to go into plain Python instead of IPython, as the IPython CLI will introspect the new instance, triggering its transformation.

3 - Initialize on operator use - specialized __magic__ methods.

Have a metaclass that wraps the baseclass magic methods with a decorator that will compute the new class and perform the initialization.

The code for this would be quite similar to the previous one, but on a metaclass to Base, the __new__ method would have to check all __magic__ methods and decorate then with a call to self._initialize.

This has some twists to make the magic methods behave properly both in case they are overriden in the subclass and in the case they are called in the initial base. Anyway, all possible magic methods to be used by the subclasses have to be defined in Base, even if all they do is raise a "NotImplementedError" -

from functools import lru_cache, wraps

def decorate_magic_method(method):
    @wraps(method)
    def method_wrapper(self, *args, **kwargs):
        self._initialize()
        original_method = self.__class__._initial_wrapped[method.__name__]
        final_method = getattr(self.__class__, method.__name__)
        if final_method is method_wrapper:
            # If magic method has not been overriden in the subclass
            final_method = original_method
        return final_method(self, *args, **kwargs)
    return method_wrapper


class MetaLazyInit(type):
    def __new__(mcls, name, bases, namespace, **kwargs):
        wrapped = {}
        if name == "Base":
            # Just wrap the magic methods in the Base class itself

            for key, value in namespace.items():
                if key in ("__repr__", "__getattribute__", "__init__"):
                    # __repr__ does not need to be in the exclusion - just for the demo.
                    continue

                if key.startswith("__") and key.endswith("__") and callable(value):
                    wrapped[key] = value
                    namespace[key] = decorate_magic_method(value)

            namespace["_initial_wrapped"] = wrapped
        namespace["_initialized"] = False
        return super().__new__(mcls, name, bases, namespace, **kwargs)


class Base(metaclass=MetaLazyInit):
    def __init__(self, *args, **kwargs):
        # just annotate intialization parameters that can be later
        # fed into sublasses' init. Also, this can be called
        # more than once (if subclasses call "super"), and it won't hurt
        self._initial_args = args
        self._initial_kwargs = kwargs

    def _initialize(self):
        print("_initialize called")
        if not self._initialized:
            self._initialized = True
            subclass = compute_subclass(self._initial_kwargs["url"])
            self.__class__ = subclass
            self.__init__(*self._initial_args, **self._initial_kwargs)

    def __repr__(self):
        return f"Instance of {self.__class__.__name__}"

    def __getattribute__(self, attr):
        if attr.startswith(("_init", "__class__")) : # return real attribute, no side-effects:
            return object.__getattribute__(self, attr)
        if not self._initialized:
            self._initialize()
        return object.__getattribute__(self, attr)

    def __len__(self):
        return 5

    def __getitem__(self, item):
        raise NotImplementedError()

@lru_cache
def compute_subclass(url):
    # function that eagerly computes the correct subclass
    # given the url .
    # It is strongly suggested that this uses some case
    # of registry, so that classes that where computed once,
    # are readly available when the input parameter is defined.
    # Python's  lru_cache decorator can do that

    print(f"Fetching initialization data from {url!r}")
    ...

    class TrimmedMagicMethods(Base):
        """This intermediate class have the initial magic methods
        as declared in Base - so that after the subclass instance
        is initialized, there is no overhead call to "self._initialize"
        """
        for key, value in Base._initial_wrapped.items():
            locals()[key] = value
            # Special use of "locals()" in the class body itself,
            # not inside a method, creates new class attributes

    class DerivedMapping(TrimmedMagicMethods):
        def __init__(self, *args, **kwargs):
            self.parameter = kwargs.pop("parameter", None)

        def __getitem__(self, item):
            return 42


    ...
    subclass = DerivedMapping

    return subclass

And on the terminal:

>>> reload(lazy_init); Base=lazy_init.Base
<module 'lazy_init' from '/home/local/GERU/jsbueno/tmp01/lazy_init.py'>
>>> instance = Base(parameter=23, url="fetching from there")
>>> instance
Instance of Base
>>> 
>>> instance[0]
_initialize called
Fetching initialization data from 'fetching from there'
42
>>> instance[1]
42
>>> len(instance)
5
>>> instance2 = Base(parameter=23, url="fetching from there")
>>> len(instance2)
_initialize called
5
>>> instance3 = Base(parameter=23, url="fetching from some other place")
>>> len(instance3)
_initialize called
Fetching initialization data from 'fetching from some other place'
5