How to modify the behavior of def in Python?

Question

Most things in Python are readily modifiable, up to and including being able to instantiate functions directly with Types.FunctionType using your favorite code object. Not that one should ever necessarily do so, but as a learning experience I'm trying to figure out how to modify the behavior of def from within Python itself (modifying the language definition feels like cheating for this one).

For classes, this can be done fairly easily in 3.1+ with the __build_class__ hook. Is there similar machinery for hooking function building?

So far I've tried modifying compile(), eval(), exec(), type(), Types.FunctionType, and anything else that seemed relevant in any place I could find them. As best as I can tell, def does not execute any of these when it creates a function object and loads it into globals(). In as much detail as possible, what exactly happens when we define a function? Is the entire process done behind the scenes in the underlying C code?

Answer 1

There be dragons here. Continue at your own peril.

Judging by the overwhelming lack of a response and my continued inability to find documentation for the feature I want, I'm going to go out on a limb and say it doesn't exist. That said, you can patch class definitions to behave like function definitions. Intriguing, right?

In Python 3.1+, the following (with a helper file that I'll get to in a minute) is legal code that behaves roughly as you might expect.

class fib(n=5):
    if n < 2:
        res = 1
    a = b = 1
    for i in range(2, n+1):
        a, b = b, a+b
    res = b

Now examine the code's output:

>>> fib(n=3)
3

>>> fib(n=4)
5

>>> fib()
8

>>> fib(n=10)
89

We're able to call this class like a function with default arguments and get the correct values. Notice the complete lack of __init__(), __new__(), or any of the other dunder methods.

Warning: It probably comes as no surprise to you, but the following is definitely not production ready (I'm still learning, sorry).

Our weapon of choice is to override builtins.__build_class__ for our own gain and profit. Note that there is only one copy of builtins in a python application, and messing it up in one module affects all modules. To mitigate the damage, we're going to move all the voodoo to its own module, which I just called base for simplicity.

The way I chose to do that override is to allow each module to register itself with base, along with a decorator they'd like to apply to their class-functions (why go through the trouble to modify every class if you aren't doing something to all of them?)

import builtins

_overrides = {}

def register(f, module=None):
    module = module if module is not None else f.__module__
    _overrides[module] = f

def revoke(x):
    try:
        del _overrides[x]
    except KeyError:
        del _overrides[x.__module__]

That looks like a lot of code, but all it's doing is making a dictionary _overrides and allowing code from any module to register itself in that dictionary. In case they want to use an external function but still have the weird class behavior only apply to themselves, we allow modules to explicitly pass themselves into the register() function.

Before we start fiddling with anything, we need to store the old __build_class__() function so we can use it in any unregistered modules.

_obc = builtins.__build_class__

The new __build_class__() function then just checks if a module is registered. If so, it does some magic, and otherwise it calls the original builtin.

def _bc(f, name, *a, mc=None, **k):
    mc = type if mc is None else mc
    try:
        w = _overrides[f.__module__]
    except KeyError:
        return _obc(f, name, *a, metaclass=mc, **k)
    return _cbc(f, name, w, *a, **k)

Note that the default type of a class is type. Also, we're explicitly passing the wrapper w from _overrides into our custom method _cbc() because we don't have control over revoke(). If we just checked if a module was registered, a user might very well unregister it before we could query _overrides for the wrapper.

As far as the magic that treats a class like code, it's a drop-in replacement for __build_code__().

def _cbc(f, name, w, **k):
    def g(**x):
        for key in k:
            if key not in x:
                x[key] = k[key]
        exec(f.__code__, {}, x)
        return x['res']
    t = type(name, (), {})
    t.__new__ = lambda self, **kwargs: w(g)(**kwargs)
    return t

Walking through this, the function _cbc() takes the function object f returned by the Python interpreter as it reads our class definition and passes its code straight into exec(). If you happened to pass in any keyword arguments to the function g() it happily throws those into exec() as well. When it's all said and done, your class-function is expected to have assigned a value to res, so we return that.

We still have to actually create a class though. The type metaclass is the standard way to create a vanilla class, so we make one. To actually call the g() thing we just created, we assign it to __new__() on the new class so that when somebody tries to instantiate our class everything is instead passed into __new__() (along with an extra self argument that we don't care about).

Lastly, we overwrite the builtins with our custom method.

builtins.__build_class__ = _bc

To use the new toys we need to import them. I called my library base, but you could use almost anything.

import base

Then base.register() is the hook to start changing how class definitions work. Our lazy implementation requires a function to be passed in, so we can just use the identity.

base.register(lambda f: f)

At this point, the Fibonacci code from the beginning will work exactly as advertised. If you want a normal class, just call base.revoke(lambda:1) to temporarily exclude the current module from the abnormal behavior.

To make things interesting, we can apply wrappers that affect every class-function defined this way. You might use this for some sort of logging or user-validation.

import datetime

def verbose(f):
    def _f(*a,**k):
        print (f'Running at {datetime.datetime.now()}')
        return f(*a,**k)
    return _f

base.register(verbose)

>>> fib(n=10)
    Running at 2018-08-25 06:07:56.258317
89

One last time, this is not production ready. Nesting function definitions inside the class-functions works fine, but other kinds of nesting and recursion are a little messy. My naive way of handling closures is pretty fragile. If anybody has some good documentation for Python's internals, I would be very appreciative.