Accueil / Blog / Métier / 2016 / Monkey-patching a Python instance method

Monkey-patching a Python instance method

Par Alex Marandon publié 09/11/2016
Dynamically adding or overwriting an instance method in Python is rarely needed, but it's a good excuse to explore interesting aspects of the language that aren't always well known: the descriptor protocol, types.MethodType and partial function applications.

For a Django project, I needed an easy way to automatically add a specific query string parameter named token to any URL. The solution I came up with was to create a template tag that delegates to the built-in url template tag to build the base URL and then adds the query string parameter. I my templates I can call this template tag like this:

{% token_url 'url_name' %}

The template tag code looks like this:

from urlparse import urlsplit, urlunsplit, parse_qs
from urllib import urlencode
from django.template.defaulttags import URLNode, url
from django import template

register = template.Library()


# My custom render method
def render_with_token(self, context):
    result = URLNode.render(self, context)
    request = context['request']
    token = request.META.get('HTTP_X_ACME_TOKEN')
    if token:
        url = urlsplit(result)
        query_dict = parse_qs(url.query)
        query_dict['token'] = token
        query_string = urlencode(query_dict, doseq=True)
        parts = list(url)
        parts[3] = query_string
        return urlunsplit(parts)
    else:
        return result


@register.tag
def token_url(parser, token):
    node = url(parser, token)

    # call __get__ to turn the function into a bound method
    bound_method = render_with_token.__get__(node, URLNode)
    # Patch the URLNode instance.
    node.render = bound_method
    return node

When called programmatically, the url template tag returns a URLNode instance. Patching the URLNode class was not going to work because it would have affected invocations of the standard url template tag. I couldn't just overwrite the render method of the URLNode class with code like this:

URLNode.render = render_with_token  # Won't do what I need

The descriptor protocol

What I needed to do was to customize the render method for the current URLNode instance, without patching the whole URLNode class. So I turned my render_to_token function into a method of the current instance with this line of code:

bound_method = render_with_token.__get__(node, URLNode)

To understand this line, you need to understand Python's descriptor protocol and specifically how this protocol is used to implement methods. The descriptor protocol allows to customize how objects behave when they are accessed as attributes of other objects. When you put def statements inside a class statement, Python creates regular functions and set them as attributes of the class, just like it would do for numbers, strings or any other object. It just happens that Python functions implement the descriptor protocol and know what to do when they're accessed as class attributes or as instance attributes. The intelligence is in the function, not in the class. Usually, the descriptor protocol is triggered implicitly when an object is accessed as an attribute. But in our situation we cannot do that since the function is not yet an attribute of the instance. So here we trigger the descriptor protocol explicitly by calling the __get__ method of the function object. Since we're passing an instance as the the first argument, the __get__ method knows that it should return an instance method. If we had passed None, we would get back the function unchanged:

>>> def foo():
...     pass
...
>>> class Bar:
...     pass
...
>>> foo.__get__(Bar(), Bar)
<bound method foo of <__main__.Bar object at 0x7fbdd27dfeb8>>
>>> foo.__get__(None, Bar)
<function foo at 0x7fbdd2b82f28>
>>> foo.__get__(Bar(), Bar) is foo
False
>>> foo.__get__(None, Bar) is foo
True

types.MethodType

If you've read the section on functions and methods of the Descriptor HowTo linked above, you might have noticed that it refers to types.MethodType. This type can be used to create instance methods or class methods:

>>> import types
>>> def foo(arg):
...     print("I got arg {0}".format(arg))
...
>>> class Bar:
...     pass
...
>>> obj = Bar()
>>> obj.foo = types.MethodType(foo, obj)
>>> obj.foo()
I got arg <__main__.Bar object at 0x7f49f8227080>
>>> obj.foo = types.MethodType(foo, Bar)
>>> obj.foo()
I got arg <class '__main__.Bar'>

So I could have patched the render method of a specific URLNode instance like this:

import types

@register.tag
def token_url(parser, token):
    node = url(parser, token)

    # Construct an instance method
    bound_method = types.MethodType(render_with_token, node)
    # Patch the URLNode instance
    node.render = bound_method
    return node

Although it requires an extra import, this solution is probably more explicit and understandable by someone who doesn't know about the descriptor protocol. We can tell what is going on just by looking at the code.

Partial function application

If you think about it, what is a Python method? In essence, it's just a function that receives its first parameter automatically. In the case of instance methods, this parameter is the instance on which the method is invoked, usually named self. It turns out we have in the standard library a generic tool to create functions that get some of their parameters automatically when they're called:

>>> import functools
>>> def foo(arg):
...     print("I got arg {0}".format(arg))
...
>>> foo42 = functools.partial(foo, 42)
>>> foo42()
I got arg 42

We can use functools.partial to create functions that behave like methods:

>>> class Bar:
...      pass
...
>>> obj = Bar()
>>> obj.foo = functools.partial(foo, obj)
>>> obj.foo()
I got arg <__main__.Bar object at 0x7f49f76cd748>

So I could have implemented my Django template tag like this:

import functools

@register.tag
def token_url(parser, token):
    node = url(parser, token)

    # Use partial function application to pass the node instance as `self`
    bound_method = functools.partial(render_with_token, node)
    # Patch the URLNode instance.
    node.render = bound_method
    return node

Although this works, here we haven't created a real method:

>>> type(obj.foo)
<class 'functools.partial'>
>>> isinstance(obj.foo, types.MethodType)
False

This could be an issue with code that relies on introspection and expects to find actual methods rather than functions that behave like methods. But I feel this solution is rather elegant because it relies on a generic programming concept, partial function application, that is by no means specific to python and with which most programmers should be familiar. This is the solution suggested by Raymond Hettinger.

In conclusion

I hope you found this little problem interesting and that it was an opportunity to learn a thing or two about Python. Each of these three solutions as trade-offs and, as an exception to the rule, there isn't one obvious way to do it (but I'm not Dutch).

ABONNEZ-VOUS À LA NEWSLETTER !
Voir aussi
Récupérer des données Hadoop avec Python 02/05/2017

Accéder à des données situées sur un cluster Hadoop peut se faire de différentes manières en ...

Les curseurs PostgreSQL 11/06/2015

Découvrons comment utiliser les curseurs PostgreSQL pour effectuer des requêtes renvoyant de ...

Présentation de l'écosystème Python scientifique Présentation de l'écosystème Python scientifique 10/11/2016

Au fil des années Python est devenu un outil du quotidien pour les ingénieurs et chercheurs de ...

Formation Python avancé du 12 au 16 juin à Paris Formation Python avancé du 12 au 16 juin à Paris 12/04/2017

Pour les développeurs qui veulent approfondir leur connaissance du langage Python : de la ...

Python : Bien configurer son environnement de développement Python : Bien configurer son environnement de développement 07/12/2015

Comment utiliser les bonnes pratiques de développement Python.