Login With Github

Python Decorator Tutorial In 30 Minutes

The decorator in Python is a difficulty for getting started with Python. But you have to overcome it.

Why Do You Need a Decorator?

Suppose you have implemented the following two functions say_hello() and say_goodbye() in your program:

def say_hello():
    print "hello!"
    
def say_goodbye():
    print "hello!"  # bug here

if __name__ == '__main__':
    say_hello()
    say_goodbye()

But you find that both of the two above functions print hello when the program runs. After debugging, you get to know the function say_goodbye() goes wrong. So your leader makes a request: each function's name must be recorded before calling the function, like this:

[DEBUG]: Enter say_hello()
Hello!
[DEBUG]: Enter say_goodbye()
Goodbye!

A just begins to work, and here's how he implement it:

def say_hello():
    print "[DEBUG]: enter say_hello()"
    print "hello!"

def say_goodbye():
    print "[DEBUG]: enter say_goodbye()"
    print "hello!"

if __name__ == '__main__':
    say_hello()
    say_goodbye()

B has been working for quite some time, and he advises A to write like this:

def debug():
    import inspect
    caller_name = inspect.stack()[1][3]
    print "[DEBUG]: enter {}()".format(caller_name)   

def say_hello():
    debug()
    print "hello!"

def say_goodbye():
    debug()
    print "goodbye!"

if __name__ == '__main__':
    say_hello()
    say_goodbye()

Which is better? of course the latter. But it needs to call debug() for each function. What if one day your leader requires the functions in the say module not to call debug(), but the functions in other modules to call debug()?

So you need to use the decorator now.

A decorator is essentially a Python function which allows other functions to add extra functionalities without making any code changes, and its return value is a function object as well. The classic application scenarios for decorators include inserting logs, performance testing, transaction processing, caching, permission checking and so on. Decorators are great for solving these problems. With the help of decorators you can extract a lot of the same code that is not related to the function itself and reuse them later.

In a nutshell, decorators are used to add extra functionalities to existing functions or objects.

How To Write a Decorator

Earlier (Python Version < 2.4), the way to add an extra functionality to a function is like this:

def debug(func):
    def wrapper():
        print "[DEBUG]: enter {}()".format(func.__name__)
        return func()
    return wrapper

def say_hello():
    print "hello!"

say_hello = debug(say_hello)  # Add the functionality and remain the original function name unchanged

The above debug function is actually a decorator. It wraps the original function and returns another function which is added some extra functionalities. Because this is not very elegant, syntactic sugar is supported in the later Python versions. So the above code is equivalent to:

def debug(func):
    def wrapper():
        print "[DEBUG]: enter {}()".format(func.__name__)
        return func()
    return wrapper

@debug
def say_hello():
    print "hello!"

This is the simplest decorator, but there is a problem: If you need to pass parameters to the decorated function, then the decorator will be broken. The reason is that the returned function does not accept the parameters. So you can specify the decorator function wrapper to accept the same parameters as the original function. For example:

def debug(func):
    def wrapper(something):  # Specify the same parameters
        print "[DEBUG]: enter {}()".format(func.__name__)
        return func(something)
    return wrapper  # Return the wrapped function

@debug
def say(something):
    print "hello {}!".format(something)

However, there are thousands of functions in a project, and usually we only know the parameters of part functions. Fortunately, Python provides the variable argument *args and the keyword argument **kwargs. So the decorator can be used for any target function now.

def debug(func):
    def wrapper(*args, **kwargs):  # Specify variable parameters
        print "[DEBUG]: enter {}()".format(func.__name__)
        print 'Prepare and say...',
        return func(*args, **kwargs)
    return wrapper  # Return the wrapped function

@debug
def say(something):
    print "hello {}!".format(something)

You have learned the basic usage of decorators.

Advanced Usage of Decorators

The knowledge of decorators and class decorators with parameters is much deeper. Before understanding these decorators, it's best to get to know the closure of the function and the interface convention of the decorator.

Decorators with parameters

Suppose that the previous decorator not only needs to print the log information after entering a function, but also needs to specify the log level, then the decorator will be like this:

def logging(level):
    def wrapper(func):
        def inner_wrapper(*args, **kwargs):
            print "[{level}]: enter function {func}()".format(
                level=level,
                func=func.__name__)
            return func(*args, **kwargs)
        return inner_wrapper
    return wrapper

@logging(level='INFO')
def say(something):
    print "say {}!".format(something)

# If you don't use the @ syntax
# say = logging(level='INFO')(say)

@logging(level='DEBUG')
def do(something):
    print "do {}...".format(something)

if __name__ == '__main__':
    say('hello')
    do("my work")

When you add a decorator with parameters to a function, such as @logging(level='DEBUG'), you should know that it is actually a function which will be executed immediately. So as long as the result it returns is a decorator, it works.

Class-based decorators

A decorator function is actually an interface constraint that takes a callable object as a parameter and then returns a callable object. In general, callable objects in Python are functions, but there are exceptions. The object will be callable as long as an object overrides the __call__() method.

class Test():
    def __call__(self):
        print 'call me!'

t = Test()
t()  # call me

Methods like __call__ of which the head and tail are underscores are called built-in methods in Python, and they are also called magic methods sometimes. In general, it will change the internal behavior of the object when you override these magic methods. For example, the above code lets the class object get the called behavior.

You can also make the decorator accept a callable object and return a callable object (This is not too rigorous. Please read on for more), so it is also possible to implement it with classes. We can make the class's constructor __init__() accept a function, and then overload __call__() and return a function, which can also achieve the effect of the decorator function.

class logging(object):
    def __init__(self, func):
        self.func = func

    def __call__(self, *args, **kwargs):
        print "[DEBUG]: enter function {func}()".format(
            func=self.func.__name__)
        return self.func(*args, **kwargs)
@logging
def say(something):
    print "say {}!".format(something)

Class decorators with parameters

It will be a bit more complicated if you need to implement a decorator with parameters through the form of a class. Now what you pass to the constructor are not functions, but parameters. It saves the parameters through classes. And then, when overriding the __call__ method, it needs to pass a function and return a function.

class logging(object):
    def __init__(self, level='INFO'):
        self.level = level
        
    def __call__(self, func): # Pass a function
        def wrapper(*args, **kwargs):
            print "[{level}]: enter function {func}()".format(
                level=self.level,
                func=func.__name__)
            func(*args, **kwargs)
        return wrapper  #Return a function

@logging(level='INFO')
def say(something):
    print "say {}!".format(something)

Built-in Decorators

The built-in decorators are almost the same as the normal decorators, except that what it returns is not a function but a class object, so it's harder to understand.

@property

Before you get to understand this kind of decorators, you need to know how to write a property without using decorators.

def getx(self):
    return self._x

def setx(self, value):
    self._x = value
    
def delx(self):
   del self._x

# create a property
x = property(getx, setx, delx, "I am doc for x property")

The above is a standard way for writing Python properties, which is quite similar to Java, except that the code looks too cumbersome. But if you use the syntax sugar, not only it can achieve the same effect, but the code looks simpler.

@property
def x(self): ...

# It's equal to

def x(self): ...
x = property(x)

The property includes three decorators: setter, getter, deleter , which are all packaged on the basis of property(). Because the setter and deleter are the second and third arguments of property(), you can't apply the @ syntax directly. The function decorated by @property returns no longer a function, but a property object.

>>> property()
<property object at 0x10ff07940>

@staticmethod, @classmethod

You'll find that the principles for these two decorators are similar after getting to know the @property decorator. @staticmethod returns a staticmethod object, while @classmethod returns a classmethod object. They call their own __init__() constructors.

class classmethod(object):
    """
    classmethod(function) -> method
    """    
    def __init__(self, function): # for @classmethod decorator
        pass
    # ...
class staticmethod(object):
    """
    staticmethod(function) -> method
    """
    def __init__(self, function): # for @staticmethod decorator
        pass
    # ...

The @ syntax of the decorator is equivalent to calling the constructors of these two classes.

class Foo(object):

    @staticmethod
    def bar():
        pass
    
    # It's equivalent to bar = staticmethod(bar)

Now, we know more about the decorator interface mentioned above. The decorator must accept a callable object, but the result returned can be another callable object (in most cases), or another class object, such as the property.

Problems

Decorators can make your code more elegant and help to reduce duplication, but there are some problems as well.

Wrong location

Let's look at the sample code directly.

def html_tags(tag_name):
    print 'begin outer function.'
    def wrapper_(func):
        print "begin of inner wrapper function."
        def wrapper(*args, **kwargs):
            content = func(*args, **kwargs)
            print "<{tag}>{content}</{tag}>".format(tag=tag_name, content=content)
        print 'end of inner wrapper function.'
        return wrapper
    print 'end of outer function'
    return wrapper_

@html_tags('b')
def hello(name='Toby'):
    return 'Hello {}!'.format(name)

hello()
hello()

I added a print statement at every possible location in the decorator to record the called situation. What's the result? If you are not sure, then it is best not to add logic outside the decorator function, otherwise the decorator will be out of your control. The following is the output:

begin outer function.
end of outer function
begin of inner wrapper function.
end of inner wrapper function.
<b>Hello Toby!</b>
<b>Hello Toby!</b>

Wrong function signature and documentation

The name of the function decorated by the decorator looks the same, but the function has changed.

def logging(func):
    def wrapper(*args, **kwargs):
        """print log before a function."""
        print "[DEBUG] {}: enter {}()".format(datetime.now(), func.__name__)
        return func(*args, **kwargs)
    return wrapper

@logging
def say(something):
    """say something"""
    print "say {}!".format(something)

print say.__name__  # wrapper

Here @ is equivalent to:

say = logging(say)

The name of the function that logging returns is wrapper. The above statement is to assign the result to say, so the __name__ of say is wrapper. In addition to name, other properties are also from wrapper, such as doc, source, etc.

You can use the functools.wraps that is in the standard library to solve this problem.

from functools import wraps

def logging(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        """print log before a function."""
        print "[DEBUG] {}: enter {}()".format(datetime.now(), func.__name__)
        return func(*args, **kwargs)
    return wrapper

@logging
def say(something):
    """say something"""
    print "say {}!".format(something)

print say.__name__  # say
print say.__doc__ # say something

The main problem has been solved, but it is still not perfect. Because the signature and source code of the function are still not available.

import inspect
print inspect.getargspec(say)  # failed
print inspect.getsource(say)  # failed

If you want to solve this problem completely, you can use a third-party package, such as wrapt. It will be introduced later.

Cannot decorate @staticmethod or @classmethod

When you use the decorator in a static method, the program will raise an exception.

class Car(object):
    def __init__(self, model):
        self.model = model

    @logging  # Decorate the instance method,OK
    def run(self):
        print "{} is running!".format(self.model)

    @logging  # Decorate the static method,Failed
    @staticmethod
    def check_model_for(obj):
        if isinstance(obj, Car):
            print "The model of your car is {}".format(obj.model)
        else:
            print "{} is not a car!".format(obj)

"""
Traceback (most recent call last):
...
  File "example_4.py", line 10, in logging
    @wraps(func)
  File "C:\Python27\lib\functools.py", line 33, in update_wrapper
    setattr(wrapper, attr, getattr(wrapped, attr))
AttributeError: 'staticmethod' object has no attribute '__module__'
"""

The @staticmethod decorator does not return a callable object, but returns a staticmethod object, so it does not meet the requirement of passing the callable object in. Thus you can't add another decorator to it. It's very simple to solve the problem: you just need to put your decorator before @staticmethod; your decorator returns a normal function, so it won't go wrong for your adding @staticmethod.

class Car(object):
    def __init__(self, model):
        self.model = model

    @staticmethod
    @logging  # Decorate before @staticmethod,OK
    def check_model_for(obj):
        pass

How To Optimize Your Decorator

Nested decorators are less intuitive, so we use third-party packages to make decorators more readable.

Decorator.py

Decorator.py is a very simple decorator enhancement package. You can define the wrapper() function intuitively, and then use the decorate(func, wrapper) method to implement a decorator.

from decorator import decorate

def wrapper(func, *args, **kwargs):
    """print log before a function."""
    print "[DEBUG] {}: enter {}()".format(datetime.now(), func.__name__)
    return func(*args, **kwargs)

def logging(func):
    return decorate(func, wrapper)  # Decorate func with wrapper.

You can also use the @decorator that comes with it to implement your decorator.

from decorator import decorator

@decorator
def logging(func, *args, **kwargs):
    print "[DEBUG] {}: enter {}()".format(datetime.now(), func.__name__)
    return func(*args, **kwargs)

The decorator implemented by decorator.py can retain the name, doc and args of the original function. The only problem is that the inspect.getsource(func) returns the source code of the decorator. So you need to change it to inspect.getsource(func.__wrapped__).

Wrapt

Wrapt is a very functional package for implementing a variety of decorators. If you use the decorators implemented with wrapt, you won't run into all of the problems that are encountered in the preceding inspect. It will handle these problems for you, and even inspect.getsource(func) becomes free of errors.

import wrapt

# without argument in decorator
@wrapt.decorator
def logging(wrapped, instance, args, kwargs):  # instance is must
    print "[DEBUG]: enter {}()".format(wrapped.__name__)
    return wrapped(*args, **kwargs)

@logging
def say(something): pass

In order to use wrapt, you only need to define a decorator function, but the function signature is fixed and must be (wrapped, instance, args, kwargs). Note that the second argument instance is required, even if you don't use it. When the decorator is decorated in different positions it will get different values, for example, you can get the class instance if the decorator is decorated in the class instance method. You can adjust your decorator more flexibly according to the value of instance. In addition, args and kwargs are also fixed. Note that there is no asterisk in front, and the asterisk is only used when calling the original function inside the decorator.

If you need to use wrapt to write a decorator with parameters, you can write like this:

def logging(level):
    @wrapt.decorator
    def wrapper(wrapped, instance, args, kwargs):
        print "[{}]: enter {}()".format(level, wrapped.__name__)
        return wrapped(*args, **kwargs)
    return wrapper

@logging(level="INFO")
def do(work): pass

It is recommended to check the official documentation for the use of wrapt.

http://wrapt.readthedocs.io/en/latest/quick-start.html

Summary

The decorator is to enhance the original function and object, which is equivalent to re-encapsulation, so in general, the decorator function is named as wrapper(). The function only plays its role when it is called. For example, the @logging decorator can also output logs when the function is executed, and the function decorated with @cache can cache calculation results.

There are still some things that are not mentioned, such as the decorator used to decorate classes. I'll fill it in later. Thanks for reading.

0 Comment

temp