Lintel Technologies: Python metaclasses explained

Python is having obscure wonderful mechanism of classes and it’s implementation.   In python every thing is an object. If you define a class, that class it self is an object in memory and it is an instance of other class. You may call it as a class object. If you instantiate it you will get brand new object called instance( instance of class).

Probably, meta classes  are  sort of confusing concept. Many people are afraid of this concept, but believe me it is very nice and simple if you understand it.

In simple words, meta classes are classes which are responsible to create  a class object( in memory).

As mentioned above, when you define a class. A class object will be created in memory but behind the senses it is an instance of other class by default type. So, classes are created by meta classes. You can specify your custom meta class which is responsible to create a class.

Mostly, meta classes are used when you write APIs or frameworks. Like django uses metaclasses in models.

I would like to explain this concept by taking real world example, to get you good understanding. Let’s take a look at following picture.

As show in picture, A factory can be treated a metaclass. Where, it produces vending machine, can be considered as a class. Where, the class (vending machine) produces instances, which are cans, bottles etc.

An example representing normal class definition and instance creation

>>> class Foo(object):
...     a = 3 
... 
>>> 
>>> Foo
<class '__main__.Foo'>
>>> 
>>> i = Foo()
>>> i
<__main__.Foo object at 0x110126910>
>>> 
>>> isinstance(i, Foo)
True 
>>> isinstance(Foo, type)
True

Python is having builtin function called isinstance. Using this function we can determine if any object is an instance of specified object.

In the above example we haven’t specified the metaclass. So, python would use default one. i.e type.

In python 2, we specify metaclass using magic method __metaclass__.
__metaclass__ can be specified either as a class attribute or global variable. When you specify it globally in module level, all classes will become instances of this metaclass. If you specify this as a class attribute, only that class will become instance of specified metaclass. In python3 this __meatclass__ attribute is removed in favour of using metaclass argument that you have to specify in class definition.

Implementation of simple class with metaclass in place

>>> class MetaClass(type):
...     pass 
... 
>>> 
>>> class Foo(object):
...    __metaclass__ = MetaClass
...    pass 
... 
>>> 
>>> i = Foo()
>>> 
>>> isinstance(i, Foo)
True
>>> isinstance(Foo, MetaClass)
True
>>> isinstance(MetaClass, type)
True
>>>

Here in this example, you can see class Foo is the instance of metaclass MetaClass. Thus, Foo gets created as an instance of MetaClass. And, i is an instance of class Foo. As you can see, we confirmed this relation using isinstance function.

There is no restriction over how many times one class can be used as a metaclass for other classes.

>>> class MetaClass(type):
...     pass
... 
>>> 
>>> class Foo(object):
...     __metaclass__ = MetaClass 
... 
>>> 
>>> class Bar(object):
...     __metaclass__ = MetaClass
... 
>>> 
>>> 
>>> i = Foo()
>>> j = Bar()
>>> 
>>> isinstance(i, Foo)
True
>>> isinstance(j, Bar)
True
>>> isinstance(Foo, MetaClass)
True
>>> isinstance(Bar, MetaClass)
True
>>>

In the above example,  the meta class MetaCalss is used in both Foo and Bar classes. Like wise you can use it in as many classes you want. If you want to apply it for all module level classes. You better use global __metaclass__ attribute. As you can see in the following example

"""
This is a simple module to demostrate global __metaclass__ attribute

"""

class MetaClass(type):
   pass


__metaclass__ = MetaClass   # This will affect all class in module

class Foo():
   pass


class Bar():
   pass


class NewStyleClass(object):
   pass 


print "is instance of Foo: MetaClass:: %s" % isinstance(Foo, MetaClass)
print "is instance of Bar: MetaClass:: %s" % isinstance(Bar, MetaClass)

print 
print "is instance of NewStyleClass: MetaClass:: %s" % isinstance(NewStyleClass, MetaClass)

OutPut:

python global_metaclass.py 
is instance of Foo: MetaClass:: True
is instance of Bar: MetaClass:: True

is instance of NewStyleClass: MetaClass:: False

This module level __metaclass__ magic variable doesn’t work on new style classes as show in above code.

Python Metaclasses Key Notes

• Metaclasses are callables
• Subclasses inherit the metaclass
• Restriction over multiple metaclasses in multiple inheritance

MetaClasses are callable

The MetaClass always need not be a class. You can use any callable as a metaclass. A Simple example demonstrates using callable (function) as a metaclass

>>> def metaClass(name, bases, d):
...     print "Creating  Class %s" % name
...     c = type(name, bases, d)
...     return c
... 
>>> 
>>> 
>>> class Foo(object):
...    pass 
... 
>>> 
>>> class Foo(object):
...    __metaclass__ = metaClass
... 
Creating  Class Foo
>>> 
>>> class Bar(object):
...     __metaclass__ = metaClass 
... 
Creating  Class Bar
>>> 
>>> Foo 
<class '__main__.Foo'>
>>> Bar
<class '__main__.Bar'>
>>>

If you are using callable as a metaclass class. It should have same signature(arguments) as type. That is,

type(name of the class, tuple of the parent class (for inheritance, can be empty), dictionary containing attributes names and values)

Following example is useless but explanatory. You can hack the class creation using metaclass. You can see below example, Foo became 3 instead of class object as we did return 3 from metaclass.

>>> def makeClass(name, bases, d):
...     print name, bases, d
...     return 3
... 
>>> 
>>> class Foo(object):
...     __metaclass__ = makeClass
...     
... 
>>>
>>> Foo 
3

Subclasses inherit the metaclass

Like all other attributes and methods subclasses inherit metaclass.

class M1(type):
    def __new__(meta, name, bases, atts):
         print "Meta Class M1 called for class " + name
         return super(M1, meta).__new__(meta, name, bases, atts)

class Base(object):
     __metaclass__ = M1


class Sub(Base):
     pass

OutPut:

python metaclass_inheritance.py 
Meta Class M1 called for class Base
Meta Class M1 called for class Sub

#TODO

Restriction over multiple metaclasses in inheritance

Classes can have multiple base classes. Those maetaclasses may have different metaclass. If so, every thing should be linear in the inheritance. That is, multi-level inheritance works fine. But, python is having restriction over having multiple metaclasses in multiple inheritance  from different bases. Following example shows that, raised exception because of having multiple metaclasses from two bases.

>>> class MetaClass1(type):
...     pass
...
>>> class MetaClass2(type):
...     pass
...
>>> class Base1(object):
...     __metaclass__ = MetaClass1
...
>>> class Base2(object):
...     __metaclass__ = MetaClass2
...
>>> class Foobar(Base1, Base2):
...     pass
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases

Where metaclasses used ?

The main use case for a metaclass is creating an API, best example of this is Django ORM.

If you are aware of django. We usually define a model as follows

class Person(models.Model):
    name = models.CharField(max_length=30)
    age = models.IntegerField()

But,

guy = Person(name='bob', age='35')
  print(guy.age)

It won’t return an IntegerField object. Instead it will return int.

This hack is actually possible by defining  __metaclass__ on models.Model and it uses some magic that will turn the model Person you defined with simple statements into a complex hook to a database field.

Django makes complex look simple by exposing a simple API and using metaclasses inside.

Metaclasses in python3

In python3 we specify metaclass by passing keyword argument metaclass to class definition. This is one of the difference between python2 and python3.

class Foo(base1, base2, metaclass=MetaClass):
         pass

Py3K translation tool can be used to convert old syntax to new syntax.

Summary

In python classes are actually instances of other classes (soft of)  called metaclasses. You can specify metaclass using magic method __metaclass__ either in class definition or gobally module level. These metaclasses are useful when you want customize class creation.

Conclusion

Metaclasses is the obscure feature in python. Use it, when you really need them.

Metaclasses are deeper magic that 99% of users should never worry about. If you wonder whether you need them, you don’t (the people who actually need them know with certainty that they need them, and don’t need an explanation about why).

Python Guru Tim Peters

Best real life example where metaclasses are being used is Django ORM. As like Django if you are writing framework or API, you can leverage metaclasses and you really need them.

Please share. Let me know your comments and queries " class="wp-smiley" style="height: 1em; max-height: 1em;" />" class="wp-smiley" style="height: 1em; max-height: 1em;" />

References:

[1] https://www.python.org/dev/peps/pep-3115/
[2]http://stackoverflow.com/questions/2149846/customary-to-inherit-metaclasses-from-type
[3]http://stackoverflow.com/questions/100003/what-is-a-metaclass-in-python

The post Python metaclasses explained appeared first on Lintel Technologies Blog.