Machinalis: A day with mypy: Part 1

What is it?

For those unfamiliar with mypy, I’ll add a short introduction here. If you have already been following it, you can skip this section. For finer details read the official documentation which is quite good as a tutorial. My goal here is to give you a quick taste of what’s going on.

mypy is a type checking tool for python code, so it can detect inconsistencies in your code; a typical example would be adding a str with an int, but perhaps more realistic examples of errors we make are passing a file name to a function that actually expected a file object, or mistyping an attribute name of an object, or setting an attribute to an int when it should a function returning an int.

If you’re familiar with other statically typed languages like C or Java, there are some important differences:

• The type check phase is separate from the compiling/running phase. It’s something that you run whenever you like, probably before delivery and while developing. In that sense it feels more like running tests.
• Something that is partly a consequence of the above is that typing errors are not something you need to “fix” before running a program. You can run your Python program as always (or invoke them as from an interactive interpreter, or a test suite) even if they have type errors, which can be useful to debug a problem or if there is something that you know it looks like a type error but you know it’s a situation that can never happen.
• Using types is optional per function/method and you can choose how much or how little to use it. You can add type information to a few functions at a time, and only those will be checked. This make easier to apply stricter type rules to places of the code where it makes sense, and leave those parts where the cost/benefit of adding types is low unchecked.
• Even when adding types you can opt out at more granular levels. You can say that the type of a variable is Any which means “everything I do with this variable is OK, don’t check it”, or you can silence type errors in a single line.

In summary, it’s essentially an opt-in mechanism, designed that way to not interfere with the things that Python allows you to do and you like.

Another difference with some statically typed languages, but not all is that it can use types even where there aren’t type declarations. Local variable and object attribute types are inferred from context (with some exceptions), so you only specify types for function parameters and returns. It reduces the amount of work to use mypy (and the syntactic overhead), and function signatures are the place where type annotations make more sense also as a self-documenting property of the code.

Getting started

To use it you can just install it with pip (you should do it inside a virtualenv/venv, which you probably are using anyway).

$ pip install mypy-lang

Note that you should install “mypy-lang”. There is a mypy package in pip which is a completely unrelated library. Once you have that installed, you can run it on one or many python files:

$ mypy somescript.py
$ mypy somescript.py somemodule.py
$ mypy somepackage/*.py

Note that you use paths to python files, not import paths. You should run mypy from the same place that you run your code to make sure it finds imports in the same way that your code does.

Running it as it is won’t do much checking; it will ignore all the function bodies and only infer some types and check the on the body of each module. You can get a few error messages for your module body, and also about imports it can’t find, or library modules that have no type information. When starting you probably don’t want a lot of warnings on imported modules so you can use the -s command line flag to avoid following and checking imported modules.

The interesting stuff happens when you start adding type information to your functions. The type information is done with function annotations (those tags on parameters and results that exist since 3.0 times), following some conventions established in PEP484 which use the now standard typing module (typing is standard in 3.5 but if you’re using an older one mypy provides its own backport). So you turn normal python function or method declarations like:

deftrailing_blank_lines(physical_line,lines,line_number,total_lines):

into:

fromtypingimportList,TupleSourcePosition=Tuple[int,int]CheckResult=Tuple[SourcePosition,str]deftrailing_blank_lines(physical_line:str,lines:List[str],line_number:int,total_lines:int)->CheckResult:# ...Function body goes  here...

Some things to notice above are:

• You can use built-in types like int and str in type declarations
• For container types you can use the typing types List and Tuple that allow you to also specify element types. For example Tuple[int,int] is the type of tuples with two integer components; List[str] is the type of lists with string elements.
• You can create auxiliary definitions (like SourcePosition and CheckResult) to make declarations easier to read. I could have declared the above example as a function returning Tuple[Tuple[int,int],str] but it’s harder to understand and less clear about purpose.

Once you’ve added annotations like those above, mypy can then detect many kinds of problems. A (non comprehensive) list of common errors that you can detect can be:

• If any code calls the above function with the wrong type/count/order of arguments, mypy will warn you about it.
• Inside the function body, misuses of the arguments (like doing foriinrange(physical_line) will be detected). Of course this ends up propagating to almost any value misuse inside the function, not necessarily an operation in the argument.
• The return types of the functions will be checked to match the declared type. If the function above has a return1,2,"foo" instead of return(1,2),"foo" you’ll get an error message after running mypy.

Working with libraries

The more function annotations you have, the more effective are the checks on function calls that mypy does. However, in most python code bases, a lot of your function calls will not be internal to your code base but to other library, possibly the python standard library or some other third party library you’re using.

In those cases it’s impractical and complicated to modify the third party code to add annotations. mypy provides an alternate mechanism that allows you to provide “stub files”, which are files with just a lot of type specifications for another module. Those files are written with python 3 syntax but with no function bodies (use ... instead), and saved with a .pyi extension. Those are called typesheds

mypy provides typesheds for the most important (but not all) python standard library modules and some popular third party libraries. As its usage increases that coverage will grow, and it’s possible that module providers will start providing their own typesheds.

Working with legacy python

If you still have to support python 2.x, there are ways to do it adding the type annotations in specially formatted comments. The documentation covers it well and it’s not the scope of this article so I won’t cover the details.

Diving in

With that general picture of what we’ll be seeing let’s start discussing my small project. I downloaded pycodestyle from its github repository. The pycodestyle tool is actually a single-file python script with about 2200 lines of code, 46 functions, 6 classes with 44 method definitions. There are other python files in the repo but related to installation and testing. It doesn’t depend on third party libraries, but uses many different components of the standard library (command-line parsing, regular expressions, the tokenizer, introspection via inspect)

After downloading the source, I created a virtualenv and installed mypy-lang with pip with no problems (at that point I used mypy 0.4.1; 0.4.2 is already available). Then I was able to run:

$ mypy pycodestyle.py

Which results in a few errors, but most of them uninteresting. The code is not annotated so mypy only checks the top level of the script. A lot of the errors are about imported library modules not having yet the “typesheds”, so you’ll see messages like:

pycodestyle.py:52:error:Nolibrarystubfileforstandardlibrarymodule'keyword'pycodestyle.py:60:error:Nolibrarystubfileforstandardlibrarymodule'optparse'...

At this point it was quite reasonable to use the “silent” flag to avoid messages about external code:

$ mypy -s pycodestyle.py

If you are doing this on a large project you can later remove the -s flag once you have some more typing information if you like, to make sure you haven’t missed large parts.

At this point I started iterating by doing different things:

• Fixing the errors that mypy was already detecting
• Finding some function definition(deffoo(...)) and adding types to it, which in turn helped mypy find more inconsistencies and report more issues.

After each of these changes, I re-ran mypy to make sure that I was moving in the right direction. Also, when I wanted to make sure that I wasn’t missing some function definitions that should be typed (or forgot to type one of the parameters) I used:

$ mypy -s --disallow-untyped-defs pycodestyle.py

which reported stuff like:

pycodestyle.py:note:Infunction"tabs_or_spaces":pycodestyle.py:134:error:Functionismissingatypeannotationpycodestyle.py:note:Infunction"tabs_obsolete":pycodestyle.py:153:error:Functionismissingatypeannotation

What’s next

This is the end of Part 1 of this article. On the next ones I’ll go through some of examples of where this process led me through, and the insight I gained regarding the tool. Finally, Part 3 will go fore some general conclusions. See you in my next post!

Useful resources

1. PEP-484: https://www.python.org/dev/peps/pep-0484/
2. The standard library typing module documentation: https://docs.python.org/3/library/typing.html
3. Official mypy documentation: http://mypy.readthedocs.io/en/latest/