I want to finish coverage.py 5.0. It has some big changes, so I need people to try it and tell me if it’s ready. Please install coverage.py 5.0 beta 1 and try it in your environment.

I especially want to hear from you if you tried the earlier alphas of 5.0. There have been some changes in the SQLite database that were needed to make measurement efficient enough for large test suites, but that hinder ad-hoc querying.

If you haven’t taken a look at coverage.py 5.0 yet, the big change is the addition of “contexts.” These can record not just that a line was executed, but something about why it was executed. Any number of contexts can be recorded for a line. They could be different operating systems, or versions of Python, or the name of the test that was running. I think it could enable some really interesting tooling.

If you are interested in recording test names as contexts, the pytest-cov pytest plugin now has a “--cov-context” option to do just that.

Contexts increase the data requirements, so data storage is now a SQLite file rather than a JSON file. The summary of what’s new in 5.0 is here: Major changes in 5.0.

Please try this. Soon 5.0 will be done, and people will begin installing it unknowingly. I would really like to minimize the turmoil when that happens.

In this course, you’ll learn about recursion. Recursion is a powerful tool you can use to solve a problem that can be broken down into smaller variations of itself. You can create very complex recursive algorithms with only a few lines of code.

You’ll cover:

• What recursion is
• How to define a recursive function
• How practical examples of recursive functions work
• How to maintain state
• How to optimize recursion

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

In order to include categorical features in your Machine Learning model, you have to encode them numerically using "dummy" or "one-hot" encoding. But how do you do this correctly using scikit-learn?

In this 28-minute video, you'll learn:

• How to use OneHotEncoder and ColumnTransformer to encode your categorical features and prepare your feature matrix in a single step
• How to include this step within a Pipeline so that you can cross-validate your model and preprocessing steps simultaneously
• Why you should use scikit-learn (rather than pandas) for preprocessing your dataset

If you want to follow along with the code, you can download the Jupyter notebook from GitHub.

Click on a timestamp below to jump to a particular section:

0:22 Why should you use a Pipeline?
2:30 Preview of the lesson
3:35 Loading and preparing a dataset
6:11 Cross-validating a simple model
10:00 Encoding categorical features with OneHotEncoder
15:01 Selecting columns for preprocessing with ColumnTransformer
19:00 Creating a two-step Pipeline
19:54 Cross-validating a Pipeline
21:44 Making predictions on new data
23:43 Recap of the lesson
24:50 Why should you use scikit-learn (rather than pandas) for preprocessing?

Related Resources

• scikit-learn documentation for OneHotEncoder, ColumnTransformer, and Pipeline
• My video series: Introduction to Machine Learning in Python
• My videos on cross-validation and grid search
• My lesson notebook on StandardScaler

P.S. Want to master Machine Learning in Python? Enroll in my online course, Machine Learning with Text in Python!

#394 – NOVEMBER 12, 2019
View in Browser »

Collaborative Python and R Notebooks Integrated With SQL. All in One Platform. Free Forever.

Mode Studio combines a SQL editor, Python & R notebooks, and visualization builder in one platform. Connect your data warehouse and analyze with your preferred language. Make custom viz (D3.js, HTML/CSS) or use out-of-the-box charts →
MODE ANALYTICSsponsor

Python Jobs

Senior Python/Django Developer (Eindhoven, Netherlands)

Sendcloud

Django Full Stack Web Developer (Austin, TX, USA)

Zeitcode

Full Stack Developer (Toronto, ON, Canada)

Beanfield Metroconnect

Sr. Security Software Engineer (Remote)

TER Consulting Group

Full Stack Software Developer (Remote)

CyberCoders

Scientific Software Engineer (Remote)

Incendia Partners

More Python Jobs >>>

Articles & Tutorials

Automated Python Code Reviews, Directly From Your Git Workflow

Codacy lets developers spend more time shipping code and less time fixing it. Set custom standards and automatically track quality measures like coverage, duplication, complexity and errors. Integrates with GitHub, GitLab and Bitbucket, and works with 28 different languages. Get started today for free →
CODACYsponsor

Projects & Code

Events

Happy Pythoning!
This was PyCoder’s Weekly Issue #394.
View in Browser »

[ Subscribe to 🐍 PyCoder’s Weekly 💌 – Get the best Python news, articles, and tutorials delivered to your inbox once a week >> Click here to learn more ]

Introduction

Following the previous article, Understanding OpenGL through Python where we've set the foundation for further learning, we can jump into OpenGL using PyGame and PyOpenGL.

PyOpenGL is the standardized library used as a bridge between Python and the OpenGL APIs, and PyGame is a standardized library used for making games in Python. It offers built-in handy graphical and audio libraries and we'll be using it to render the result more easily at the end of the article.

As mentioned in the previous article, OpenGL is very old so you won't find many tutorials online on how to properly use it and understand it because all of the top dogs are already knee-deep in new technologies.

In this article, we'll jump into several fundamental topics you'll need to know:

• Initializing a Project Using PyGame
• Drawing Objects
• Iterative Animation
• Utilizing Transformation Matrices
• Multiple Transformation Execution
• Implementation Example

Initializing a Project Using PyGame

First off, we need to install PyGame and PyOpenGL if you haven't already:

$ python3 -m pip install -U pygame --user
$ python3 -m pip install PyOpenGL PyOpenGL_accelerate

Note: You can find a more detailed installation in the previous OpenGL article.

If you have problems concerning the installation, PyGame's "Getting Started" section might be a good place to visit.

Since there's no point in unloading 3 books worth of graphics theory on you, we'll be using the PyGame library to give us a head start. It will essentially just shorten the process from project initialization to actual modeling and animating.

To start off, we need to import everything necessary from both OpenGL and PyGame:

import pygame as pg
from pygame.locals import *

from OpenGL.GL import *
from OpenGL.GLU import *

Next, we get to the initialization:

pg.init()
windowSize = (1920,1080)
pg.display.set_mode(display, DOUBLEBUF|OPENGL)

While the initialization is only three lines of code, each deserves at least a simple explanation:

• pg.init(): Initialization of all the PyGame modules - this function is a godsend
• windowSize = (1920, 1080): Defining a fixed window size
• pg.display.set_mode(display, DOUBLEBUF|OPENGL): Here, we specify that we'll be using OpenGL with double buffering

Double buffering means that there are two images at any given time - one that we can see and one that we can transform as we see fit. We get to see the actual change caused by the transformations when the two buffers swap.

Since we have our viewport set up, next we need to specify what we'll be seeing, or rather where the "camera" will be placed, and how far and wide it can see.

This is known as the frustum - which is just a cut off pyramid that visually represents the camera's sight (what it can and can't see).

A frustum is defined by 4 key parameters:

1. The FOV (Field of View): Angle in degrees
2. The Aspect Ratio: Defined as the ratio of the width and height
3. The z coordinate of the near Clipping Plane: The minimum draw distance
4. The z coordinate of the far Clipping Plane: The maximum draw distance

So, let's go ahead and implement the camera with these parameters in mind, using OpenGL C code:

void gluPerspective(GLdouble fovy, GLdouble aspect, GLdouble zNear, GLdouble zFar);
gluPerspective(60, (display[0]/display[1]), 0.1, 100.0)

To better understand how a frustum works, here's a reference picture:

Near and far planes are used for better performance. Realistically, rendering anything outside our field of vision is a waste of hardware performance that could be used rendering something that we can actually see.

So everything that the player can't see is implicitly stored in memory, even though it isn't visually present. Here's a great video of how rendering only within the frustum looks like.

Drawing Objects

After this setup, I imagine we're asking ourselves the same question:

Well this is all fine and dandy, but how do I make a Super Star Destroyer?

Well... with dots. Every model in OpenGL object is stored as a set of vertices and a set of their relations (which vertices are connected). So theoretically if you knew the position of every single dot that is used to draw a Super Star Destroyer, you could very well draw one!

There are a few ways we can model objects in OpenGL:

1. Drawing using vertices, and depending on how OpenGL interprets these vertices, we can draw with:
   • points: as in literal points that are not connected in any way
   • lines: every pair of vertices constructs a connected line
   • triangles: every three vertices make a triangle
   • quadrilateral: every four vertices make a quadrilateral
   • polygon: you get the point
   • many more...
2. Drawing using the built in shapes and objects that were painstakingly modeled by OpenGL contributors
3. Importing fully modeled objects

So, to draw a cube for example, we first need to define its vertices:

cubeVertices = ((1,1,1),(1,1,-1),(1,-1,-1),(1,-1,1),(-1,1,1),(-1,-1,-1),(-1,-1,1),(-1, 1,-1))

Then, we need to define how they're all connected. If we want to make a wire cube, we need to define the cube's edges:

cubeEdges = ((0,1),(0,3),(0,4),(1,2),(1,7),(2,5),(2,3),(3,6),(4,6),(4,7),(5,6),(5,7))

This is pretty intuitive - the point 0 has an edge with 1, 3, and 4. The point 1 has an edge with points 3, 5, and 7, and so on.

And if we want to make a solid cube, then we need to define the cube's quadrilaterals:

cubeQuads = ((0,3,6,4),(2,5,6,3),(1,2,5,7),(1,0,4,7),(7,4,6,5),(2,3,0,1))

This is also intuitive - to make a quadrilateral on the top side of the cube, we'd want to "color" everything in-between the points 0, 3, 6, and 4.

Keep in mind there's an actual reason we label the vertices as indexes of the array they're defined in. This makes writing code that connects them very easy.

The following function is used to draw a wired cube:

def wireCube():
    glBegin(GL_LINES)
    for cubeEdge in cubeEdges:
        for cubeVertex in cubeEdge:
            glVertex3fv(cubeVertices[cubeVertex])
    glEnd()

glBegin() is a function that indicates we'll defining the vertices of a primitive in the code below. When we're done defining the primitive, we use the function glEnd().

GL_LINES is a macro that indicates we'll be drawing lines.

glVertex3fv() is a function that defines a vertex in space, there are a few versions of this function, so for the sake of clarity let's look at how the names are constructed:

• glVertex: a function that defines a vertex
• glVertex3: a function that defines a vertex using 3 coordinates
• glVertex3f: a function that defines a vertex using 3 coordinates of type GLfloat
• glVertex3fv: a function that defines a vertex using 3 coordinates of type GLfloat which are put inside a vector (tuple) (the alternative would be glVertex3fl which uses a list of arguments instead of a vector)

Following similar logic, the following function is used to draw a solid cube:

def solidCube():
    glBegin(GL_QUADS)
    for cubeQuad in cubeQuads:
        for cubeVertex in cubeQuad:
            glVertex3fv(cubeVertices[cubeVertex])
    glEnd()

Iterative Animation

For our program to be "killable" we need to insert the following code snippet:

for event in pg.event.get():
    if event.type == pg.QUIT:
        pg.quit()
        quit()

It's basically just a listener that scrolls through PyGame's events, and if it detects that we clicked the "kill window" button, it quits the application.

We'll cover more of PyGame's events in a future article - this one was introduced right away because it would be quite uncomfortable for users and yourselves to have to fire up the task manager every time they want to quit the application.

In this example, we'll be using double buffering, which just means that we'll be using two buffers (you can think of them as canvases for drawing) which will swap in fixed intervals and give the illusion of motion.

Knowing this, our code has to have the following pattern:

handleEvents()
glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)
doTransformationsAndDrawing()
pg.display.flip()
pg.time.wait(1)

• glClear: Function that clears the specified buffers (canvases), in this case, the color buffer (which contains color information for drawing the generated objects) and depth buffer (a buffer which stores in-front-of or in-back-of relations of all the generated objects).
• pg.display.flip(): Function that updated the window with the active buffer contents
• pg.time.wait(1): Function that pauses the program for a period of time

glClear has to be used because if we don't use it, we'll be just painting over an already painted canvas, which in this case, is our screen and we're going to end up with a mess.

Next, if we want to continuously update our screen, just like an animation, we have to put all our code inside a while loop in which we:

1. Handle events (in this case, just quitting)
2. Clear the color and depth buffers so that they can be drawn on again
3. Transform and draw objects
4. Update the screen
5. GOTO 1.

The code ought to look something like this:

while True:
    handleEvents()
    glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)
    doTransformationsAndDrawing()
    pg.display.flip()
    pg.time.wait(1)

Utilizing Transformation Matrices

In the previous article, we explained how, in theory, we need to construct a transformation that has a referral point.

OpenGL works the same way, as can be seen in the following code:

glTranslatef(1,1,1)
glRotatef(30,0,0,1)
glTranslatef(-1,-1,-1)

In this example, we did a z-axis rotation in the xy-plane with the center of rotation being (1,1,1) by 30 degrees.

Let's have a little refresher if these terms sound a bit confusing:

1. z-axis rotation means that we're rotating around the z-axis

   This just means we're approximating a 2D plane with a 3D space, this whole transformation is basically like doing a normal rotation around a referral point in 2D space.

2. We get the xy-plane by squashing an entire 3D space into a plane that has z=0 (we eliminate the z parameter in every way)
3. Center of rotation is a vertex around which we will be rotating a given object (the default center of rotation is the origin vertex (0,0,0))

But there's a catch - OpenGL understands the code above by constantly remembering and modifying one global transformation matrix.

So when you write something in OpenGL, what you're saying is:

# This part of the code is not translated
# transformation matrix = E (neutral)
glTranslatef(1,1,1)
# transformation matrix = TxE
# ALL OBJECTS FROM NOW ON ARE TRANSLATED BY (1,1,1)

As you might imagine, this poses a huge problem, because sometimes we want to utilize a transformation on a single object, not on the whole source code. This is a very common reason for bugs in low-level OpenGL.

To combat this problematic feature of OpenGL, we're presented with pushing and popping transformation matrices - glPushMatrix() and glPopMatrix():

# Transformation matrix is T1 before this block of code
glPushMatrix()
glTranslatef(1,0,0)
generateObject() # This object is translated
glPopMatrix()
generateSecondObject() # This object isn't translated

These work in a simple Last-in-First-Out (LIFO) principle. When we wish to perform a translation to a matrix, we first duplicate it and then push it on top of the stack of the transformation matrices.

In other words, it isolates all the transformations we're performing in this block by creating a local matrix that we can scrap after we're done.

Once the object is translated, we pop the transformation matrix from the stack, leaving the rest of the matrices untouched.

Multiple Transformation Execution

In OpenGL, as previously mentioned, transformations are added to the active transformation matrix that's on top of stack of transformation matrices.

This means that the transformations are executed in reverse order. For example:

######### First example ##########
glTranslatef(-1,0,0)
glRotatef(30,0,0,1)
drawObject1()
##################################

######## Second Example #########
glRotatef(30,0,0,1)
glTranslatef(-1,0,0)
drawObject2()
#################################

In this example, Object1 is first rotated, then translated, and Object2 is first translated, and then rotated. The last two concepts won't be used in the implementation example, but will be practically used in the next article in the series.

Implementation Example

The code below draws a solid cube on the screen and continuously rotates it by 1 degree around the (1,1,1) vector. And it can be very easily modified to draw a wire cube by swapping out the cubeQuads with the cubeEdges:

import pygame as pg
from pygame.locals import *

from OpenGL.GL import *
from OpenGL.GLU import *

cubeVertices = ((1,1,1),(1,1,-1),(1,-1,-1),(1,-1,1),(-1,1,1),(-1,-1,-1),(-1,-1,1),(-1,1,-1))
cubeEdges = ((0,1),(0,3),(0,4),(1,2),(1,7),(2,5),(2,3),(3,6),(4,6),(4,7),(5,6),(5,7))
cubeQuads = ((0,3,6,4),(2,5,6,3),(1,2,5,7),(1,0,4,7),(7,4,6,5),(2,3,0,1))

def wireCube():
    glBegin(GL_LINES)
    for cubeEdge in cubeEdges:
        for cubeVertex in cubeEdge:
            glVertex3fv(cubeVertices[cubeVertex])
    glEnd()

def solidCube():
    glBegin(GL_QUADS)
    for cubeQuad in cubeQuads:
        for cubeVertex in cubeQuad:
            glVertex3fv(cubeVertices[cubeVertex])
    glEnd()

def main():
    pg.init()
    display = (1680, 1050)
    pg.display.set_mode(display, DOUBLEBUF|OPENGL)

    gluPerspective(45, (display[0]/display[1]), 0.1, 50.0)

    glTranslatef(0.0, 0.0, -5)

    while True:
        for event in pg.event.get():
            if event.type == pg.QUIT:
                pg.quit()
                quit()

        glRotatef(1, 1, 1, 1)
        glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFFER_BIT)
        solidCube()
        #wireCube()
        pg.display.flip()
        pg.time.wait(10)

if __name__ == "__main__":
    main()

Running this piece of code, a PyGame window will pop up, rendering the cube animation:

Conclusion

There is a lot more to learn about OpenGL - lighting, textures, advanced surface modeling, composite modular animation, and much more.

But fret not, all of this will be explained in the following articles teaching the public about OpenGL the proper way, from the ground up.

And don't worry, in the next article, we'll actually draw something semi-decent.

Seeking developers for paid contract on pip; apply by Nov. 22

One is that I helped the Packaging Working Group of the Python Software Foundation get funding for a long-needed improvement to pip. I led the writing of a few proposals -- grantwriting, to oversimplify -- and, starting possibly as soon as next month, contractors will start work. As Dustin Ingram explains:

Big news: the Python Packaging Working Group has secured >$400K in grants from multiple funders (TBA) to improve one of the most fundamental parts of pip: its dependency resolver. https://pyfound.blogspot.com/2019/11/seeking-developers-for-paid-contract.html

The dependency resolver is the algorithm which takes multiple constrained requirements (e.g. "some_package>=1.0,=2.0") and finds a version of all dependencies (and sub-dependencies) which satisfy all the constraints.
https://pip.pypa.io/en/stable/user_guide/#requirements-files

Right now, pip's resolver mostly works for most use cases... However the algorithm it uses is naïve, and isn't always guaranteed to produce an optimal (or correct) result.

.....

These funds will pay multiple developers to work on completing the design, implementation and rollout of this new dependency resolver for pip, finally closing issue #988.

Not only will this give pip a better resolver, but it will "enable us to untangle pip’s internals from the resolver, enabling pip to share code for dependency resolution with other packaging tooling". https://pradyunsg.me/blog/2019/06/23/oss-update-1/

This is great news for pip and Python packaging in general. Huge shout out to @pradyunsg for his existing work on the resolver issue and guidance here, and to @brainwane for all her tireless work acquiring and directing funding for Python projects.

If you or your organization is interested in participating in this project, we've just posted the RFP, which includes instructions for submitting proposals, evaluation criteria and scope of work.
https://github.com/python/request-for/blob/master/2020-pip/RFP.md

If you're interested, please apply by 22 November.

NYU, Secure Systems Lab, and my new title

In further news: New York University's Tandon School of Engineering has now announced that:

...pioneering open-source software nonprofits the Tor Project and Python Software Foundation (PSF) are the newest tenants at 370 Jay Street, a recently renovated addition to the University’s engineering and applied sciences programs in Downtown Brooklyn. NYU Tandon is donating work space to both organizations for their first offices in New York City. ....

Sumana Harihareswara, a volunteer with the PSF’s Packaging Working Group and a contracted project manager for the Python Packaging Index, is among the first to move into the NYU facility. Harihareswara is also a visiting scholar in [Professor Justin] Cappos's Secure Systems Lab.

Yes, I am now a Visiting Scholar at NYU's Secure Systems Lab and I get to use an office with a door, shelves, whiteboards, and so on (per the picture at right). If you contribute to Python packaging/distribution tools and live in/near or sometimes visit New York City, let me know and perhaps we could cowork a bit?

The Secure Systems Lab stewards The Update Framework (TUF) and related projects, and works to improve the security of the software supply chain. The Python Package Index is likely going to implement TUF to add cryptographic signatures to packages on PyPI, and so I've gotten to give TUF's developers some advice to help that work move along. (I won't be the manager on that project but I'll be watching with great interest.)

PSF projects

I'm grateful to get to help connect the Python Software Foundation with more resources and volunteers. Changeset's current and recent projects have mostly been for the PSF. Last month we finished accessibility, security, and internationalization work on PyPI that was funded by the Open Technology Fund, and Changeset's work on communicating about the sunsetting of Python 2.x continues and will go through April 2020.

Availability for one-day engagements in San Francisco in February

But I am interested in taking on new clients for short engagements starting in February 2020. In particular, I will be in the San Francisco Bay Area in mid- to late February. If you're in SF or nearby, I could offer you a one-day engagement doing one of the following:

• developing a contributor outreach/intake strategy
• researching potential funders and writing a rough draft of a grant proposal
• auditing and improving your developer onboarding documents

I'd spend a little time talking with you, then sit in your office and finish the document before leaving that afternoon. (Photo at right provides a sample of how I look while sitting.) Drop me a line for a free initial 30-minute chat and we can talk pricing.

If you’ve recently downloaded Python onto your computer, then you may have noticed a new program on your machine called IDLE. You might be wondering, “What is this program doing on my computer? I didn’t download that!” While you may not have downloaded this program on your own, IDLE comes bundled with every Python installation. It’s there to help you get started with the language right out of the box. In this tutorial, you’ll learn how to work in Python IDLE and a few cool tricks you can use on your Python journey!

In this tutorial, you’ll learn:

• What Python IDLE is
• How to interact with Python directly using IDLE
• How to edit, execute, and debug Python files with IDLE
• How to customize Python IDLE to your liking

What Is Python IDLE?

Every Python installation comes with an Integrated Development and Learning Environment, which you’ll see shortened to IDLE or even IDE. These are a class of applications that help you write code more efficiently. While there are many IDEs for you to choose from, Python IDLE is very bare-bones, which makes it the perfect tool for a beginning programmer.

Python IDLE comes included in Python installations on Windows and Mac. If you’re a Linux user, then you should be able to find and download Python IDLE using your package manager. Once you’ve installed it, you can then use Python IDLE as an interactive interpreter or as a file editor.

An Interactive Interpreter

The best place to experiment with Python code is in the interactive interpreter, otherwise known as a shell. The shell is a basic Read-Eval-Print Loop (REPL). It reads a Python statement, evaluates the result of that statement, and then prints the result on the screen. Then, it loops back to read the next statement.

The Python shell is an excellent place to experiment with small code snippets. You can access it through the terminal or command line app on your machine. You can simplify your workflow with Python IDLE, which will immediately start a Python shell when you open it.

A File Editor

Every programmer needs to be able to edit and save text files. Python programs are files with the .py extension that contain lines of Python code. Python IDLE gives you the ability to create and edit these files with ease.

Python IDLE also provides several useful features that you’ll see in professional IDEs, like basic syntax highlighting, code completion, and auto-indentation. Professional IDEs are more robust pieces of software and they have a steep learning curve. If you’re just beginning your Python programming journey, then Python IDLE is a great alternative!

How to Use the Python IDLE Shell

The shell is the default mode of operation for Python IDLE. When you click on the icon to open the program, the shell is the first thing that you see:

This is a blank Python interpreter window. You can use it to start interacting with Python immediately. You can test it out with a short line of code:

Here, you used print() to output the string "Hello, from IDLE!" to your screen. This is the most basic way to interact with Python IDLE. You type in commands one at a time and Python responds with the result of each command.

Next, take a look at the menu bar. You’ll see a few options for using the shell:

You can restart the shell from this menu. If you select that option, then you’ll clear the state of the shell. It will act as though you’ve started a fresh instance of Python IDLE. The shell will forget about everything from its previous state:

In the image above, you first declare a variable, x = 5. When you call print(x), the shell shows the correct output, which is the number 5. However, when you restart the shell and try to call print(x) again, you can see that the shell prints a traceback. This is an error message that says the variable x is not defined. The shell has forgotten about everything that came before it was restarted.

You can also interrupt the execution of the shell from this menu. This will stop any program or statement that’s running in the shell at the time of interruption. Take a look at what happens when you send a keyboard interrupt to the shell:

A KeyboardInterrupt error message is displayed in red text at the bottom of your window. The program received the interrupt and has stopped executing.

How to Work With Python Files

Python IDLE offers a full-fledged file editor, which gives you the ability to write and execute Python programs from within this program. The built-in file editor also includes several features, like code completion and automatic indentation, that will speed up your coding workflow. First, let’s take a look at how to write and execute programs in Python IDLE.

Opening a File

To start a new Python file, select File → New File from the menu bar. This will open a blank file in the editor, like this:

From this window, you can write a brand new Python file. You can also open an existing Python file by selecting File → Open… in the menu bar. This will bring up your operating system’s file browser. Then, you can find the Python file you want to open.

If you’re interested in reading the source code for a Python module, then you can select File → Path Browser. This will let you view the modules that Python IDLE can see. When you double click on one, the file editor will open up and you’ll be able to read it.

The content of this window will be the same as the paths that are returned when you call sys.path. If you know the name of a specific module you want to view, then you can select File → Module Browser and type in the name of the module in the box that appears.

Editing a File

Once you’ve opened a file in Python IDLE, you can then make changes to it. When you’re ready to edit a file, you’ll see something like this:

The contents of your file are displayed in the open window. The bar along the top of the window contains three pieces of important information:

1. The name of the file that you’re editing
2. The full path to the folder where you can find this file on your computer
3. The version of Python that IDLE is using

In the image above, you’re editing the file myFile.py, which is located in the Documents folder. The Python version is 3.7.1, which you can see in parentheses.

There are also two numbers in the bottom right corner of the window:

1. Ln: shows the line number that your cursor is on.
2. Col: shows the column number that your cursor is on.

It’s useful to see these numbers so that you can find errors more quickly. They also help you make sure that you’re staying within a certain line width.

There are a few visual cues in this window that will help you remember to save your work. If you look closely, then you’ll see that Python IDLE uses asterisks to let you know that your file has unsaved changes:

The file name shown in the top of the IDLE window is surrounded by asterisks. This means that there are unsaved changes in your editor. You can save these changes with your system’s standard keyboard shortcut, or you can select File → Save from the menu bar. Make sure that you save your file with the .py extension so that syntax highlighting will be enabled.

Executing a File

When you want to execute a file that you’ve created in IDLE, you should first make sure that it’s saved. Remember, you can see if your file is properly saved by looking for asterisks around the filename at the top of the file editor window. Don’t worry if you forget, though! Python IDLE will remind you to save whenever you attempt to execute an unsaved file.

To execute a file in IDLE, simply press the F5 key on your keyboard. You can also select Run → Run Module from the menu bar. Either option will restart the Python interpreter and then run the code that you’ve written with a fresh interpreter. The process is the same as when you run python3 -i [filename] in your terminal.

When your code is done executing, the interpreter will know everything about your code, including any global variables, functions, and classes. This makes Python IDLE a great place to inspect your data if something goes wrong. If you ever need to interrupt the execution of your program, then you can press Ctrl+C in the interpreter that’s running your code.

How to Improve Your Workflow

Now that you’ve seen how to write, edit, and execute files in Python IDLE, it’s time to speed up your workflow! The Python IDLE editor offers a few features that you’ll see in most professional IDEs to help you code faster. These features include automatic indentation, code completion and call tips, and code context.

Automatic Indentation

IDLE will automatically indent your code when it needs to start a new block. This usually happens after you type a colon (:). When you hit the enter key after the colon, your cursor will automatically move over a certain number of spaces and begin a new code block.

You can configure how many spaces the cursor will move in the settings, but the default is the standard four spaces. The developers of Python agreed on a standard style for well-written Python code, and this includes rules on indentation, whitespace, and more. This standard style was formalized and is now known as PEP 8. To learn more about it, check out How to Write Beautiful Python Code With PEP 8.

Code Completion and Call Tips

When you’re writing code for a large project or a complicated problem, you can spend a lot of time just typing out all of the code you need. Code completion helps you save typing time by trying to finish your code for you. Python IDLE has basic code completion functionality. It can only autocomplete the names of functions and classes. To use autocompletion in the editor, just press the tab key after a sequence of text.

Python IDLE will also provide call tips. A call tip is like a hint for a certain part of your code to help you remember what that element needs. After you type the left parenthesis to begin a function call, a call tip will appear if you don’t type anything for a few seconds. For example, if you can’t quite remember how to append to a list, then you can pause after the opening parenthesis to bring up the call tip:

The call tip will display as a popup note, reminding you how to append to a list. Call tips like these provide useful information as you’re writing code.

Code Context

The code context functionality is a neat feature of the Python IDLE file editor. It will show you the scope of a function, class, loop, or other construct. This is particularly useful when you’re scrolling through a lengthy file and need to keep track of where you are while reviewing code in the editor.

To turn it on, select Options → Code Context in the menu bar. You’ll see a gray bar appear at the top of the editor window:

As you scroll down through your code, the context that contains each line of code will stay inside of this gray bar. This means that the print() functions you see in the image above are a part of a main function. When you reach a line that’s outside the scope of this function, the bar will disappear.

How to Debug in IDLE

A bug is an unexpected problem in your program. They can appear in many forms, and some are more difficult to fix than others. Some bugs are tricky enough that you won’t be able to catch them by just reading through your program. Luckily, Python IDLE provides some basic tools that will help you debug your programs with ease!

Interpreter DEBUG Mode

If you want to run your code with the built-in debugger, then you’ll need to turn this feature on. To do so, select Debug → Debugger from the Python IDLE menu bar. In the interpreter, you should see [DEBUG ON] appear just before the prompt (>>>), which means the interpreter is ready and waiting.

When you execute your Python file, the debugger window will appear:

In this window, you can inspect the values of your local and global variables as your code executes. This gives you insight into how your data is being manipulated as your code runs.

You can also click the following buttons to move through your code:

• Go: Press this to advance execution to the next breakpoint. You’ll learn about these in the next section.
• Step: Press this to execute the current line and go to the next one.
• Over: If the current line of code contains a function call, then press this to step over that function. In other words, execute that function and go to the next line, but don’t pause while executing the function (unless there is a breakpoint).
• Out: If the current line of code is in a function, then press this to step out of this function. In other words, continue the execution of this function until you return from it.

Be careful, because there is no reverse button! You can only step forward in time through your program’s execution.

You’ll also see four checkboxes in the debug window:

1. Globals: your program’s global information
2. Locals: your program’s local information during execution
3. Stack: the functions that run during execution
4. Source: your file in the IDLE editor

When you select one of these, you’ll see the relevant information in your debug window.

Breakpoints

A breakpoint is a line of code that you’ve identified as a place where the interpreter should pause while running your code. They will only work when DEBUG mode is turned on, so make sure that you’ve done that first.

To set a breakpoint, right-click on the line of code that you wish to pause. This will highlight the line of code in yellow as a visual indication of a set breakpoint. You can set as many breakpoints in your code as you like. To undo a breakpoint, right-click the same line again and select Clear Breakpoint.

Once you’ve set your breakpoints and turned on DEBUG mode, you can run your code as you would normally. The debugger window will pop up, and you can start stepping through your code manually.

Errors and Exceptions

When you see an error reported to you in the interpreter, Python IDLE lets you jump right to the offending file or line from the menu bar. All you have to do is highlight the reported line number or file name with your cursor and select Debug → Go to file/line from the menu bar. This is will open up the offending file and take you to the line that contains the error. This feature works regardless of whether or not DEBUG mode is turned on.

Python IDLE also provides a tool called a stack viewer. You can access it under the Debug option in the menu bar. This tool will show you the traceback of an error as it appears on the stack of the last error or exception that Python IDLE encountered while running your code. When an unexpected or interesting error occurs, you might find it helpful to take a look at the stack. Otherwise, this feature can be difficult to parse and likely won’t be useful to you unless you’re writing very complicated code.

How to Customize Python IDLE

There are many ways that you can give Python IDLE a visual style that suits you. The default look and feel is based on the colors in the Python logo. If you don’t like how anything looks, then you can almost always change it.

To access the customization window, select Options → Configure IDLE from the menu bar. To preview the result of a change you want to make, press Apply. When you’re done customizing Python IDLE, press OK to save all of your changes. If you don’t want to save your changes, then simply press Cancel.

There are 5 areas of Python IDLE that you can customize:

1. Fonts/Tabs
2. Highlights
3. Keys
4. General
5. Extensions

Let’s take a look at each of them now.

Fonts/Tabs

The first tab allows you to change things like font color, font size, and font style. You can change the font to almost any style you like, depending on what’s available for your operating system. The font settings window looks like this:

You can use the scrolling window to select which font you prefer. (I recommend you select a fixed-width font like Courier New.) Pick a font size that’s large enough for you to see well. You can also click the checkbox next to Bold to toggle whether or not all text appears in bold.

This window will also let you change how many spaces are used for each indentation level. By default, this will be set to the PEP 8 standard of four spaces. You can change this to make the width of your code more or less spread out to your liking.

Highlights

The second customization tab will let you change highlights. Syntax highlighting is an important feature of any IDE that highlights the syntax of the language that you’re working in. This helps you visually distinguish between the different Python constructs and the data used in your code.

Python IDLE allows you to fully customize the appearance of your Python code. It comes pre-installed with three different highlight themes:

1. IDLE Day
2. IDLE Night
3. IDLE New

You can select from these pre-installed themes or create your own custom theme right in this window:

Unfortunately, IDLE does not allow you to install custom themes from a file. You have to create customs theme from this window. To do so, you can simply start changing the colors for different items. Select an item, and then press Choose color for. You’ll be brought to a color picker, where you can select the exact color that you want to use.

You’ll then be prompted to save this theme as a new custom theme, and you can enter a name of your choosing. You can then continue changing the colors of different items if you’d like. Remember to press Apply to see your changes in action!

Keys

The third customization tab lets you map different key presses to actions, also known as keyboard shortcuts. These are a vital component of your productivity whenever you use an IDE. You can either come up with your own keyboard shortcuts, or you can use the ones that come with IDLE. The pre-installed shortcuts are a good place to start:

The keyboard shortcuts are listed in alphabetical order by action. They’re listed in the format Action - Shortcut, where Action is what will happen when you press the key combination in Shortcut. If you want to use a built-in key set, then select a mapping that matches your operating system. Pay close attention to the different keys and make sure your keyboard has them!

Creating Your Own Shortcuts

The customization of the keyboard shortcuts is very similar to the customization of syntax highlighting colors. Unfortunately, IDLE does not allow you to install custom keyboard shortcuts from a file. You must create a custom set of shortcuts from the Keys tab.

Select one pair from the list and press Get New Keys for Selection. A new window will pop up:

Here, you can use the checkboxes and scrolling menu to select the combination of keys that you want to use for this shortcut. You can select Advanced Key Binding Entry >> to manually type in a command. Note that this cannot pick up the keys you press. You have to literally type in the command as you see it displayed to you in the list of shortcuts.

General

The fourth tab of the customization window is a place for small, general changes. The general settings tab looks like this:

Here, you can customize things like the window size and whether the shell or the file editor opens first when you start Python IDLE. Most of the things in this window are not that exciting to change, so you probably won’t need to fiddle with them much.

Extensions

The fifth tab of the customization window lets you add extensions to Python IDLE. Extensions allow you to add new, awesome features to the editor and the interpreter window. You can download them from the internet and install them to right into Python IDLE.

To view what extensions are installed, select Options → Configure IDLE -> Extensions. There are many extensions available on the internet for you to read more about. Find the ones you like and add them to Python IDLE!

Conclusion

In this tutorial, you’ve learned all the basics of using IDLE to write Python programs. You know what Python IDLE is and how you can use it to interact with Python directly. You’ve also learned how to work with Python files and customize Python IDLE to your liking.

You’ve learned how to:

• Work with the Python IDLE shell
• Use Python IDLE as a file editor
• Improve your workflow with features to help you code faster
• Debug your code and view errors and exceptions
• Customize Python IDLE to your liking

Now you’re armed with a new tool that will let you productively write Pythonic code and save you countless hours down the road. Happy programming!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Another quick matplotlib tip today: specifically, how easily specify colours from the standard matplotlib colour cycle.

A while back, when matplotlib overhauled their themes and colour schemes, they changed the default cycle of colours used for lines in matplotlib. Previously the first line was pure blue (color='b' in matplotlib syntax), then red, then green etc. They, very sensibly, changed this to a far nicer selection of colours.

However, this change made one thing a bit more difficult – as I found recently. I had plotted a couple of simple lines:

x_values = [0, 1, 2]
line1 = np.array([10, 20, 30])
line2 = line1[::-1]

plt.plot(x_values, line1)
plt.plot(x_values, line2)

which gives

I then wanted to plot a shaded area around the second line (the yellow one) – for example, to show the uncertainty in that line.

You can do this with the plt.fill_between function, like this:

plt.fill_between(x_values, line2 - 5, line2 + 5, alpha=0.3, color='y')

This produces a shaded line which extends from 5 below the line to 5 above the line:

Unfortunately the colours don’t look quite right: the line isn’t yellow, so doing a partially-transparent yellow background doesn’t look quite right.

I spent a while looking into how to extract the colour of the line so I could use this for the shading, before finding a really easy way to do it. To get the colours in the default colour cycle you can simply use the strings 'C0', 'C1', 'C2' etc. So, in this case just

plt.fill_between(x_values, line2 - 5, line2 + 5, alpha=0.3, color='C1')

The result looks far better now the colours match:

I found out about this from a wonderful graphical matplotlib cheatsheet created by Nicolas Rougier – I’d strongly suggest you check it out, there are all sorts of useful things on there that I never knew about!

Just in case you need to do this the manual way, then there are two fairly straightforward ways to get the colour of the second line.

The first is to get the default colour cycle from the matplotlib settings, and extract the relevant colour:

cycle_colors = plt.rcParams['axes.prop_cycle'].by_key()['color']

which gives a list of colours like this:

['#1f77b4', '#ff7f0e', '#2ca02c', '#d62728', ...]

You can then just use one of these colours in the call to plt.fill_between– for example:

plt.fill_between(x_values, line2 - 5, line2 + 5, alpha=0.3, color=cycle_colors[1])

The other way is to actually extract the colour of the actual line you plotted, and then use that for the plt.fill_between call

x_values = [0, 1, 2]
line1 = np.array([10, 20, 30])
line2 = line1[::-1]

plt.plot(x_values, line1)
plotted_line = plt.plot(x_values, line2)
plt.fill_between(x_values, line2 - 5, line2 + 5, alpha=0.3, color=plotted_line[0].get_color())

Here we save the result of the plt.plot call when we plot the second line. This gives us a list of the Line2D objects that were created, and we then extract the first (and only) element and call the get_color() method to extract the colour.

I do freelance work in data science and data visualisation – including using matplotlib. If you’d like to work with me, have a look at my freelance website or email me.

The post Tutorial: How to Read Stata Files in Python with Pandas appeared first on Erik Marsja.

In this post, we are going to learn how to read Stata (.dta) files in Python.

As previously described (in the read .sav files in Python post) Python is a general-purpose language that also can be used for doing data analysis and data visualization. One example of data visualization will be found in this post.

One potential downside, however, is that Python is not really user-friendly for data storage. This has, of course, lead to that our data many times are stored using Excel, SPSS, SAS, or similar software. See, for instance, the posts about reading .sav, and sas files in Python:

• How to read and write SPSS files in Python
• How to read SAS files in Pandas

Can I Open a Stata File in Python?

We are soon going to practically answer how to open a Stata file in Python? In Python, there are two useful packages called Pyreadstat, and Pandas that enable us to open .dta files. If we are working with Pandas, the  read_stata method will help us import a .dta into a Pandas dataframe. Furthermore, the package Pyreadstat, which is dependent on Pandas, will also create a Pandas dataframe from a .dta file.

How to install Pyreadstat:

First, before learning how to read .dat files using Python and Pyreadstat we need to install it. As many Python packages this package can be installed using pip or conda:

1. Install Pyreadstat using pip:
   Open up the Windows Command Prompt and type pip install pyreadstat
   
2. Install using Conda:
   Open up the Anaconda Prompt, and type conda install -c conda-forge pyreadstat

How to Open a Stata file in Python

In this section, we are finally ready to learn how to read a .dta file in Python using the Python packages Pyreadstat and Pandas.

How to Load a Stata File in Python Using Pyreadstat

In this section, we are going to use pyreadstat to import a .dta file into a Pandas dataframe. First, we import pyreadstat:

import pyreadstat

Second, we are ready to import Stata files using the method read_dta. Note that, when we load a file using the Pyreadstat package, it will look for the .dta file in Python’s working directory. In the read Stata files example below, the FifthDaydata.dta is located in a subdirectory (i.e., “SimData”).

dtafile = './SimData/FifthDayData.dta'
df, meta = pyreadstat.read_dta(dtafile)

In the code chunk above, two variables were created; df, and meta. If we use the Python function type we can see that “df” is a Pandas dataframe:

This means that we can use all the available methods for Pandas dataframe objects. In the next line of code, we are Pandas head method to print the first 5 rows.

df.head()

Learn more about working with Pandas dataframes in the following tutorials:

• Python Groupby Tutorial: Here you will learn about working the groupby method to group Pandas dataframes.
• Learn how to take random samples from a pandas dataframe
• A more general, overview, of how to work with Pandas dataframe objects can be found in the Pandas Dataframe tutorial.

How to Read a Stata file with Python Using Pandas

In this section, we are going to read the same Stata file into a Pandas dataframe. However, this time we will use Pandas read_stata method. This has the advantage that we can load the Statafile from a URL.

Before we continue, we need to import Pandas:

import pandas as pd

Now, when we have done that, we can read the .dta file into a Pandas dataframe using the read_stata method. In the read Stata example here, we are importing the same data file as in the previous example.

After we have loaded the Stata file using Python Pandas, we print the last 5 rows of the dataframe with the tail method.

dtafile = './SimData/FifthDayData.dta'

df = pd.read_stata(dtafile)
df.tail()

How to Read .dta Files from URL

In this section, we are going to use Pandas read_stata method, again. However, this time we will read the Stata file from a URL.

url = 'http://www.principlesofeconometrics.com/stata/broiler.dta'

df = pd.read_stata(url)
df.head()

Note, the only thing we changed was we used a URL as input (url) and Pandas read_stata will import the .dta file that the URL is pointing to.

Pandas Scatter Plot

Here, we will create a scatter plot in Python using Pandas scatter method. This is to illustrate how we can work with data imported from .dta files.

df.plot.scatter(x='pchick',
                       y='cpi')

Scatter Plot in Python

Learn more about data visualization in Python:

• How to Make a Scatter Plot in Python using Seaborn
• 9 Data Visualization Techniques You Should Learn in Python

How to Read Specific Columns from a Stata file

Now using pyreadstat read_dta and Pandas read_stat both enables us to read specific columns from a Stata file. Note, that read_dta have the argument usecols and Pandas the argument columns.

Reading Specific Columns using Pyreadstat

In this Python read dta example, we use the argument usecols that takes a list as parameter.

import pyreadstat

dtafile = './SimData/FifthDayData.dta'
df, meta = pyreadstat.read_dta(dtafile,
                              usecols=['index', 'Name', 'ID',
                                      'Gender'])
df.head()

Dataframe from .dta

Reading Specific Columns using Pandas read_stata

Here, we are going to use Pandas read_stata method and the argument columns. This argument, as in the example above, takes a list as input.

import pandas as pd
url = 'http://www.principlesofeconometrics.com/stata/broiler.dta'

df = pd.read_stata(url,
                  columns=['year', 'pchick', 'time',
                                      'meatex'])
df.head()

Dataframe

Note, the behavior of Pandas read_stata; in the resulting dataframe the order of the column will be the same as in the list we put in.

How to Save a Stata file

In this section of the Python Stata tutorial, we are going to save the dataframe as a .dta file. This is easily done, we just have to use the write_dta method when using Pyreadstat and the dataframe method to_stata in Pandas.

Saving a dataframe as a Stata file using Pyreadstat

In the example below, we are using the dataframe we created in the previous section and write it as a dta file.

pyreadstat.write_dta(df, 'broilerdata_edited.dta')

Now, between the parentheses is where the important stuff happens. The first argument is our dataframe and the second is the file path. Note, only having the filename, as in the example above, will make the write_dta method to write the Stata file to the current directory.

How to Save a dataframe as .dta with Pandas to_stata

In this example, we are going to save the same dataframe using Pandas to_stata:

df.to_stata('broilerdata_edited.dta')

As can be seen in the image above, the dataframe object has the to_stata method. Within, the parentheses we put the file path.

Save a CSV file as a Stata File

In this section, we are going to work with Pandas read_csv to read a CSV file, containing data. After we have imported the CSV to a dataframe we are going to save it as a .dta file using Pandas to_stat:

df = pd.read_csv('./SimData/FifthDayData.csv')
df.to_stata('./SimData/FifthDayData.dta')

Export an Excel file as a Stata File

In the final example, we are going to use Pandas read_excel to import a .xslx file and then save this dataframe as a Stata file using Pandas to_stat:

df = pd.read_excel('./SimData/example_concat.xlsx')
df.to_stata('./SimData/example_concat.dta')

Note, that in both of the last two examples above we save the data to a folder called SimData. If we want to save the CSV and Excel file to the current directory we simply remove the “./SimData/” part of the string.

Learn more about importing data using Pandas:

• Pandas Read CSV Tutorial
• Pandas Read Excel Tutorial

Note, all the files we have read using read_dta, read_stata, read_csv, and read_excel can be found here and a Jupyter Notebook here. It is, of course, possible to open SPSS and SAS files using Pandas and save them as .dta files as well.

Summary: Read Stata Files using Python

In this post, we have learned how to read Stata files in Python. Furthermore, we have learned how to write Pandas dataframes to Stata files.

The post Tutorial: How to Read Stata Files in Python with Pandas appeared first on Erik Marsja.

Wing Python IDE includes a boatload of features aimed at making it easier to navigate and understand the structure of Python code. Some of these allow for quick navigation between the definition and uses of a symbol. Others provide a convenient index into source code. And still others quickly find and open files or navigate to symbols matching a name fragment.

In the this and the next two Wing Tips, we'll take a look at each of these in turn.

Goto Definition

To get from any use of a symbol in Python code to its definition, use GotoSelectedSymbolDefn in the Source menu. This jumps to the def, class, or the point at which a variable or attribute was first defined.

Another way to do this is to right-click on the symbol in the editor and select GotoDefinition or GotoDefinitioninOtherSplit:

The menus also give the key bindings for the commands, or you can bind your own key to the command goto-selected-symbol-defn with the UserInterface>Keyboard>CustomKeyBindings preference.

In some cases, jumping to a definition successfully depends on resolving imported modules correctly using the Python Path configured by Python. In most cases you will not need to add to this configuration, but doing so is possible with ProjectProperties from Wing's Project menu.

Find Uses

In Wing Pro only, FindPointsofUse in the Source menu or the editor's right-click context menu finds all points of use of a symbol in Python code:

This search distinguishes between different but like-named symbols and will cover all project files and other files Wing finds on the configured Python Path. The tool's Options menu provides control over which files are searched and what types of matches are shown.

Search in Files

To find all occurrences of other strings in Python files or in project files of any type, use the SearchinFiles tool from the Tools menu with Lookin set to ProjectFiles and Filter set to the narrowest filter that includes the files that you wish to search:

This tool supports text matching, wildcard, and regular expression searching and automatically updates the search results as files change.

Searching on ProjectFiles assumes that you have used AddExistingDirectory in the Project menu to add your source code to your project. Typically the project should contain the code you are actively working on. Packages that your code uses can be left out of the project, unless you anticipate often wanting to search them with SearchinFiles.

That's it for now! We'll be back next week to continue this Wing Tips mini-series on navigating Python code with Wing.

As always, please don't hesitate to email support@wingware.com if you run into problems or have any questions.

We’re pleased to announce that pandas, the open-source library providing high-performance data structures for tabular data analysis, has received grant funding from the Chan Zuckerberg Initiative (CZI) as part of their Essential Open Source Software…

The post Essential Open-Source Library pandas Awarded CZI Grant to Further Development appeared first on Anaconda.

A veces gente me pide consejos sobre su carrera y cosas así... probablemente no soy la persona correcta para darlos.

Following the recent introduction of Python type annotations (aka "type hints") in Mercurial (see, e.g. this changeset by Augie Fackler), I've been playing a bit with this and pytype.

pytype is a static type analyzer for Python code. It compares with the more popular mypy but I don't have enough perspective to make a meaningful comparison at the moment. In this post, I'll illustrate how I worked with pytype to gradually add type hints in a Mercurial module and while doing so, fix bugs!

The module I focused on is mercurial.mail, which contains mail utilities and that I know quite well. Other modules are also being worked on, this one is a good starting point because it has a limited number of "internal" dependencies, which both makes it faster to iterate with pytype and reduces side effects of other modules not being correctly typed already.

$ pytype mercurial/mail.py
Computing dependencies
Analyzing 1 sources with 36 local dependencies
ninja: Entering directory `.pytype'
[19/19] check mercurial.mail
Success: no errors found

The good news is that the module apparently already type-checks. Let's go deeper and merge the type annotations generated by pytype:

$ merge-pyi -i mercurial/mail.py out/mercurial/mail.pyi

(In practice, we'd use --as-comments option to write type hints as comments, so that the module is still usable on Python 2.)

Now we have all declarations annotated with types. Typically, we'd get many things like:

def codec2iana(cs) -> Any:
   cs = pycompat.sysbytes(email.charset.Charset(cs).input_charset.lower())
   # "latin1" normalizes to "iso8859-1", standard calls for "iso-8859-1"
   if cs.startswith(b"iso") and not cs.startswith(b"iso-"):
      return b"iso-" + cs[3:]
   return cs

The function signature has been annotated with Any (omitted for parameters, explicit for return value). This somehow means that type inference failed to find the type of that function. As it's (quite) obvious, let's change that into:

def codec2iana(cs: bytes) -> bytes:
   ...

And re-run pytype:

$ pytype mercurial/mail.py
Computing dependencies
Analyzing 1 sources with 36 local dependencies
ninja: Entering directory `.pytype'
[1/1] check mercurial.mail
FAILED: .pytype/pyi/mercurial/mail.pyi
pytype-single --imports_info .pytype/imports/mercurial.mail.imports --module-name mercurial.mail -V 3.7 -o .pytype/pyi/mercurial/mail.pyi --analyze-annotated --nofail --quick mercurial/mail.py
File "mercurial/mail.py", line 253, in codec2iana: Function Charset.__init__ was called with the wrong arguments [wrong-arg-types]
  Expected: (self, input_charset: str = ...)
  Actually passed: (self, input_charset: bytes)

For more details, see https://google.github.io/pytype/errors.html#wrong-arg-types.
ninja: build stopped: subcommand failed.

Interesting! email.charset.Charset is apparently instantiated with the wrong argument type. While it's not exactly a bug, because Python will handle bytes instead of str well in general, we can again change the signature (and code) to:

def codec2iana(cs: str) -> str:
   cs = email.charset.Charset(cs).input_charset.lower()
   # "latin1" normalizes to "iso8859-1", standard calls for "iso-8859-1"
   if cs.startswith("iso") and not cs.startswith("iso-"):
      return "iso-" + cs[3:]
   return cs

Obviously, this involves a larger refactoring in client code of this simple function, see respective changeset for details.

Another example is this function:

def _encode(ui, s, charsets) -> Any:
    '''Returns (converted) string, charset tuple.
    Finds out best charset by cycling through sendcharsets in descending
    order. Tries both encoding and fallbackencoding for input. Only as
    last resort send as is in fake ascii.
    Caveat: Do not use for mail parts containing patches!'''
    sendcharsets = charsets or _charsets(ui)
    if not isinstance(s, bytes):
        # We have unicode data, which we need to try and encode to
        # some reasonable-ish encoding. Try the encodings the user
        # wants, and fall back to garbage-in-ascii.
        for ocs in sendcharsets:
            try:
                return s.encode(pycompat.sysstr(ocs)), ocs
            except UnicodeEncodeError:
                pass
            except LookupError:
                ui.warn(_(b'ignoring invalid sendcharset: %s\n') % ocs)
        else:
            # Everything failed, ascii-armor what we've got and send it.
            return s.encode('ascii', 'backslashreplace')
    # We have a bytes of unknown encoding. We'll try and guess a valid
    # encoding, falling back to pretending we had ascii even though we
    # know that's wrong.
    try:
        s.decode('ascii')
    except UnicodeDecodeError:
        for ics in (encoding.encoding, encoding.fallbackencoding):
            ics = pycompat.sysstr(ics)
            try:
                u = s.decode(ics)
            except UnicodeDecodeError:
                continue
            for ocs in sendcharsets:
                try:
                    return u.encode(pycompat.sysstr(ocs)), ocs
                except UnicodeEncodeError:
                    pass
                except LookupError:
                    ui.warn(_(b'ignoring invalid sendcharset: %s\n') % ocs)
    # if ascii, or all conversion attempts fail, send (broken) ascii
    return s, b'us-ascii'

It quite clear from the return value (last line) that we can change its type signature to:

def _encode(ui, s:Union[bytes, str], charsets: List[bytes]) -> Tuple[bytes, bytes]
   ...

And re-run pytype:

$ pytype mercurial/mail.py
Computing dependencies
Analyzing 1 sources with 36 local dependencies
ninja: Entering directory `.pytype'
[1/1] check mercurial.mail
FAILED: .pytype/pyi/mercurial/mail.pyi
pytype-single --imports_info .pytype/imports/mercurial.mail.imports --module-name mercurial.mail -V 3.7 -o .pytype/pyi/mercurial/mail.pyi --analyze-annotated --nofail --quick mercurial/mail.py
File "mercurial/mail.py", line 342, in _encode: bad option in return type [bad-return-type]
  Expected: Tuple[bytes, bytes]
  Actually returned: bytes
[...]

For more details, see https://google.github.io/pytype/errors.html.
ninja: build stopped: subcommand failed.

That's a real bug. Line 371 contains return s.encode('ascii', 'backslashreplace') in the middle of the function. We indeed return a bytes value instead of a Tuple[bytes, bytes] as elsewhere in the function. The following changes fixes the bug:

diff --git a/mercurial/mail.py b/mercurial/mail.py
--- a/mercurial/mail.py
+++ b/mercurial/mail.py
@@ -339,7 +339,7 @@ def _encode(ui, s, charsets) -> Any:
                 ui.warn(_(b'ignoring invalid sendcharset: %s\n') % ocs)
         else:
             # Everything failed, ascii-armor what we've got and send it.
-            return s.encode('ascii', 'backslashreplace')
+            return s.encode('ascii', 'backslashreplace'), b'us-ascii'
     # We have a bytes of unknown encoding. We'll try and guess a valid
     # encoding, falling back to pretending we had ascii even though we
     # know that's wrong.

Steps by steps, by replacing Any with real types, adjusting "wrong" types like in the first example and fixing bugs, we finally get the whole module (almost) completely annotated.