The concept of scope rules how variables and names are looked up in your code. It determines the visibility of a variable within the code. The scope of a name or variable depends on the place in your code where you create that variable. The Python scope concept is generally presented using a rule known as the LEGB rule.

The letters in the acronym LEGB stand for Local, Enclosing, Global, and Built-in scopes. This summarizes not only the Python scope levels but also the sequence of steps that Python follows when resolving names in a program.

In this tutorial, you’ll learn:

• What scopes are and how they work in Python
• Why it’s important to know about Python scope
• What the LEGB rule is and how Python uses it to resolve names
• How to modify the standard behavior of Python scope using global and nonlocal
• What scope-related tools Python offers and how you can use them

With this knowledge at hand, you can take advantage of Python scopes to write more reliable and maintainable programs. Using Python scope will help you avoid or minimize bugs related to name collision as well as bad use of global names across your programs.

You’ll get the most out of this tutorial if you’re familiar with intermediate Python concepts like classes, functions, inner functions, variables, exceptions, comprehensions, built-in functions, and standard data structures.

Understanding Scope

In programming, the scope of a name defines the area of a program in which you can unambiguously access that name, such as variables, functions, objects, and so on. A name will only be visible to and accessible by the code in its scope. Several programming languages take advantage of scope for avoiding name collisions and unpredictable behaviors. Most commonly, you’ll distinguish two general scopes:

1. Global scope: The names that you define in this scope are available to all your code.

2. Local scope: The names that you define in this scope are only available or visible to the code within the scope.

Scope came about because early programming languages (like BASIC) only had global names. With this kind of name, any part of the program could modify any variable at any time, so maintaining and debugging large programs could become a real nightmare. To work with global names, you’d need to keep all the code in mind at the same time to know what the value of a given name is at any time. This was an important side-effect of not having scopes.

Some languages like Python use scope to avoid this kind of problem. When you use a language that implements scope, there’s no way for you to access all the variables in a program at all locations in that program. In this case, your ability to access a given name will depend on where you’ve defined that name.

The names in your programs will have the scope of the block of code in which you define them. When you can access the value of a given name from someplace in your code, you’ll say that the name is in scope. If you can’t access the name, then you’ll say that the name is out of scope.

Names and Scopes in Python

Since Python is a dynamically-typed language, variables in Python come into existence when you first assign them a value. On the other hand, functions and classes are available after you define them using def or class, respectively. Finally, modules exist after you import them. As a summary, you can create Python names through one of the following operations:

All these operations create or, in the case of assignments, update new Python names because all of them assign a name to a variable, constant, function, class, instance, module, or other Python object.

Python uses the location of the name assignment or definition to associate it with a particular scope. In other words, where you assign or define a name in your code determines the scope or visibility of that name.

For example, if you assign a value to a name inside a function, then that name will have a local Python scope. In contrast, if you assign a value to a name outside of all functions—say, at the top level of a module—then that name will have a global Python scope.

Python Scope vs Namespace

In Python, the concept of scope is closely related to the concept of the namespace. As you’ve learned so far, a Python scope determines where in your program a name is visible. Python scopes are implemented as dictionaries that map names to objects. These dictionaries are commonly called namespaces. These are the concrete mechanisms that Python uses to store names. They’re stored in a special attribute called .__dict__.

Names at the top level of a module are stored in the module’s namespace. In other words, they’re stored in the module’s .__dict__ attribute. Take a look at the following code:

>>> importsys>>> sys.__dict__.keys()dict_keys(['__name__', '__doc__', '__package__',..., 'argv', 'ps1', 'ps2'])

After you import sys, you can use .keys() to inspect the keys of sys.__dict__. This returns a list with all the names defined at the top level of the module. In this case, you can say that .__dict__ holds the namespace of sys and is a concrete representation of the module scope.

As a further example, suppose that you need to use the name ps1, which is defined in sys. If you know how .__dict__ and namespaces work in Python, then you can reference ps1 in at least two different ways:

1. Using the dot notation on the module’s name in the form module.name
2. Using a subscription operation on .__dict__ in the form module.__dict__['name']

Take a look at the following code:

>>> sys.ps1'>>> '>>> sys.__dict__['ps1']'>>> '

Once you’ve imported sys you can access ps1 using the dot notation on sys. You can also access ps1 using a dictionary key lookup with the key 'ps1'. Both actions return the same result, '>>> '.

Whenever you use a name, such as a variable or a function name, Python searches through different scope levels (or namespaces) to determine whether the name exists or not. If the name exists, then you’ll always get the first occurrence of it. Otherwise, you’ll get an error. You’ll cover this search mechanism in the next section.

Using the LEGB Rule for Python Scope

Python resolves names using the so-called LEGB rule, which is named after the Python scope for names. The letters in LEGB stand for Local, Enclosing, Global, and Built-in. Here’s a quick overview of what these terms mean:

• Local (or function) scope is the code block or body of any Python function or lambda expression. This Python scope contains the names that you define inside the function. These names will only be visible from the code of the function. It’s created at function call, not at function definition, so you’ll have as many different local scopes as function calls. This is true even if you call the same function multiple times, or recursively. Each call will result in a new local scope being created.

• Enclosing (or nonlocal) scope is a special scope that only exists for nested functions. If the local scope is an inner or nested function, then the enclosing scope is the scope of the outer or enclosing function. This scope contains the names that you define in the enclosing function. The names in the enclosing scope are visible from the code of the inner and enclosing functions.

• Global (or module) scope is the top-most scope in a Python program, script, or module. This Python scope contains all of the names that you define at the top level of a program or a module. Names in this Python scope are visible from everywhere in your code.

• Built-in scope is a special Python scope that’s created or loaded whenever you run a script or open an interactive session. This scope contains names such as keywords, functions, exceptions, and other attributes that are built into Python. Names in this Python scope are also available from everywhere in your code. It’s automatically loaded by Python when you run a program or script.

The LEGB rule is a kind of name lookup procedure, which determines the order in which Python looks up names. For example, if you reference a given name, then Python will look that name up sequentially in the local, enclosing, global, and built-in scope. If the name exists, then you’ll get the first occurrence of it. Otherwise, you’ll get an error.

In summary, when you use nested functions, names are resolved by first checking the local scope or the innermost function’s local scope. Then, Python looks at all enclosing scopes of outer functions from the innermost scope to the outermost scope. If no match is found, then Python looks at the global and built-in scopes. If it can’t find the name, then you’ll get an error.

At any given time during execution, you’ll have at most four active Python scopes—local, enclosing, global, and built-in—depending on where you are in the code. On the other hand, you’ll always have at least two active scopes, which are the global and built-in scopes. These two scopes will always be available for you.

Functions: The Local Scope

The local scope or function scope is a Python scope created at function calls. Every time you call a function, you’re also creating a new local scope. On the other hand, you can think of each def statement and lambda expression as a blueprint for new local scopes. These local scopes will come into existence whenever you call the function at hand.

By default, parameters and names that you assign inside a function exist only within the function or local scope associated with the function call. When the function returns, the local scope is destroyed and the names are forgotten. Here’s how this works:

>>> defsquare(base):... result=base**2... print(f'The square of {base} is: {result}')...>>> square(10)The square of 10 is: 100>>> result# Isn't accessible from outside square()Traceback (most recent call last):
  File "<stdin>", line 1, in <module>resultNameError: name 'result' is not defined>>> base# Isn't accessible from outside square()Traceback (most recent call last):
  File "<stdin>", line 1, in <module>baseNameError: name 'base' is not defined>>> square(20)The square of 20 is: 400

square() is a function that computes the square of a given number, base. When you call the function, Python creates a local scope containing the names base (an argument) and result (a local variable). After the first call to square(), base holds a value of 10 and result holds a value of 100. The second time, the local names will not remember the values that were stored in them the first time the function was called. Notice that base now holds the value 20 and result holds 400.

Since you can’t access local names from statements that are outside the function, different functions can define objects with the same name. Check out this example:

>>> defcube(base):... result=base**3... print(f'The cube of {base} is: {result}')...>>> cube(30)The square of 30 is: 27000

Notice that you define cube() using the same variable and parameter that you used in square(). However, since cube() can’t see the names inside the local scope of square() and vice versa, both functions work as expected without any name collision.

You can avoid name collisions in your programs by properly using the local Python scope. This also makes functions more self-contained and creates maintainable program units. Additionally, since you can’t change local names from remote places in your code, your programs will be easier to debug, read, and modify.

You can inspect the names and parameters of a function using .__code__, which is an attribute that holds information on the function’s internal code. Take a look at the code below:

>>> square.__code__.co_varnames('base', 'result')>>> square.__code__.co_argcount1>>> square.__code__.co_consts(None, 2, 'The square of ', ' is: ')>>> square.__code__.co_name'square'

In this code example, you inspect .__code__ on square(). This is a special attribute that holds information about the code of a Python function. In this case, you see that .co_varnames holds a tuple containing the names that you define inside square().

Nested Functions: The Enclosing Scope

Enclosing or nonlocal scope is observed when you nest functions inside other functions. The enclosing scope was added in Python 2.2. It takes the form of the local scope of any enclosing function’s local scopes. Names that you define in the enclosing Python scope are commonly known as nonlocal names. Consider the following code:

>>> defouter_func():... # This block is the Local scope of outer_func()... var=100# A nonlocal var... # It's also the enclosing scope of inner_func()... definner_func():... # This block is the Local scope of inner_func()... print(f"Printing var from inner_func(): {var}")...... inner_func()... print(f"Printing var from outer_func(): {var}")...>>> outer_func()Printing var from inner_func(): 100Printing var from outer_func(): 100>>> inner_func()Traceback (most recent call last):
  File "<stdin>", line 1, in <module>NameError: name 'inner_func' is not defined

When you call outer_func(), you’re also creating a local scope. The local scope of outer_func() is, at the same time, the enclosing scope of inner_func(). From inside inner_func(), this scope is neither the global scope nor the local scope. It’s a special scope that lies in between those two scopes and is known as the enclosing scope.

All the names that you create in the enclosing scope are visible from inside inner_func(), except for those created after you call inner_func(). Here’s a new version of outer_fun() that shows this point:

>>> defouter_func():... var=100... definner_func():... print(f"Printing var from inner_func(): {var}")... print(f"Printing another_var from inner_func(): {another_var}")...... inner_func()... another_var=200# This is defined after calling inner_func()... print(f"Printing var from outer_func(): {var}")...>>> outer_func()Printing var from inner_func(): 100Traceback (most recent call last):
  File "<stdin>", line 1, in <module>outer_func()
  File "<stdin>", line 7, in outer_funcinner_func()
  File "<stdin>", line 5, in inner_funcprint(f"Printing another_var from inner_func(): {another_var}")NameError: free variable 'another_var' referenced before assignment in enclosing scope

When you call outer_func() the code runs down to the point in which you call inner_func(). The last statement of inner_func() tries to access another_var. At this point, another_var isn’t defined yet, so Python raises a NameError because it can’t find the name that you’re trying to use.

Last but not least, you can’t modify names in the enclosing scope from inside a nested function unless you declare them as nonlocal in the nested function. You’ll cover how to use nonlocallater in this tutorial.

Modules: The Global Scope

From the moment you start a Python program, you’re in the global Python scope. Internally, Python turns your program’s main script into a module called __main__ to hold the main program’s execution. The namespace of this module is the main global scope of your program.

If you’re working in a Python interactive session, then you’ll notice that '__main__' is also the name of its main module. To check that out, open an interactive session and type in the following:

>>> __name__'__main__'

Whenever you run a Python program or an interactive session like in the above code, the interpreter executes the code in the module or script that serves as an entry point to your program. This module or script is loaded with the special name, __main__. From this point on, you can say that your main global scope is the scope of __main__.

To inspect the names within your main global scope, you can use dir(). If you call dir() without arguments, then you’ll get the list of names that live in your current global scope. Take a look at this code:

>>> dir()['__annotations__', '__builtins__',..., '__package__', '__spec__']>>> var=100# Assign var at the top level of __main__>>> dir()['__annotations__', '__builtins__',..., '__package__', '__spec__', 'var']

When you call dir() with no arguments, you get the list of names available in your main global Python scope. Note that if you assign a new name (like var here) at the top level of the module (which is __main__ here), then that name will be added to the list returned by dir().

There’s only one global Python scope per program execution. This scope remains in existence until the program terminates and all its names are forgotten. Otherwise, the next time you were to run the program, the names would remember their values from the previous run.

You can access or reference the value of any global name from any place in your code. This includes functions and classes. Here’s an example that clarifies these points:

>>> var=100>>> deffunc():... returnvar# You can access var from inside func()...>>> func()100>>> var# Remains unchanged100

Inside func(), you can freely access or reference the value of var. This has no effect on your global name var, but it shows you that var can be freely accessed from within func(). On the other hand, you can’t assign global names inside functions unless you explicitly declare them as global names using a global statement, which you’ll see later on.

Whenever you assign a value to a name in Python, one of two things can happen:

1. You create a new name
2. You update an existing name

The concrete behavior will depend on the Python scope in which you’re assigning the name. If you try to assign a value to a global name inside a function, then you’ll be creating that name in the function’s local scope, shadowing or overriding the global name. This means that you won’t be able to change most variables that have been defined outside the function from within the function.

If you follow this logic, then you’ll realize that the following code won’t work as you might expect:

>>> var=100# A global variable>>> defincrement():... var=var+1# Try to update a global variable...>>> increment()Traceback (most recent call last):
  File "<stdin>", line 1, in <module>increment()
  File "<stdin>", line 2, in incrementvar=var+1UnboundLocalError: local variable 'var' referenced before assignment

Within increment(), you try to increment the global variable, var. Since var isn’t declared global inside increment(), Python creates a new local variable with the same name, var, inside the function. In the process, Python realizes that you’re trying to use the local var before its first assignment (var + 1), so it raises an UnboundLocalError.

Here’s another example:

>>> var=100# A global variable>>> deffunc():... print(var)# Reference the global variable, var... var=200# Define a new local variable using the same name, var...>>> func()Traceback (most recent call last):
  File "<stdin>", line 1, in <module>func()
  File "<stdin>", line 2, in funcprint(var)UnboundLocalError: local variable 'var' referenced before assignment

You likely expect to be able to print the global var and be able to update var later, but again you get an UnboundLocalError. What happens here is that when you run the body of func(), Python decides that var is a local variable because it’s assigned within the function scope. This isn’t a bug, but a design choice. Python assumes that names assigned in the body of a function are local to that function.

Up to this point, you’ve covered three Python scopes. Check out the following example for a summary on where they’re located in your code and how Python looks up names through them:

>>> # This area is the global or module scope>>> number=100>>> defouter_func():... # This block is the local scope of outer_func()... # It's also the enclosing scope of inner_func()... definner_func():... # This block is the local scope of inner_func()... print(number)...... inner_func()...>>> outer_func()100

When you call outer_func(), you get 100 printed on your screen. But how does Python look up the name number in this case? Following the LEGB rule, you’ll look up number in the following places:

1. Inside inner_func(): This is the local scope, but number doesn’t exist there.
2. Inside outer_func(): This is the enclosing scope, but number isn’t defined there either.
3. In the module scope: This is the global scope, and you find number there, so you can print number to the screen.

If number isn’t defined inside the global scope, then Python continues the search by looking at the built-in scope. This is the last component of the LEGB rule, as you’ll see in the next section.

builtins: The Built-In Scope

The built-in scope is a special Python scope that’s implemented as a standard library module named builtins in Python 3.x. All of Python’s built-in objects live in this module. They’re automatically loaded to the built-in scope when you run the Python interpreter. Python searches builtins last in its LEGB lookup, so you get all the names it defines for free. This means that you can use them without importing any module.

Notice that the names in builtins are always loaded into your global Python scope with the special name __builtins__, as you can see in the following code:

>>> dir()['__annotations__', '__builtins__',..., '__package__', '__spec__']>>> dir(__builtins__)['ArithmeticError', 'AssertionError',..., 'tuple', 'type', 'vars', 'zip']

In the output of the first call to dir(), you can see that __builtins__ is always present in the global Python scope. If you inspect __builtins__ itself using dir(), then you’ll get the whole list of Python built-in names.

The built-in scope brings more than 150 names to your current global Python scope. For example, in Python 3.8 you can get to know the exact number of names as follows:

>>> len(dir(__builtins__))152

With the call to len(), you get the number of items in the list returned by dir(). This returns 152 names that include exceptions, functions, types, special attributes, and other Python built-in objects.

Even though you can access all of these Python built-in objects for free (without importing anything), you can also explicitly import builtins and access the names using the dot notation. Here’s how this works:

>>> importbuiltins# Import builtins as a regular module>>> dir(builtins)['ArithmeticError', 'AssertionError',..., 'tuple', 'type', 'vars', 'zip']>>> builtins.sum([1,2,3,4,5])15>>> builtins.max([1,5,8,7,3])8>>> builtins.sorted([1,5,8,7,3])[1, 3, 5, 7, 8]>>> builtins.pow(10,2)100

You can import builtins as you would any other Python module. From this point on, you can access all the names in builtins by using the dotted attribute lookup or fully-qualified names. This can be quite useful if you want to make sure that you won’t have a name collision if any of your global names override any built-in name.

You can override or redefine any built-in name in your global scope. If you do so, then keep in mind that this will affect all your code. Take a look at the following example:

>>> abs(-15)# Standard use of a built-in function15>>> abs=20# Redefine a built-in name in the global scope>>> abs(-15)Traceback (most recent call last):
  File "<stdin>", line 1, in <module>TypeError: 'int' object is not callable

If you override or re-assign abs, then the original built-in abs() is affected all over your code. Now, suppose that you need to call the original abs() and you forget that you re-assigned the name. In this case, when you call abs() again, you’d get a TypeError because abs now holds a reference to an integer, which is not callable.

If you’re experimenting with some code and you accidentally re-assign a built-in name at the interactive prompt, then you can either restart your session or run del name to remove the redefinition from your global Python scope. This way, you’re restoring the original name in the built-in scope. If you revisit the example of abs(), then you can do something like this:

>>> delabs# Remove the redefined abs from your global scope>>> abs(-15)# Restore the original abs()15

When you delete the custom abs name, you’re removing the name from your global scope. This allows you to access the original abs() in the built-in scope again.

To work around this kind of situation, you can explicitly import builtins and then use fully-qualified names, like in the following code fragment:

>>> importbuiltins>>> builtins.abs(-15)15

Once you explicitly import builtins, you have the module name available in your global Python scope. From this point on, you can use fully-qualified names to unambiguously get the names you need from builtins, just like you did with builtins.abs() in the above example.

As a quick summary, some of the implications of Python scope are shown in the following table:

Additionally, code in different scopes can use the same name for different objects. This way, you can use a local variable named spam and also a global variable with the same name, spam. However, this is considered bad programming practice.

Modifying the Behavior of a Python Scope

So far, you’ve learned how a Python scope works and how they restrict the visibility of variables, functions, classes, and other Python objects to certain portions of your code. You now know that you can access or reference global names from any place in your code, but they can be modified or updated from within the global Python scope.

You also know that you can access local names only from inside the local Python scope they were created in or from inside a nested function, but you can’t access them from the global Python scope or from other local scopes. Additionally, you’ve learned that nonlocal names can be accessed from inside nested functions, but they can’t be modified or updated from there.

Even though Python scopes follow these general rules by default, there are ways to modify this standard behavior. Python provides two keywords that allow you to modify the content of global and nonlocal names. These two keywords are:

1. global
2. nonlocal

In the next two sections, you’ll cover how to use these Python keywords to modify the standard behavior of Python scopes.

The global Statement

You already know that when you try to assign a value to a global name inside a function, you create a new local name in the function scope. To modify this behavior, you can use a global statement. With this statement, you can define a list of names that are going to be treated as global names.

The statement consists of the global keyword followed by one or more names separated by commas. You can also use multiple global statements with a name (or a list of names). All the names that you list in a global statement will be mapped to the global or module scope in which you define them.

Here’s an example where you try to update a global variable from within a function:

>>> counter=0# A global name>>> defupdate_counter():... counter=counter+1# Fail trying to update counter...>>> update_counter()Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 2, in update_counterUnboundLocalError: local variable 'counter' referenced before assignment

When you try to assign counter inside update_counter(), Python assumes that counter is local to update_counter() and raises an UnboundLocalError because you’re trying to access a name that isn’t defined yet.

If you want this code to work the way you expect here, then you can use a global statement as follows:

>>> counter=0# A global name>>> defupdate_counter():... globalcounter# Declare counter as global... counter=counter+1# Successfully update the counter...>>> update_counter()>>> counter1>>> update_counter()>>> counter2>>> update_counter()>>> counter3

In this new version of update_counter(), you add the statement global counter to the body of the function right before you try to change counter. With this tiny change, you’re mapping the name counter in the function scope to the same name in the global or module scope. From this point on, you can freely modify counter inside update_counter(). All the changes will reflect in the global variable.

With the statement global counter, you’re telling Python to look in the global scope for the name counter. This way, the expression counter = counter + 1 doesn’t create a new name in the function scope, but updates it in the global scope.

>>> global_counter=0# A global name>>> defupdate_counter(counter):... returncounter+1# Rely on a local name...>>> global_counter=update_counter(global_counter)>>> global_counter1>>> global_counter=update_counter(global_counter)>>> global_counter2>>> global_counter=update_counter(global_counter)>>> global_counter3

You can also use a global statement to create lazy global names by declaring them inside a function. Take a look at the following code:

>>> defcreate_lazy_name():... globallazy# Create a global name, lazy... lazy=100... returnlazy...>>> create_lazy_name()100>>> lazy# The name is now available in the global scope100>>> dir()['__annotations__', '__builtins__',..., 'create_lazy_name', 'lazy']

When you call create_lazy_name(), you’re also creating a global variable called lazy. Notice that after calling the function, the name lazy is available in the global Python scope. If you inspect the global namespace using dir(), then you’ll see that lazy appears last in the list.

Finally, it’s worth noting that you can use global from inside any function or nested function and the names listed will always be mapped to names in the global Python scope.

Also notice that, even though using a global statement at the top level of a module is legal, it doesn’t make much sense because any name assigned in the global scope is already a global name by definition. Take a look at the following code:

>>> name=100>>> dir()['__annotations__', '__builtins__',..., '__spec__', 'name']>>> globalname>>> dir()['__annotations__', '__builtins__',..., '__spec__', 'name']

The use of a global statement like global name doesn’t change anything in your current global scope, as you can see in the output of dir(). The variable name is a global variable whether you use global or not.

The nonlocal Statement

Similarly to global names, nonlocal names can be accessed from inner functions, but not assigned or updated. If you want to modify them, then you need to use a nonlocal statement. With a nonlocal statement, you can define a list of names that are going to be treated as nonlocal.

The nonlocal statement consists of the nonlocal keyword followed by one or more names separated by commas. These names will refer to the same names in the enclosing Python scope. The following example shows how you can use nonlocal to modify a variable defined in the enclosing or nonlocal scope:

>>> deffunc():... var=100# A nonlocal variable... defnested():... nonlocalvar# Declare var as nonlocal... var+=100...... nested()... print(var)...>>> func()200

With the statement nonlocal var, you tell Python that you’ll be modifying var inside nested(). Then, you increment var using an augmented assignment operation. This change is reflected in the nonlocal name var, which now has a value of 200.

Unlike global, you can’t use nonlocal outside of a nested or enclosed function. To be more precise, you can’t use a nonlocal statement in either the global scope or in a local scope. Here’s an example:

>>> nonlocalmy_var# Try to use nonlocal in the global scope  File "<stdin>", line 1SyntaxError: nonlocal declaration not allowed at module level>>> deffunc():... nonlocalvar# Try to use nonlocal in a local scope... print(var)...
  File "<stdin>", line 2SyntaxError: no binding for nonlocal 'var' found

Here, you first try to use a nonlocal statement in the global Python scope. Since nonlocal only works inside an inner or nested function, you get a SyntaxError telling you that you can’t use nonlocal in a module scope. Notice that nonlocal doesn’t work inside a local scope either.

In contrast to global, you can’t use nonlocal to create lazy nonlocal names. Names must already exist in the enclosing Python scope if you want to use them as nonlocal names. This means that you can’t create nonlocal names by declaring them in a nonlocal statement in a nested function. Take a look at the following code example:

>>> deffunc():... defnested():... nonlocallazy_var# Try to create a nonlocal lazy name...
  File "<stdin>", line 3SyntaxError: no binding for nonlocal 'lazy_var' found

In this example, when you try to define a nonlocal name using nonlocal lazy_var, Python immediately raises a SyntaxError because lazy_var doesn’t exist in the enclosing scope of nested().

Using Enclosing Scopes as Closures

Closures are a special use case of the enclosing Python scope. When you handle a nested function as data, the statements that make up that function are packaged together with the environment in which they execute. The resulting object is known as a closure. In other words, a closure is an inner or nested function that carries information about its enclosing scope, even though this scope has completed its execution.

Closures provide a way to retain state information between function calls. This can be useful when you want to write code based on the concept of lazy or delayed evaluation. Take a look at the following code for an example of how closures work and how you can take advantage of them in Python:

>>> defpower_factory(exp):... defpower(base):... returnbase**exp... returnpower...>>> square=power_factory(2)>>> square(10)100>>> cube=power_factory(3)>>> cube(10)1000>>> cube(5)125>>> square(15)225

Your closure factory function power_factory() takes an argument called exp. You can use this function to build closures that run different power operations. This works because each call to power_factory() gets its own set of state information. In other words, it gets its value for exp.

In the above example, the inner function power() is first assigned to square. In this case, the function remembers that exp equals 2. In the second example, you call power_factory() using 3 as an argument. This way, cube holds a function object, which remembers that exp is 3. Notice that you can freely reuse square and cube because they don’t forget their respective state information.

For a final example on how to use closures, suppose that you need to calculate the mean of some sample data. You collect the data through a stream of successive measurements of the parameter you’re analyzing. In this case, you can use a closure factory to generate a closure that remembers the previous measurements in the sample. Take a look at the following code:

>>> defmean():... sample=[]... def_mean(number):... sample.append(number)... returnsum(sample)/len(sample)... return_mean...>>> current_mean=mean()>>> current_mean(10)10.0>>> current_mean(15)12.5>>> current_mean(12)12.333333333333334>>> current_mean(11)12.0>>> current_mean(13)12.2

The closure that you create in the above code remembers the state information of sample between calls of current_mean. This way, you can solve the problem in an elegant and Pythonic way.

Notice that if your data stream gets too large, then this function can become a problem in terms of memory usage. That’s because with each call to current_mean, sample will hold a bigger and bigger list of values. Take a look at the following code for an alternative implementation using nonlocal:

>>> defmean():... total=0... length=0... def_mean(number):... nonlocaltotal,length... total+=number... length+=1... returntotal/length... return_mean...>>> current_mean=mean()>>> current_mean(10)10.0>>> current_mean(15)12.5>>> current_mean(12)12.333333333333334>>> current_mean(11)12.0>>> current_mean(13)12.2

Even though this solution is more verbose, you don’t have an endlessly growing list anymore. You now have a single value for total and length. This implementation is a lot more efficient in terms of memory consumption than the previous solution.

Finally, you can find some examples of using closures in the Python standard library. For example, functools provides a function named partial() that makes use of the closure technique to create new function objects that can be called using predefined arguments. Here’s an example:

>>> fromfunctoolsimportpartial>>> defpower(exp,base):... returnbase**exp...>>> square=partial(power,2)>>> square(10)100

You use partial to build a function object that remembers the state information, where exp=2. Then, you call this object to perform the power operation and get the final result.

Bringing Names to Scope With import

When you write a Python program, you typically organize the code into several modules. For your program to work, you’ll need to bring the names in those separate modules to your __main__ module. To do that, you need to import the modules or the names explicitly. This is the only way you can use those names in your main global Python scope.

Take a look at the following code for an example of what happens when you import some standard modules and names:

>>> dir()['__annotations__', '__builtins__',..., '__spec__']>>> importsys>>> dir()['__annotations__', '__builtins__',..., '__spec__', 'sys']>>> importos>>> dir()['__annotations__', '__builtins__',..., '__spec__', 'os', 'sys']>>> fromfunctoolsimportpartial>>> dir()['__annotations__', '__builtins__',..., '__spec__', 'os', 'partial', 'sys']

You first import sys and os from the Python standard library. By calling dir() with no arguments, you can see that these modules are now available for you as names in your current global scope. This way, you can use dot notation to get access to the names that are defined in sys and os.

In the latest import operation, you use the form from <module> import <name>. This way, you can use the imported name directly in your code. In other words, you don’t need to explicitly use the dot notation.

Discovering Unusual Python Scopes

You’ll find some Python structures where name resolution seems not to fit into the LEGB rule for Python scopes. These structures include:

• Comprehensions
• Exception blocks
• Classes and instances

In the next few sections, you’ll cover how Python scope works on these three structures. With this knowledge, you’ll be able to avoid subtle errors related to the use of names in these kinds of Python structures.

Comprehension Variables Scope

The first structure you’ll cover is the comprehension. A comprehension is a compact way to process all or part of the elements in a collection or sequence. You can use comprehensions to create lists, dictionaries, and sets.

Comprehensions consist of a pair of brackets ([]) or curly braces ({}) containing an expression, followed by one or more for clauses and then zero or one if clause per for clause.

The for clause in a comprehension works similarly to a traditional for loop. The loop variable in a comprehension is local to the structure. Check out the following code:

>>> [itemforiteminrange(5)][0, 1, 2, 3, 4]>>> item# Try to access the comprehension variableTraceback (most recent call last):
  File "<stdin>", line 1, in <module>itemNameError: name 'item' is not defined

Once you run the list comprehension, the variable item is forgotten and you can’t access its value anymore. It’s unlikely that you need to use this variable outside of the comprehension, but regardless, Python makes sure that its value is no longer available once the comprehension finishes.

Note that this only applies to comprehensions. When it comes to regular for loops, the loop variable holds the last value processed by the loop:

>>> foriteminrange(5):... print(item)...01234>>> item# Access the loop variable4

You can freely access the loop variable item once the loop has finished. Here, the loop variable holds the last value processed by the loop, which is 4 in this example.

Exception Variables Scope

Another atypical case of Python scope that you’ll encounter is the case of the exception variable. The exception variable is a variable that holds a reference to the exception raised by a try statement. In Python 3.x, such variables are local to the except block and are forgotten when the block ends. Check out the following code:

>>> lst=[1,2,3]>>> try:... lst[4]... exceptIndexErroraserr:... # The variable err is local to this block... # Here you can do anything with err... print(err)...list index out of range>>> err# Is out of scopeTraceback (most recent call last):
  File "<stdin>", line 1, in <module>errNameError: name 'err' is not defined

err holds a reference to the exception raised by the try clause. You can use err only inside the code block of the except clause. This way, you can say that the Python scope for the exception variable is local to the except code block. Also note that if you try to access err from outside the except block, then you’ll get a NameError. That’s because once the except block finishes, the name doesn’t exist anymore.

To work around this behavior, you can define an auxiliary variable out of the try statement and then assign the exception to that variable inside the except block. Check out the following example:

>>> lst=[1,2,3]>>> ex=None>>> try:... lst[4]... exceptIndexErroraserr:... ex=err... print(err)...list index out of range>>> err# Is out of scopeTraceback (most recent call last):
  File "<stdin>", line 1, in <module>NameError: name 'err' is not defined>>> ex# Holds a reference to the exceptionlist index out of range

You use ex as an auxiliary variable to hold a reference to the exception raised by the try clause. This can be useful when you need to do something with the exception object once the code block has finished. Note that if no exception is raised, then ex remains None.

Class and Instance Attributes Scope

When you define a class, you’re creating a new local Python scope. The names assigned at the top level of the class live in this local scope. The names that you assigned inside a class statement don’t clash with names elsewhere. You can say that these names follow the LEGB rule, where the class block represents the L level.

Unlike functions, the class local scope isn’t created at call time, but at execution time. Each class object has its own .__dict__ attribute that holds the class scope or namespace where all the class attributes live. Check out this code:

>>> classA:... attr=100...>>> A.__dict__.keys()dict_keys(['__module__', 'attr', '__dict__', '__weakref__', '__doc__'])

When you inspect the keys of .__dict__ you’ll see that attr is in the list along with other special names. This dictionary represents the class local scope. The names in this scope are visible to all instances of the class and to the class itself.

To get access to a class attribute from outside the class, you need to use the dot notation as follows:

>>> classA:... attr=100... print(attr)# Access class attributes directly...100>>> A.attr# Access a class attribute from outside the class100>>> attr# Isn't defined outside ATraceback (most recent call last):
  File "<stdin>", line 1, in <module>attrNameError: name 'attr' is not defined

Inside the local scope of A, you can access the class attributes directly, just like you did in the statement print(attr). To access any class attribute once the code block of the class is executed, you’ll need to use the dot notation or attribute reference, as you did with A.attr. Otherwise, you’ll get a NameError, because the attribute attr is local to the class block.

On the other hand, if you try to access an attribute that isn’t defined inside a class, then you’ll get an AttributeError. Check out the following example:

>>> A.undefined# Try to access an undefined class attributeTraceback (most recent call last):
  File "<stdin>", line 1, in <module>A.undefinedAttributeError: type object 'A' has no attribute 'undefined'

In this example, you try to access the attribute undefined. Since this attribute doesn’t exist in A, you get an AttributeError telling you that A doesn’t have an attribute named undefined.

You can also access any class attribute using an instance of the class as follows:

>>> obj=A()>>> obj.attr100

Once you have the instance you can access the class attributes using the dot notation, as you did here with obj.attr. Class attributes are specific to the class object, but you can access them from any instances of the class. It’s worth noting that class attributes are common to all instances of a class. If you modify a class attribute, then the changes will be visible in all instances of the class.

Whenever you call a class, you’re creating a new instance of that class. Instances have their own .__dict__ attribute that holds the names in the instance local scope or namespace. These names are commonly called instance attributes and are local and specific to each instance. This means that if you modify an instance attribute, then the changes will be visible only to that specific instance.

To create, update, or access any instance attribute from inside the class, you need to use self along with the dot notation. Here, self is a special attribute that represents the current instance. On the other hand, to update or access any instance attribute from outside the class, you need to create an instance and then use the dot notation. Here’s how this works:

>>> classA:... def__init__(self,var):... self.var=var# Create a new instance attribute... self.var*=2# Update the instance attribute...>>> obj=A(100)>>> obj.__dict__{'var': 200}>>> obj.var200

The class A takes an argument called var, which is automatically doubled inside .__init__() using the assignment operation self.var *= 2. Note that when you inspect .__dict__ on obj, you get a dictionary containing all instance attributes. In this case, the dictionary contains only the name var, whose value is now 200.

Even though you can create instance attributes within any method in a class, it’s good practice to create and initialize them inside .__init__(). Take a look at this new version of A:

>>> classA:... def__init__(self,var):... self.var=var...... defduplicate_var(self):... returnself.var*2...>>> obj=A(100)>>> obj.var100>>> obj.duplicate_var()200>>> A.varTraceback (most recent call last):
  File "<stdin>", line 1, in <module>A.varAttributeError: type object 'A' has no attribute 'var'

Here, you modify A to add a new method called duplicate_var(). Then, you create an instance of A by passing in 100 to the class initializer. After that, you can now call duplicate_var() on obj to duplicate the value stored in self.var. Finally, if you try to access var using the class object instead of an instance, then you’ll get an AttributeError because instance attributes can’t be accessed using class objects.

In general, when you’re writing object-oriented code in Python and you try to access an attribute, your program takes the following steps:

1. Check the instance local scope or namespace first.
2. If the attribute is not found there, then check the class local scope or namespace.
3. If the name doesn’t exist in the class namespace either, then you’ll get an AttributeError.

This is the underlying mechanism by which Python resolves names in classes and instances.

Although classes define a class local scope or namespace, they don’t create an enclosing scope for methods. Therefore, when you’re implementing a class, references to attributes and methods must be done using the dot notation:

>>> classA:... var=100... defprint_var(self):... print(var)# Try to access a class attribute directly...>>> A().print_var()Traceback (most recent call last):
  File "<stdin>", line 1, in <module>A().print_var()
  File "<stdin>", line 4, in print_varprint(var)NameError: name 'var' is not defined

Since classes don’t create an enclosing scope for methods, you can’t access var directly from within print_var() like you try to do here. To get access to class attributes from inside any method, you need to use the dot notation. To fix the problem in this example, change the statement print(var) inside print_var() to print(A.var) and see what happens.

You can override a class attribute with an instance attribute, which will modify the general behavior of your class. However, you can access both attributes unambiguously using the dot notation like in the following example:

>>> classA:... var=100... def__init__(self):... self.var=200...... defaccess_attr(self):... # Use dot notation to access class and instance attributes... print(f'The instance attribute is: {self.var}')... print(f'The class attribute is: {A.var}')...>>> obj=A()>>> obj.access_attr()The instance attribute is: 200The class attribute is: 100>>> A.var# Access class attributes100>>> A().var# Access instance attributes200>>> A.__dict__.keys()dict_keys(['__module__', 'var', '__init__',..., '__getattribute__'])>>> A().__dict__.keys()dict_keys(['var'])

The above class has an instance attribute and a class attribute with the same name var. You can use the following code to access each of them:

1. Instance: Use self.var to access this attribute.
2. Class: Use A.var to access this attribute.

Since both cases use the dot notation, there are no name collision problems.

Finally, notice that the class .__dict__ and the instance .__dict__ are totally different and independent dictionaries. That’s why class attributes are available immediately after you run or import the module in which the class was defined. In contrast, instance attributes come to life only after an object or instance is created.

Using Scope Related Built-In Functions

There are many built-in functions that are closely related to the concept of Python scope and namespaces. In previous sections, you’ve used dir() to get information on the names that exist in a given scope. Besides dir(), there are some other built-in functions that can help you out when you’re trying to get information about a Python scope or namespace. In this section, you’ll cover how to work with:

• globals()
• locals()
• dir()
• vars()

Since all these are built-in functions, they’re available for free in the built-in scope. This means that you can use them at any time without importing anything. Most of these functions are intended to be used in an interactive session to get information on different Python objects. However, you can find some interesting use cases for them in your code as well.

globals()

In Python, globals() is a built-in function that returns a reference to the current global scope or namespace dictionary. This dictionary always stores the names of the current module. This means that if you call globals() in a given module, then you’ll get a dictionary containing all the names that you’ve defined in that module, right before the call to globals(). Here’s an example:

>>> globals(){'__name__': '__main__',..., '__builtins__': <module 'builtins' (built-in)>}>>> my_var=100>>> globals(){'__name__': '__main__',..., 'my_var': 100}

The first call to globals() returns a dictionary containing the names in your __main__ module or program. Note that when you assign a new name at the top level of the module, like in my_var = 100, the name is added to the dictionary returned by globals().

An interesting example of how you can use globals() in your code would be to dynamically dispatch functions that live in the global scope. Suppose you want to dynamically dispatch platform-dependent functions. To do this, you can use globals() as follows:

 1 # Filename: dispatch.py 2  3 fromsysimportplatform 4  5 deflinux_print(): 6 print('Printing from Linux...') 7  8 defwin32_print(): 9 print('Printing from Windows...')10 11 defdarwin_print():12 print('Printing from macOS...')13 14 printer=globals()[platform+'_print']15 16 printer()

If you run this script in your command line, then you’ll get an output that will depend on your current platform.

Another example of how to use globals() would be to inspect the list of special names in the global scope. Take a look at the following list comprehension:

>>> [namefornameinglobals()ifname.startswith('__')]['__name__', '__doc__', '__package__',..., '__annotations__', '__builtins__']

This list comprehension will return a list with all the special names that are defined in your current global Python scope. Note that you can use the globals() dictionary just like you would use any regular dictionary. For example, you can iterate through it through it using these traditional methods:

• .keys()
• .values()
• .items()

You can also perform regular subscription operations over globals() by using square brackets like in globals()['name']. For example, you can modify the content of globals() even though this isn’t recommended. Take a look at this example:

>>> globals()['__doc__']="""Docstring for __main__.""">>> __doc__'Docstring for __main__.'

Here, you change the key __doc__ to include a docstring for __main__ so that from now on, the main module’s docstring will have the value 'Docstring for __main__.'.

locals()

Another function related to Python scope and namespaces is locals(). This function updates and returns a dictionary that holds a copy of the current state of the local Python scope or namespace. When you call locals() in a function block, you get all the names assigned in the local or function scope up to the point where you call locals(). Here’s an example:

>>> deffunc(arg):... var=100... print(locals())... another=200...>>> func(300){'var': 100, 'arg': 300}

Whenever you call locals() inside func(), the resulting dictionary contains the name var mapped to the value 100 and arg mapped to 300. Since locals() only grabs the names assigned before you call it, another is not in the dictionary.

If you call locals() in the global Python scope, then you’ll get the same dictionary that you would get if you were to call globals():

>>> locals(){'__name__': '__main__',..., '__builtins__': <module 'builtins' (built-in)>}>>> locals()isglobals()True

When you call locals() in the global Python scope, you get a dictionary that’s identical to the dictionary returned by the call to globals().

Note that you shouldn’t modify the content of locals() because changes may have no effect on the values of local and free names. Check out the following example:

>>> deffunc():... var=100... locals()['var']=200... print(var)...>>> func()100

When you try to modify the content of var using locals(), the change doesn’t reflect in the value of var. So, you can say that locals() is only useful for read operations since updates to the locals dictionary are ignored by Python.

vars()

vars() is a Python built-in function that returns the .__dict__ attribute of a module, class, instance, or any other object which has a dictionary attribute. Remember that .__dict__ is a special dictionary that Python uses to implement namespaces. Take a look at the following examples:

>>> importsys>>> vars(sys)# With a module object{'__name__': 'sys',..., 'ps1': '>>> ', 'ps2': '... '}>>> vars(sys)issys.__dict__True>>> classMyClass:... def__init__(self,var):... self.var=var...>>> obj=MyClass(100)>>> vars(obj)# With a user-defined object{'var': 100}>>> vars(MyClass)# With a classmappingproxy({'__module__': '__main__',..., '__doc__': None})

When you call vars() using sys as an argument, you get the .__dict__ of sys. You can also call vars() using different types of Python objects, as long as they have this dictionary attribute.

Without any argument, vars() acts like locals() and returns a dictionary with all the names in the local Python scope:

>>> vars(){'__name__': '__main__',..., '__builtins__': <module 'builtins' (built-in)>}>>> vars()islocals()True

Here, you call vars() at the top level of an interactive session. With no argument, this call returns a dictionary containing all the names in the global Python scope. Note that, at this level, vars() and locals() return the same dictionary.

If you call vars() with an object that doesn’t have a .__dict__, then you’ll get a TypeError, like in the following example:

>>> vars(10)# Call vars() with objects that don't have a .__dict__Traceback (most recent call last):
  File "<stdin>", line 1, in <module>TypeError: vars() argument must have __dict__ attribute

If you call vars() with an integer object, then you’ll get a TypeError because this type of Python object doesn’t have a .__dict__.

dir()

You can use dir() without arguments to get the list of names in the current Python scope. If you call dir() with an argument, then the function attempts to return a list of valid attributes for that object:

>>> dir()# With no arguments['__annotations__', '__builtins__',..., '__package__', '__spec__']>>> dir(zip)# With a function object['__class__', '__delattr__',..., '__str__', '__subclasshook__']>>> importsys>>> dir(sys)# With a module object['__displayhook__', '__doc__',..., 'version_info', 'warnoptions']>>> var=100>>> dir(var)# With an integer variable['__abs__', '__add__',..., 'imag', 'numerator', 'real', 'to_bytes']

If you call dir() with no arguments, then you get a list containing the names that live in the global scope. You can also use dir() to inspect the list of names or attributes of different objects. This includes functions, modules, variables, and so on.

Even though the official documentation says that dir() is intended for interactive use, you can use the function to provide a comprehensive list of attributes of a given object. Note that you can also call dir() from inside a function. In this case, you’ll get the list of names defined in the function scope:

>>> deffunc():... var=100... print(dir())... another=200# Is defined after calling dir()...>>> func()['var']

In this example, you use dir() inside func(). When you call the function, you get a list containing the names that you define in the local scope. It’s worth noting that in this case, dir() only shows the names you declared before the function call.

Conclusion

The scope of a variable or name defines its visibility throughout your code. In Python, scope is implemented as either a Local, Enclosing, Global, or Built-in scope. When you use a variable or name, Python searches these scopes sequentially to resolve it. If the name isn’t found, then you’ll get an error. This is the general mechanism that Python uses for name resolution and is known as the LEGB rule.

You’re now able to:

• Take advantage of Python scope to avoid or minimize bugs related to name collision
• Make good use of global and local names across your programs to improve code maintainability
• Use a coherent strategy to access, modify, or update names across all your Python code

Additionally, you’ve covered some scope-related tools and techniques that Python offers and how you can use them to gather information about the names that live in a given scope or to modify the standard behavior of Python scope. Of course, there’s more to this topic that’s outside the scope of this tutorial, so get out there and continue to tackle name resolution in Python!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Today a wanted to share a neat trick that might save you some headache: debugging a hanging test.

Setup

Let's write some really simple (contrived) code to test:

fromtimeimportsleepdefcall_api():returndict(status=200,response=[1,2,3])defsum_numbers(numbers):returnsum(numbers)

And some test code to test it:

fromscriptimportsum_numbers,call_apideftest_sum_numbers():assertsum_numbers([1,2,3])==6assertsum_numbers([4,5])==9deftest_call_api():resp=call_api()assertresp['status']==200assertresp['response']==[1,2,3]

Run pytest and: 2 passed in 0.03s - all good.

Now let's emulate a hanging test by adding a sleep of 60 seconds in call_api

Oops:

(venv)[bobbelderbos@imac test-debug]$pytest=====================================================================testsessionstarts======================================================================platformdarwin-- Python 3.8.0, pytest-5.4.1, py-1.8.1, pluggy-0.13.1rootdir:/Users/bobbelderbos/code/test-debugplugins:timeout-1.3.4collected2itemstest_script.py.(hangshere)

Eventually you'll get there (2 passed in 60.06s (0:01:00)), but this might be far worse.

Even with 1 or 2 minutes delay, you need to fix it. Tests need to be fast, because you will constantly run them to refactor and adding new features.

Debugging timeouts

I discovered a neat plugin for this: pytest-timeout. With your virtualenv enabled:

pipinstallpytest-timeout

Let's use it:

(venv)[bobbelderbos@imac test-debug]$pytest--timeout=3test_script.py.F[100%]===========================================================================FAILURES===========================================================================________________________________________________________________________test_call_api_________________________________________________________________________deftest_call_api():>resp=call_api()test_script.py:10:________________________________________________________________________________defcall_api():>sleep(60)EFailed:Timeout>3.0sscript.py:5:Failed===================================================================shorttestsummaryinfo====================================================================FAILEDtest_script.py::test_call_api-Failed:Timeout>3.0s=================================================================1failed,1passedin3.07s

How cool is that, no?

A more complex use case

What if the bug is in some external module though?

Here I pulled a fork of requests and:

• added a import time; time.sleep(60) to the get function of the api module,

• added import requests; requests.get('https://pybit.es') to my call_api function,

• cd'd into the requests subfolder and did a pip install -e . to install THIS modified version (as opposed the one on the net).

  (venv)[bobbelderbos@imactest-debug]$pipfreeze|greprequest-egit+git@github.com:bbelderbos/requests.git@4bda7b66e7ece5be51b459edd046a70915b4792c#egg=requests

And ... boom!

(venv) [bobbelderbos@imactest-debug]$ pytest--timeout=3=====================================================================testsessionstarts======================================================================platformdarwin--Python3.8.0, pytest-5.4.1, py-1.8.1, pluggy-0.13.1rootdir: /Users/bobbelderbos/code/test-debugplugins: timeout-1.3.4timeout: 3.0stimeoutmethod: signaltimeoutfunc_only: Falsecollected2itemstest_script.py .F                                                                                                                                        [100%]

===========================================================================FAILURES===========================================================================________________________________________________________________________test_call_api_________________________________________________________________________deftest_call_api():
>resp=call_api()test_script.py:10:
________________________________________________________________________________script.py:5: incall_apirequests.get('https://pybit.es')________________________________________________________________________________url='https://pybit.es', params=None, kwargs= {}

    defget(url, params=None, **kwargs):
        r"""Sends a GET request.:paramurl: URLforthenew :class:`Request` object.
        :paramparams: (optional)Dictionaryorbytestobesentinthequerystringforthe :class:`Request`.
        :param \*\*kwargs: Optionalargumentsthat ``request`` takes.
        :return: :class:`Response<Response>` object
        :rtype: requests.Response""">time.sleep(60)EFailed: Timeout>3.0s

Getting more debug info

What if the failing code is more layers down?

Sometimes the issue is really nested and pytest-timeout might not be that smart.

In that case you can swap the default signal of the --timeout_method flag for thread and it will dump a complete stacktrace!

This can be very useful for debugging:

(venv)[bobbelderbos@imac test-debug]$pytest--timeout=3 --timeout_method=thread=====================================================================testsessionstarts======================================================================platformdarwin-- Python 3.8.0, pytest-5.4.1, py-1.8.1, pluggy-0.13.1rootdir:/Users/bobbelderbos/code/test-debugplugins:timeout-1.3.4timeout:3.0stimeoutmethod:thread<==using"thread now"timeoutfunc_only:Falsecollected2itemstest_script.py.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++Timeout++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~StackofMainThread(4523748800)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~...manycalls...File"/Users/bobbelderbos/code/test-debug/venv/lib/python3.8/site-packages/_pytest/python.py",line184,inpytest_pyfunc_callresult=testfunction(**testargs)File"/Users/bobbelderbos/code/test-debug/test_script.py",line10,intest_call_apiresp=call_api()File"/Users/bobbelderbos/code/test-debug/script.py",line5,incall_apirequests.get('https://pybit.es')File"/Users/bobbelderbos/Downloads/requests.org/requests/api.py",line71,inget<==externalmoduletime.sleep(60)+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++Timeout++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++(venv)[bobbelderbos@imac test-debug]$

Really cool stuff.

I hope this is useful. Comment below next time this saves you some debugging time ...

Keep Calm and Code in Python!

-- Bob

The post How to use Pandas Scatter Matrix (Pair Plot) to Visualize Trends in Data appeared first on Erik Marsja.

In this Python data visualization tutorial, we will work with Pandas scatter_matrix method to explore trends in data. Previously, we have learned how to create scatter plots with Seaborn and histograms with Pandas, for instance. In this post, we’ll focus on scatter matrices (pair plots) using Pandas. Now, Pandas is using Matplotlib to make the scatter matrix.

A scatter matrix (pairs plot) compactly plots all the numeric variables we have in a dataset against each other one. In Python, this data visualization technique can be carried out with many libraries but if we are using Pandas to load the data, we can use the base scatter_matrix method to visualize the dataset.

Prerequisites

Now, this Python data visualization tutorial will require that we have Pandas and its dependencies installed. It’s very easy to install Pandas. Either we use pip to install Python packages, such as Pandas, or we install a Python distribution (e.g., Anaconda, ActivePython). Here’s how to install Pandas with pip: pip install pandas.

Note, if a message that there’s a newer version of pip available check the post about how to upgrade pip.

Pandas scatter_matrix Syntax

In general, to create a scatter plot matrix with Pandas using the following syntax:

pandas.plotting.scatter_matrix(dataframe)

Now, there are, of course, a number of parameters we can use (see image below). In this Pandas scatter matrix tutorial, we are going to use hist_kwds, diagonal, and marker to create pair plots in Python. In the first example, however, we use the simple syntax of the scatter_matrix method (as above).

Data Simulation using Numpy

In this Pandas scatter matrix tutorial, we are going to create fake data to visualize. Here we will use NumPy to create 3 variables (x1, x2, and x3). Specifically, we use the normal method from random:

import numpy as np
import pandas as pd

np.random.seed(134)                     
N = 1000                              
 
x1 = np.random.normal(0, 1, N)                        
x2 = x1 + np.random.normal(0, 3, N)              
x3 = 2 * x1 - x2 + np.random.normal(0, 2, N)

Next step, before visualizing the data we create a Pandas dataframe from a dictionary.

df = pd.DataFrame({'x1':x1,
                   'x2':x2,
                   'x3':x3})

df.head()

Now, you can see that we have variables x1, x2, and x3 as columns. Normally, we would import data using Pandas read_csv or Pandas read_excel methods, for instance. See the summary, or the linked blog post, on how to do this.

Pandas scatter_matrix (pair plot) Example 1:

In the first Pandas scatter_matrix example, we will only use the created dataframe as input. Now, this will create the following pair plot:

pd.plotting.scatter_matrix(df)

As evident in the scatter matrix above, we are able to produce a relatively complex matrix of scatterplots and histograms using only one single line of code. Now, what does this pairs plot actually contain?

• The diagonal shows the distribution of the three numeric variables of our example data.
• In the other cells of the plot matrix, we have the scatterplots (i.e. correlation plot) of each variable combination of our dataframe. In the middle graphic in the first row we can see the correlation between x1 & x2. Furthermore, in the right graph in the first row we can see the correlation between x1 & x3; and finally, in the left cell in the second row, we can see the correlation between x1 & x2.

In this first example, we just went through the most basic usage of Pandas scatter_matrix method. In the following examples, we are going to modify the pair plot (scatter matrix) a bit…

Pandas scatter_matrix (pair plot) Example 2:

In the second example, on how to use Pandas scatter_matrix method to create a pair plot, we will use the hist_kwd parameter. Now, this parameter takes a Python dictionary as input. For instance, we can change the number of bins, in the histogram, by adding this to the code:

pd.plotting.scatter_matrix(df, hist_kwds={'bins':30})

Refer to the documentation of Pandas hist method for more information about keywords that can be used or check the post about how to make a Pandas histogram in Python.

Pandas scatter_matrix (pair plot) Example 3:

Now, in the third Pandas scatter matrix example, we are going to plot a density plot instead of a histogram. This is, also, very easy to accomplish. In the code chunk below, we added the diagonal parameter:

pd.plotting.scatter_matrix(df, diagonal='kde')

That produced a nice scatter matrix (pair plot) with density plots on the diagonal instead of a histogram. Note, that the diagonal parameter takes either “hist” or “kde” as an argument. Thus, if we wanted to have both density and histograms in our scatter matrix, we cannot.

Pandas scatter_matrix (pair plot) Example 4:

In the fourth Pandas scatter_matrix example, we are going to change the marker. This is accomplished by using the marker parameter:

pd.plotting.scatter_matrix(df, marker='+')

Scatter Matrix (pair plot) using other Python Packages

Now, there are some limitations to Pandas scatter_method. One limitation, for instance, is that we cannot plot both a histogram and the density of our data in the same plot. Another limitation is that we cannot group the data. Furthermore, we cannot plot the regression line in the scatter plot. However, if we use the Seaborn and the pairplot() method we can have more control over the scatter matrix. For instance, we can, using Seaborn pairplot() group the data, among other things. Another option is to use Plotly, to create the scatter matrix.

Summary: 3 Simple Steps to Create a Scatter Matrix with Pandas

In this post, we have learned how to create a scatter matrix (pair plot) with Pandas. It was super simple and here are three simple steps to use Pandas scatter_matrix method to create a pair plot:

Step 1: Load the Needed Libraries

In the first step, we will load pandas: import pandas as pd

Step 2: Import the Data to Visualize

In the second step, we will import data from a CSV file using Pandas read_csv method:

csv_file = 'https://vincentarelbundock.github.io/Rdatasets/csv/MASS/survey.csv'
df_s = pd.read_csv(csv_file, index_col=0)

df_s.head()

Step 3: Use Pandas scatter_matrix Method to Create the Pair Plot

In the final step, we create the pair plot using Pandas scatter_matrix method. Note, however, that we use Pandas iloc to select certain columns.

pd.plotting.scatter_matrix(df_s.iloc[:, 1:9])

Note, that in the pair plot above, Pandas scatter_matrix only chose the columns that have numerical values (from the ones we selected, of course). Here’s a Jupyter Notebook with all the code in this blog post.

The post How to use Pandas Scatter Matrix (Pair Plot) to Visualize Trends in Data appeared first on Erik Marsja.

How are you progressing with your Python? What could be holding you back?

I gave it some thought and identified 4 issues we commonly see that hold people back from becoming a proficient Pythonista and programmer.

1. Tutorial paralysis

This by far the pitfall we hear about the most: having to read 4+ books in order to start writing code. Wrong. These resources are most effective if you alternate them with writing code. So the sooner you write code and FAIL the better.

2. Improper sequencing

Taking too many steps at the same time.

I literally saw somebody in a group asking: "I need a learning path for ML. I don't know programming yet". My reply was: "Try to crawl then walk then run. Start with the basics of Python / programming first."

The same is true with other parts of your programming journey:

• Yes it's important to write thorough test code, but don't obsess over pytest features (yet), simply assert your function outputs for starters.

• Yes it's important to know data and web (where Python is used a lot), but learning a web framework without knowing at least the basics of classes, databases, some html/css, you set yourself up for a rough ride.

• Yes it's important to write clean and fast code, yet in your first iteration you should really focus on getting something to work first (sad realization: how many great apps were never written because their creators were stalled by perfectionism?).

• Yes you might be asked about binary search in a coding interview, but if you distill it down all complex problems are made up of simpler subproblems. So focus on the basic data structures/ concepts first: list vs dict, mutable vs immutable, etc. They form the fundamental building blocks of your programs.

We estimate that with ~20 well selected Bite exercises on our platform you will get further than banging your head against the wall trying to absorb a coding interview book (see 1.)

(Photo by Road Trip with Raj on Unsplash)

3. Obsessing over Pythonic code

As important it is to know to effectively use Python constructs and writing Pythonic code, you should equally invest in learning about software best practices:

• Use meaningful names

• Use proper scoping / modular code

• Write short and testable functions / methods

You can write very Pythonic code but if your functions are > 100 LOC and have a lot of cyclomatic complexity, you won't be (perceived as) an effective programmer.

Aim to read a software book each month and spend at least 1 hour a week reading top quality open source projects like requests, Flask, Django or OpenCV.

Awesome Python provides a curated list of awesome Python frameworks, libraries, software and resources. This is a great starting place and with the Github mobile app it's even easier now :)

4. Going on your own for too long

My first Python code I wrote as a support engineer among fellow support engineers.

I became the tool developer expert, the go-to guy. It felt great but cheerleaders are bad for growth. And ego is bad for growth. So I took an honest look:

• I was not exposed to a large professional code base.

• I did not use any advanced Python tooling.

• I was not surrounded by Python developers that were doing this for many many years.

As we always say at PyBites: if you're comfortable for too long, alarm bells should go off.

Remember: success leaves clues! Merely by surrounding yourself by experts you will inevitably grow (it's like gravity).

So we challenge you: become a small fish in a big pond again and your future self will thank you.

I hope this is helpful. Comment below if you see any other common mistakes people make when learning Python / programming.

Keep Calm and Code in Python!

-- Bob

“What I cannot create, I do not understand.” — Richard Feynman

As I promised you last time, today we’re going to expand on the material covered in the previous article and talk about executing nested procedure calls. Just like last time, we will limit our focus today to procedures that can access their parameters and local variables only. We will cover accessing non-local variables in the next article.

Here is the sample program for today:

programMain;procedureAlpha(a:integer;b:integer);varx:integer;procedureBeta(a:integer;b:integer);varx:integer;beginx:=a*10+b*2;end;beginx:=(a+b)*2;Beta(5,10);{ procedure call }end;begin{ Main }Alpha(3+5,7);{ procedure call }end.{ Main }

The nesting relationships diagram for the program looks like this:

Some things to note about the above program:

• it has two procedure declarations, Alpha and Beta

• Alpha is declared inside the main program (the global scope)

• the Beta procedure is declared inside the Alpha procedure

• both Alpha and Beta have the same names for their formal parameters: integers a and b

• and both Alpha and Beta have the same local variable x

• the program has nested calls: the Beta procedure is called from the Alpha procedure, which, in turn, is called from the main program

Now, let’s do an experiment. Download the part19.pas file from GitHub and run the interpreter from the previous article with the part19.pas file as its input to see what happens when the interpreter executes nested procedure calls (the main program calling Alpha calling Beta):

$ python spi.py part19.pas --stack

ENTER: PROGRAM Main
CALL STACK
1: PROGRAM Main


...


ENTER: PROCEDURE Beta
CALL STACK
2: PROCEDURE Beta
   a                   : 5
   b                   : 102: PROCEDURE Alpha
   a                   : 8
   b                   : 7
   x                   : 301: PROGRAM Main


LEAVE: PROCEDURE Beta
CALL STACK
2: PROCEDURE Beta
   a                   : 5
   b                   : 10
   x                   : 702: PROCEDURE Alpha
   a                   : 8
   b                   : 7
   x                   : 301: PROGRAM Main


...


LEAVE: PROGRAM Main
CALL STACK
1: PROGRAM Main

It just works! There are no errors. And if you study the contents of the ARs(activation records), you can see that the values stored in the activation records for the Alpha and Beta procedure calls are correct. So, what’s the catch then? There is one small issue. If you take a look at the output where it says ‘ENTER: PROCEDURE Beta’, you can see that the nesting level for the Beta and Alpha procedure call is the same, it’s 2 (two). The nesting level for Alpha should be 2 and the nesting level for Beta should be 3. That’s the issue that we need to fix. Right now the nesting_level value in the visit_ProcedureCall method is hardcoded to be 2 (two):

defvisit_ProcedureCall(self,node):proc_name=node.proc_namear=ActivationRecord(name=proc_name,type=ARType.PROCEDURE,nesting_level=2,)proc_symbol=node.proc_symbol...

Let’s get rid of the hardcoded value. How do we determine a nesting level for a procedure call? In the method above we have a procedure symbol and it is stored in a scoped symbol table that has the right scope level that we can use as the value of the nesting_level parameter (see Part 14 for more details about scopes and scope levels).

How do we get to the scoped symbol table’s scope level through the procedure symbol?

Let’s look at the following parts of the ScopedSymbolTable class:

classScopedSymbolTable:def__init__(self,scope_name,scope_level,enclosing_scope=None):...self.scope_level=scope_level...definsert(self,symbol):self.log(f'Insert: {symbol.name}')self._symbols[symbol.name]=symbol

Looking at the code above, we can see that we could assign the scope level of a scoped symbol table to a symbol when we store the symbol in the scoped symbol table (scope) inside the insert method. This way we will have access to the procedure symbol’s scope level in the visit_Procedure method during the interpretation phase. And that’s exactly what we need.

Let’s make the necessary changes:

• First, let’s add a scope_level member to the Symbol class and give it a default value of zero:

  classSymbol:def__init__(self,name,type=None):...self.scope_level=0

• Next, let’s assign the corresponding scope level to a symbol when storing the symbol in a scoped symbol table:

  classScopedSymbolTable:...definsert(self,symbol):self.log(f'Insert: {symbol.name}')symbol.scope_level=self.scope_levelself._symbols[symbol.name]=symbol

Now, when creating an AR for a procedure call in the visit_ProcedureCall method, we have access to the scope level of the procedure symbol. All that’s left to do is use the scope level of the procedure symbol as the value of the nesting_level parameter:

classInterpreter(NodeVisitor):...defvisit_ProcedureCall(self,node):proc_name=node.proc_nameproc_symbol=node.proc_symbolar=ActivationRecord(name=proc_name,type=ARType.PROCEDURE,nesting_level=proc_symbol.scope_level+1,)

That’s great, no more hardcoded nesting levels. One thing worth mentioning is why we put proc_symbol.scope_level + 1 as the value of the nesting_level parameter and not just proc_symbol.scope_level. In Part 17, I mentioned that the nesting level of an AR corresponds to the scope level of the respective procedure or function declaration plus one. Let’s see why.

In our sample program for today, the Alpha procedure symbol - the symbol that contains information about the Alpha procedure declaration - is stored in the global scope at level 1 (one). So 1 is the value of the Alpha procedure symbol’s scope_level. But as we know from Part14, the scope level of the procedure declaration Alpha is one less than the level of the variables declared inside the procedure Alpha. So, to get the scope level of the scope where the Alpha procedure’s parameters and local variables are stored, we need to increment the procedure symbol’s scope level by 1. That’s the reason we use proc_symbol.scope_level + 1 as the value of the nesting_level parameter when creating an AR for a procedure call and not simply proc_symbol.scope_level.

Let’s see the changes we’ve made so far in action. Download the updated interpreter and test it again with the part19.pas file as its input:

$ python spi.py part19.pas --stack

ENTER: PROGRAM Main
CALL STACK
1: PROGRAM Main


ENTER: PROCEDURE Alpha
CALL STACK
2: PROCEDURE Alpha
   a                   : 8
   b                   : 71: PROGRAM Main


ENTER: PROCEDURE Beta
CALL STACK
3: PROCEDURE Beta
   a                   : 5
   b                   : 102: PROCEDURE Alpha
   a                   : 8
   b                   : 7
   x                   : 301: PROGRAM Main


LEAVE: PROCEDURE Beta
CALL STACK
3: PROCEDURE Beta
   a                   : 5
   b                   : 10
   x                   : 702: PROCEDURE Alpha
   a                   : 8
   b                   : 7
   x                   : 301: PROGRAM Main


LEAVE: PROCEDURE Alpha
CALL STACK
2: PROCEDURE Alpha
   a                   : 8
   b                   : 7
   x                   : 301: PROGRAM Main


LEAVE: PROGRAM Main
CALL STACK
1: PROGRAM Main

As you can see from the output above, the nesting levels of the activation records (AR) now have the correct values:

• The Main program AR: nesting level 1

• The Alpha procedure AR: nesting level 2

• The Beta procedure AR: nesting level 3

Let’s take a look at how the scope tree (scoped symbol tables) and the call stack look visually during the execution of the program. Here is how the call stack looks right after the message “LEAVE: PROCEDURE Beta” and before the AR for the Beta procedure call is popped off the stack:

And if we flip the call stack (so that the top of the stack is at the “bottom”), you can see how the call stack with activation records relates to the scope tree with scopes (scoped symbol tables). In fact, we can say that activation records are run-time equivalents of scopes. Scopes are created during semantic analysis of a source program (the source program is read, parsed, and analyzed at this stage, but not executed) and the call stack with activation records is created at run-time when the interpreter executes the source program:

As you’ve seen in this article, we haven’t made a lot of changes to support the execution of nested procedure calls. The only real change was to make sure the nesting level in ARs was correct. The rest of the codebase stayed the same. The main reason why our code continues to work pretty much unchanged with nested procedure calls is because the Alpha and Beta procedures in the sample program access the values of local variables only (including their own parameters). And because those values are stored in the AR at the top of the stack, this allows us to continue to use the methods visit_Assignment and visit_Var without any change, when executing the body of the procedures. Here is the source code of the methods again:

defvisit_Assign(self,node):var_name=node.left.valuevar_value=self.visit(node.right)ar=self.call_stack.peek()ar[var_name]=var_valuedefvisit_Var(self,node):var_name=node.valuear=self.call_stack.peek()var_value=ar.get(var_name)returnvar_value

Okay, today we’ve been able to successfully execute nested procedure calls with our interpreter with very few changes. And now we’re one step closer to properly executing recursive procedure calls.

That’s it for today. In the next article, we’ll talk about how procedures can access non-local variables during run-time.

Stay safe, stay healthy, and take care of each other! See you next time.

Resources used in preparation for this article (links are affiliate links):

1. Language Implementation Patterns: Create Your Own Domain-Specific and General Programming Languages (Pragmatic Programmers)
2. Writing Compilers and Interpreters: A Software Engineering Approach
3. Programming Language Pragmatics, Fourth Edition

We have a new Beta version of PyCharm that can now be downloaded from our website.

This Beta brings us closer to the 2020.1 release: we’re working on polishing everything to get it ready, and this week’s version brings some great improvements.

New in PyCharm

Creating new Pyramid projects with Cookiecutter

Pyramid has adopted cookiecutter as the official way to create new projects. From this version onward, PyCharm will also use the official cookiecutter template to create Pyramid projects.

Further Improvements

• We recently improved how stepping works in the Python debugger. Previously we had a separate ‘Smart step into’ option that allows you to which function call you’d like to step into if there are multiple function calls on the same line. Unfortunately, we had a small issue where in some cases we skipped a couple lines while stepping. This has now been fixed.
• In PyCharm Professional Edition’s data view, we had an issue where if there were multiple columns, all values were implicitly converted to floats. We’ve resolved this problem.
• There is all this and more. You can find the details in the release notes.

Interested?

Download this EAP from our website. Alternatively, you can use the JetBrains Toolbox App to stay up to date throughout the entire EAP.
If you’re on Ubuntu 16.04 or later, you can use snap to get PyCharm EAP and stay up to date. You can find the installation instructions on our website.

We are pleased to announce that the March 2020 release of the Python Extension for Visual Studio Code is now available. You can download the Python extension from the Marketplace, or install it directly from the extension gallery in Visual Studio Code. If you already have the Python extension installed, you can also get the latest update by restarting Visual Studio Code. You can learn more about  Python support in Visual Studio Code in the documentation.  

This release is focused mostly on product quality. We closed a total of 66 issues, 43 of them being bug fixes.  But we’re also pleased to include a brand-new Python debugger: debugpy!  

If you’re interested, you can check the full list of improvements in our changelog. 

New Debugger 

We’re excited to announce that in this release we’re including a new debugger, debugpy. The debugger team has put a lot of effort into making it a faster and even more reliable Python debugger. Along with the debugger, a new feature also comes: an easier configuration experience to attach the debugger to local processes. 

Attaching to local processes 

Sometimes you may want to attach the debugger to a Python process that is running on your machine, but that can be tricky if, for example, you don’t have control over the application that launched that process.  

We made it easy to be done with our new configuration experience for attaching the debugger to local processes.  

If you don’t have a launch.json file on your workspace folder, you can simply start a debug session (by pressing F5 or through Run > Start Debugging) and you’ll be presented with a list of debug configuration options. When you select “Attach using Process ID”, it will display a list of processes running locally on your machine: 

Alternatively, if you already have a launch.json file on your workspace folder, you can add a configuration to it by clicking on the “Add configuration…” option under the drop-down menu in the Run viewlet: 

Then when you select “Python”, you’ll be presented with the same configuration options as above: 

Selecting the “Attach using Process ID” option from the debug configuration menu adds the below configuration to the existing launch.json file: 

{ 

            "name": "Python: Attach using Process Id", 

            "type": "python", 

            "request": "attach", 

            "processId": "${command:pickProcess}" 

}

When you start a debug session with this configuration selected, a list of processes to which you can attach the debugger will be displayed, and once you pick one, the debugger will attempt to attach to it: 

You can also filter the processes by ID, file name or interpreter name: 

Alternatively, if you already know the ID of the process to which you wish to attach the debugger, you can simply add the value directly on the configuration. For example, to attach to a process of ID 1796, you can simply use the below configuration: 

{ 

            "name": "Python: Attach using Process Id", 

            "type": "python", 

            "request": "attach", 

            "processId": 1796 

}

For more information about debugpy such as how to transition from ptvsd, API changes, CLI references, allowed debug configurations and more, you can check debugpy’s wiki page.   

Other Changes and Enhancements 

We have also added small enhancements and fixed issues requested by users that should improve your experience working with Python in Visual Studio Code. Some notable changes include: 

• Remove extra lines at the end of the file when formatting with Black. (#1877) 
• Support scrolling beyond the last line in the notebook editor and the interactive window. (#7892) 
• Added a command to allow users to select a kernel for a Notebook. (#9228) 
• Show quickfixes for launch.json. (#10245) 
• Update Jedi to 0.16.0. (#9765) 

We’re constantly A/B testing new features. If you see something different that was not announced by the team, you may be part of the experiment! To see if you are part of an experiment, you can check the first lines in the Python extension output channel. If you wish to opt-out of A/B testing, you can open the user settings.json file (View > Command Palette… and run Preferences: Open Settings (JSON)) and set the “python.experiments.enabled” setting to false.  

Be sure to download the Python extension for Visual Studio Code now to try out the above improvements. If you run into any problems, please file an issue on the Python VS Code GitHub page. 

The post Python in Visual Studio Code – March 2020 Release appeared first on Python.

Here we are at the uptick in the Covid 19 pandemic. There are many sources of data which list infections and deaths as a result of the virus. It’s very tempting to want to put your Python skills to use and crunch some numbers on the infection. By and large, go for it, but one thing I’d ask you not to do is to try to calculate a “mortality rate”. This is not because Python can’t do division but, rather, working this number out is conceptually pretty tricky. It’s something that epidemiologists need to get a lot of training in to do correctly.  You can’t just take the deaths column and divided it by the infected column because the two numbers are not properly related. For example:

• Testing is incomplete. There is a shortage of test kits where I am. So, if you present with symptoms you will not get tested unless you meet the testing protocol – that is: have you been overseas in the last 14 days; or have you been in contact with a known Covid19 case. This testing protocol means that (most) community transmission of the virus hereabouts is not included in the numbers.
• There is evidence to believe that a large cohort of those infected are asymptomatic. That is, they have no symptoms or very mild symptoms. If that’s the case, then there is a cohort of infected people who don’t feel unwell, so they don’t get tested and are, also, omitted from the infection numbers.

These factors will mean that naive division of the reported numbers will inflate the mortality rate, making it seem worse, possibly much worse, than it really is (the Economist argues [paywall] that places with extensive testing have much lower death rates – by a factor of 5 or so – simply because they are identifying more of those infected).

Ideally, if you’re going to publish these numbers make it clear what their limitations are.

PS: The Diamond Princess is probably the only cohort to have reliable infection numbers, since everyone was tested before leaving the ship. However, their mortality rate shouldn’t be used as they’re not a representative sample (ie mostly older people who are fit enough and wealthy enough to go on a cruise).

Earl Zope has settled down for a good while in Python 3 wonderland. He made friends with the inhabitants and other immigrants. He enjoys his new live.

The sunset of his original homelands took place as predicted by the beginning of January 2020. As Earl Zope was well prepared this was no longer a frightening date for him.

But even living in Python 3 wonderland is not only joy and relaxing. The Python 3 wonderland changes in a more rapid speed than the Python 2 land ever had before: Each year a new policy has to be fulfilled (aka new Python version release). Additionally it is time to drop the last connections to the old Python 2 land to ease the transformation in Python 3 wonderland to make developers and consumers happy.

Earl Zope is grateful for all the help he already gained: There where several Zope 4 releases and a first Zope 5 alpha version was just released. Even though Earl Zope still needs your help to:

• prepare dependencies to ease transition to new Python versions (aka make repositories more uniform to ease updating to new Python versions.)
• drop Python 2 support in repositories of dependencies
• support and test with newer Python 3 versions (aka current 3.9 alpha)
• improve and update the documentation

You are invited to the “Zope May sprint” located in Halle/Saale, from 13th till 15th of May 2020 hosted by gocept. In order to coordinate the participation for this sprint, we ask you to join us on Meetup. We can then coordinate the catering and requirements for space.

Update: This sprint will be held als a remote-only sprint. Details see Zope May sprint goes remote.

Scary times. We hope everyone reading this is well and keeping safe!

We thought it would be a good idea to tell you how we're managing the current crisis at PythonAnywhere. We switched over to remote working last Thursday, 12 March; there are obviously private and public health reasons why that was a good idea, but there's a reason specific to us, which we thought would be worth sharing.

Most of the team here are lucky enough to be in low risk categories, but we pair-program -- that is, we have two people working together at the same computer, all day and every day. Each day we rotate the pairs around, so that the same two people are never working together on two consecutive days. This makes sure that we spread knowledge around the team.

Unfortunately, that means that if someone caught COVID-19, we'd share that pretty quickly around the team too. And that would be bad, not just for us, but for the people who use our service. We can keep the servers running, tech support operating, and so on in the short term with just a small subset of the team, but that would be no good if all of us were unwell simultaneously, as even a mild infection could make it impossible for someone to help with tech support or system maintenance.

Right now, we're all working from home, sharing tech support, and pair-programming using shared consoles and voice chat as a way to keep development going. Tech support is going fine -- we're used to doing that from home, as that's what we do at weekends. Development is a little slower right now; over time we will probably refine the setup we're using, and maybe we'll even come up with some nice ideas for new features for PythonAnywhere to make it easier for other people to do the same kind of thing :-)

Fingers crossed, and good luck to everyone. These are challenging times -- stay safe, and, as we say when we're updating our service, we'll see you on the other side.

What happens next?

Here’s our immediate plan:

Attendees

• Registration: We will send an email by Friday, March 27th offering you a full refund, or the option to donate some or all of your registration fee, and how to join the remote content mailing list. Please allow us time to set up the registration refund process.
• Hotel/lodging: We will cancel hotel reservations booked through PyCon US’s website for you. If you booked elsewhere, please contact the booking company for cancellation details. You should be able to cancel at this time without any financial penalty. 
• Airfare/transportation: Many airlines are offering options to rebook tickets or receive vouchers without extra fees. Please see our airline cancellation resources page.

Financial aid recipients

Speakers, Presenters, Summit & Hatchery Hosts

Sponsors & Startup Row Participants

What does this cancellation mean for the Python Software Foundation?

• Become a free or supporting member of the PSF to get involved in our future
• Donate some or all of your PyCon registration when you receive our email
• Donate directly to the PSF
• Donate some or all of your company's sponsorship when you receive our email
• Ask your employer to sponsor the PSF
• Ask your employer if they match donations to 501(c)(3) non-profits, and ask for your donations to the PSF to be matched
• Sign up for the PSF’s free newsletter to stay up to date

It is with a sincere heart that we have decided to postpone DjangoCon Europe 2020 to September 16-20.

As you might be aware, conferences are being cancelled worldwide. We still have hope, and before throwing the towel, we have decided to postpone. We took particular care to choose safe dates, far enough from the current outbreak peek estimates, but also far from the next fall/winter. Some experts state this is here to stay, and if they are correct, we should have troubles next fall/winter, let's just hope to a smaller degree.

Many of you have already bought tickets, and to those, we kindly ask to hope with us.

Ticket refunds will happen in any of the following scenarios:

• Participant's inability to participate due to restrictions in place at the time or unavailability on the new dates.
• Conference cancellation

Please do not rush requesting ticket refunds. We will process them as we can. Please note that a ticket refunded and latter on bought again has double the fees for us. So delay your decision for as long as you can. If everything goes well, we will have a conference, and we count on you to be there! When booking your trip to Porto for September, please make sure to book with the possibility to cancel.

For those of you who are planning or have already booked your flights with TAP, our official carrier, here is some crucial information. In short, they have plans for flight reschedules free of charge. You can reschedule your tickets to the new dates.

During the conference, we will follow the World Health Organization (WHO) guidelines, as well as the Portuguese Direcção Geral da Saude (DGS) guidelines.

Thank you for your understanding and patience during this uncertain time. We appreciate your support of this difficult decision and look forward to seeing you soon in September.

Hopping for the best,

The DjangoCon Europe 2020 Organisers