dynaconf - The dynamic configurator for your Python Project

dynaconf a layered configuration system for Python applications - with strong support for 12-factor applications and extensions for Flask and Django.

Features

• Strict separation of settings from code (following 12-factor applications Guide).
• Define comprehensive default values.
• Store parameters in multiple file formats (.toml, .json, .yaml, .ini and .py).
• Sensitive secrets like tokens and passwords can be stored in safe places like .secrets file or vault server.
• Parameters can optionally be stored in external services like Redis server.
• Simple feature flag system.
• Layered [environment] system.
• Environment variables can be used to override parameters.
• Support for .env files to automate the export of environment variables.
• Correct data types (even for environment variables).
• Have only one canonical settings module to rule all your instances.
• Drop in extension for Flaskapp.config object.
• Drop in extension for Djangoconf.settings object.
• Powerful $ dynaconf CLI to help you manage your settings via console.
• Customizable Validation System to ensure correct config parameters.
• Allow the change of dynamic parameters on the fly without the need to redeploy your application.
• Easily extensible
• 100% test coverage
• 100% documented

Read the Full Documentation at: http://dynaconf.readthedocs.io/

Repository: http://github.com/rochacbruno/dynaconf/

Getting Started

Installation

Python 3.x is required

$ pip install dynaconf

Default installation supports .toml, .py and .json file formats and also environment variables (.env supported)

Usage

Accessing config variables in your Python application

In your Python program wherever you need to access a settings variable you use the canonical object from dynaconf import settings:

NOTE: Read the full documentation for more examples like using Dynaconf with Flask or Django

Example of program to connect to some database

from some.db import Client

from dynaconf import settings               # The only object you need to import

conn = Client(
    username=settings.USERNAME,             # attribute style access
    password=settings.get('PASSWORD'),      # dict get style access
    port=settings['PORT'],                  # dict item style access
    timeout=settings.as_int('TIMEOUT'),     # Forcing casting if needed
    host=settings.get('HOST', 'localhost')  # Providing defaults
)

Where the settings values are stored

Dynaconf aims to have a flexible and usable configuration system. Your applications can be configured via a configuration files, through environment variables, or both. Configurations are separated into environments: [development], [staging], [testing] and [production]. The working environment is selected via an environment variable.

Sensitive data like tokens, secret keys and password can be stored in .secrets.* files and/or external storages like Redis or vault secrets server.

Besides the built-in optional support to redis as settings storage dynaconf allows you to create custom loaders and store the data wherever you want e.g: databases, memory storages, other file formats, nosql databases etc.

Working environments

At any point in time, your application is operating in a given configuration environment. By default there are four such environments:

• [development]
• [staging]
• [testing]
• [production]

You can also define [custom environment] and use the pseudo-envs [default] to provide comprehensive default values and [global] to provide global values to overrride in any other environment.

Without any action, your applications by default run in the [development] environment. The environment can be changed via the ÈNV_FOR_DYNACONF environment variable. For example, to launch an application in the [staging] environment, we can run:

export ENV_FOR_DYNACONF=staging

or

ENV_FOR_DYNACONF=staging python yourapp.py

NOTE: When using FLask Extension the environment can be changed via FLASK_ENV variable and for Django Extension you can use DJANGO_ENV.

The settings files

NOTE:Read the full documentaion about dynaconf CLI to learn how to automatically create the settings files for your project.

An optional settings.{toml|py|json|ini|yaml} file can be used to specify the configuration parameters for each environment. If it is not present, only the values from environment variables are used (.env file is also supported). Dynaconf searches for the file starting at the current working directory. If it is not found there, Dynaconf checks the parent directory. Dynaconf continues checking parent directories until the root is reached.

The recommended file format is TOML but you can choose to use any of .{toml|py|json|ini|yaml}.

The file must be a series of sections, at most one for [default], optionally one for each [environment], and an optional [global] section. Each section contains key-value pairs corresponding to configuration parameters for that [environment]. If a configuration parameter is missing, the value from [default] is used. The following is a complete settings.toml file, where every standard configuration parameter is specified within the [default] section:

NOTE: if the file format choosen is .py as it does not support sections you can create multiple files like settings.py for [default], development_settings.py, production_settings.py and global_settings.py. ATTENTION using .py is not recommended for configuration use TOML!

[default]
username = "admin"
port = 5000
host = "localhost"
message = "default message"
value = "default value"

[development]
username = "devuser"

[staging]
host = "staging.server.com"

[testing]
host = "testing.server.com"

[production]
host = "server.com"

[awesomeenv]
value = "this value is set for custom [awesomeenv]"

[global]
message = "This value overrides message of default and other envs"

The [global] pseudo-environment can be used to set and/or override configuration parameters globally. A parameter defined in a [global] section sets, or overrides if already present, that parameter in every environment. For example, given the following settings.toml file, the value of address will be "1.2.3.4" in every environment:

[global]
address = "1.2.3.4"

[development]
address = "localhost"

[production]
address = "0.0.0.0"

NOTE: The [env] name and first level variables are case insensitive as internally dynaconf will always use upper case, that means [development] and [DEVELOPMENT] are equivalent and address and ADDRESS are also equivalent. This rule does not apply for inner data structures as dictionaries and arrays.

Supported file formats

By default toml is the recommended format to store your configuration, however you can switch to a different supported format.

# If you wish to include support for more sources
pip3 install dynaconf[yaml|ini|redis|vault]

# for a complete installation
pip3 install dynaconf[all]

Once the support is installed no extra configuration is needed to load data from those files, dynaconf will search for settings files in the root directory of your application looking for the following files in the exact order below:

DYNACONF_LOADING_ORDER = [
 'settings.py',
 '.secrets.py',
 'settings.toml',
 '.secrets.toml',
 'settings.yaml',
 '.secrets.yaml',
 'settings.ini',
 '.secrets.ini',
 'settings.json',
 '.secrets.json',
 # redis server if REDIS_ENABLED_FOR_DYNACONF=true
 # vault server if VAULT_ENABLED_FOR_DYNACONF=true
 # other sources if custom loaders are defined
 # All environment variables prefixed with DYNACONF_
]

NOTE: Dynaconf works in an layered override mode based on the above order, so if you have multiple file formats with conflicting keys defined, the precedence will be based on the loading order.

Take a look at the example folder to see some examples of use with different file formats.

Sensitive secrets

Using .secrets files

To safely store sensitive data Dynaconf also searches for a .secrets.{toml|py|json|ini|yaml} file to look for data like tokens and passwords.

example .secrets.toml:

[default]
password = "sek@987342$"

The secrets file supports all the environment definitions supported in the settings file.

IMPORTANT: The reason to use a .secrets.* file is the ability to omit this file when commiting to the repository so a recommended .gitignore should include .secrets.* line.

Using Vault server

The vaultproject.io/ is a key:value store for secrets and Dynaconf can load variables from a Vault secret.

1. Run a vault server

Run a Vault server installed or via docker:

$ docker run -d -e 'VAULT_DEV_ROOT_TOKEN_ID=myroot' -p 8200:8200 vault

1. Install support for vault in dynaconf

$ pip install dynaconf[vault]

1. In your .env file or in exported environment variables define:

VAULT_ENABLED_FOR_DYNACONF=true
VAULT_URL_FOR_DYNACONF="http://localhost:8200"
VAULT_TOKEN_FOR_DYNACONF="myroot"

Now you can have keys like PASSWORD and TOKEN defined in the vault and dynaconf will read it.

To write a new secret you can use http://localhost:8200 web admin and write keys under the /secret/dynaconf secret database.

You can also use the Dynaconf writer via console

$ dynaconf write vault -s password=123456

Environment variables

overloading parameters via env vars

All configuration parameters, including custom environments and dynaconf configuration, can be overridden through environment variables.

To override the configuration parameter {param}, use an environment variable named DYNACONF_{PARAM}. For instance, to override the "HOST" configuration parameter, you can run your application with:

DYNACONF_HOST='otherhost.com' python yourapp.py

.env files

If you don't want to declare the variables on every program call you can run export DYNACONF_{PARAM} in your shell or put the values in a .env file located in the same directory as your settings files (the root directory of your application), variables in .env does not overrride existing environment variables.

IMPORTANT: Dynaconf will search for a .env located in the root directory of your application, if not found it will continue searching in parent directories until it reaches the root. To avoid conflicts we recommend to have a .env even if it is empty.

Precedence and type casting

Environment variables take precedence over all other configuration sources: if the variable is set, it will be used as the value for the parameter even if parameter exists in settings files or in .env.

Variable values are parsed as if they were TOML syntax. As illustration, consider the following examples:

# Numbers
DYNACONF_INTEGER=42
DYNACONF_FLOAT=3.14

# Text
DYNACONF_STRING=Hello
DYNACONF_STRING="Hello"

# Booleans
DYNACONF_BOOL=true
DYNACONF_BOOL=false

# Use extra quotes to force a string from other type
DYNACONF_STRING="'42'"
DYNACONF_STRING="'true'"

# Arrays must be homogenous in toml syntax
DYNACONF_ARRAY=[1, 2, 3]
DYNACONF_ARRAY=[1.1, 2.2, 3.3]
DYNACONF_ARRAY=['a', 'b', 'c']

# Dictionaries
DYNACONF_DICT={key="abc",val=123}

# toml syntax does not allow `None/null` values so use @none
DYNACONF_NONE='@none None'

# toml syntax does not allow mixed type arrays so use @json
DYNACONF_ARRAY='@json [42, 3.14, "hello", true, ["otherarray"], {"foo": "bar"}]'

NOTE: Older versions of Dynaconf used the @casting prefixes for env vars like export DYNACONF_INTEGER='@int 123' still works but this casting is deprecated in favor of using TOML syntax described above. To disable the @casting do export AUTO_CAST_FOR_DYNACONF=false

The global prefix

The DYNACONF_{param} prefix is set by GLOBAL_ENV_FOR_DYNACONF and serves only to be used in environment variables to override config values.

This prefix itself can be changed to something more significant for your application, however we recommend kepping DYNACONF_{param} as your global env prefix.

NOTE: See the Configuring dynaconf section in documentation to learn more on how to use .env variables to configure dynaconf behavior.

Flask Extension

Dynaconf provides a drop in replacement for app.config.

As Flask encourages the composition by overriding the config_class attribute this extension follows the patterns of Flask and turns your Flask's app.config in to a dynaconf instance.

Initialize the extension

Initialize the FlaskDynaconf extension in your app

from flask import Flask
from dynaconf import FlaskDynaconf

app = Flask(__name__)
FlaskDynaconf(app)

You can optionally use init_app as well.

Use FLASK_ environment variables

Then the app.config will work as a dynaconf.settings instance and FLASK_ will be the global prefix for exporting environment variables.

Example:

export FLASK_DEBUG=true              # app.config.DEBUG
export FLASK_INTVALUE=1              # app.config['INTVALUE']
export FLASK_MAIL_SERVER='host.com'  # app.config.get('MAIL_SERVER')

Settings files

You can also have settings files for your Flask app, in the root directory (the same where you execute flask run) put your settings.toml and .secrets.toml files and then define your environments [default], [development] and [production].

To switch the working environment the FLASK_ENV variable can be used, so FLASK_ENV=development to work in development mode or FLASK_ENV=production to switch to production.

IMPORTANT: To use $ dynaconf CLI the FLASK_APP must be defined.

IF you don't want to manually create your config files take a look at the CLI

Django Extension

Dynaconf a drop in replacement to django.conf.settings.

Following this pattern recommended pattern this extension makes your Django's conf.settings in to a dynaconf instance.

Initialize the extension

In your django project's settings.py include:

INSTALLED_APPS = [
    'dynaconf.contrib.django_dynaconf',
    ...
]

NOTE: The extension must be included as the first INSTALLED_APP of the list

Use DJANGO_ environment variables

Then django.conf.settings will work as a dynaconf.settings instance and DJANGO_ will be the global prefix to export environment variables.

Example:

export DJANGO_DEBUG=true     # django.conf.settings.DEBUG
export DJANGO_INTVALUE=1     # django.conf.settings['INTVALUE]
export DJANGO_HELLO="Hello"  # django.conf.settings.get('HELLO)

Settings files

You can also have settings files for your Django app, in the root directory (the same where manage.py is located) put your settings.toml and .secrets.toml files and then define your environments [default], [development] and [production].

To switch the working environment the DJANGO_ENV variable can be used, so DJANGO_ENV=development to work in development mode or DJANGO_ENV=production to switch to production.

IMPORTANT: To use $ dynaconf CLI the DJANGO_SETTINGS_MODULE must be defined.

IF you don't want to manually create your config files take a look at the CLI

NOTE: It is recommended that all the django's internal config vars should be kept in the settings.py of your project, then application specific values you can place in dynaconf's settings.toml in the root (same folder as manage.py). You can override settings.py values in the dynaconf settings file as well.

How to contribute

In github repository issues and Pull Request are welcomed!

• New implementations
• Bug Fixes
• Bug reports
• More examples of use in /example folder
• Documentation
• Feedback as issues and comments ot joining dynaconf on Telegram or #dynaconf on freenode
• Donation to rochacbruno [at] gmail.com in PayPal

Read the docs

Documentation: http://dynaconf.readthedocs.io/

Repository: http://github.com/rochacbruno/dynaconf/

# File: isprime.py
# A port of a primality checking algorithm from the Q language to Python, 
# plus a few improvements / variations, using Python features.
# Ref: https://en.wikipedia.org/wiki/Q_(programming_language_from_Kx_Systems)
# Search for the word "prime" in that page to see the Q code.

# Author: Vasudev Ram
# Copyright 2018 Vasudev Ram
# Web site: https://vasudevram.github.io
# Blog: https://jugad2.blogspot.com
# Product store: https://gumroad.com/vasudevram

from __future__ import print_function
from debug1 import debug1

import sys

# Naive, mostly procedural port from the Q version.
def is_prime_1(x):
    # Guard against invalid argument.
    assert x > 2, "in is_prime_1: x > 2 failed"
    # The range of integers from 0 to x - 1
    til_x = range(x)
    # The range without the first 2 items, 0 and 1.
    divs = til_x[2:]
    # The remainders after dividing x by each integer in divs.
    mods = map(lambda d: x % d, divs)
    # x is prime if the minimum-valued remainder equals 1.
    return min(mods) == 1

# Shorter, more functional version, with nested calls 
# to min, map and range.
def is_prime_2(x):
    assert x > 2, "in is_prime_2: x > 2 failed"
    # Eliminate slicing used in is_prime_1, just do range(2, x).
    return min(map(lambda d: x % d, range(2, x))) == 1

# Early-terminating version, when 1st remainder equal to 0 found, 
# using a list for the range of divisors.
def is_prime_3(x):
    assert x > 2, "in is_prime_3: x > 2 failed"
    divs = range(2, x)
    # Check if x is divisible by any integer in divs; if so, 
    # x is not prime, so terminate early.
    debug1("in is_prime_3, x", x)
    for div in divs:
        debug1("  in loop, div", div)
        if x % div == 0:
            return False
    # If we reach here, x was not divisible by any integer in 
    # 2 to x - 1, so x is prime.
    return True

# Generator function to yield the divisors one at a time, to 
# avoid creating the whole list of divisors up front.
def gen_range(start, x):
    assert start > 0, "in gen_range, start > 0 failed"
    assert x > start, "in gen_range, x > start failed"
    i = start
    while i < x:
        yield i
        i += 1

# Early-terminating version, when 1st remainder equal to 0 found, 
# using a generator for the range of divisors.
def is_prime_4(x):
    assert x > 2, "in is_prime_4, x > 2 failed"
    divs = gen_range(2, x)
    debug1("in is_prime_4, x", x)
    for div in divs:
        debug1("  in loop, div", div)
        if x % div == 0:
            return False
    return True

def check_primes(low, high):
    assert low <= high, "in check_primes, low <= high failed"

"""
    print("\nWith a for loop:")
    for x in range(low, high + 1):
        print(x, "is" if is_prime_1(x) else "is not", "prime,", end="")
    print()
"""

    print("\nWith nested function calls:")
    output = [ str(x) + (" prime" if is_prime_2(x) else " not prime") \
        for x in range(low, high + 1) ]
    print(", ".join(output))

    print("\nWith a list of divisors and early termination:")
    output = [ str(x) + (" prime" if is_prime_3(x) else " not prime") \
        for x in range(low, high + 1) ]
    print(", ".join(output))

    print("\nWith a generator of divisors and early termination:")
    output = [ str(x) + (" prime" if is_prime_4(x) else " not prime") \
        for x in range(low, high + 1) ]
    print(", ".join(output))

def main():
    try:
        low = int(sys.argv[1])
        high = int(sys.argv[2])
        if low <= 2:
            print("Error: Low value must be > 2.")
            sys.exit(1)
        if high < low:
            print("Error: High value must be >= low value.")
            sys.exit(1)
        print("Checking primality of integers between {} and {}".format(low, high))
        check_primes(low, high)
        sys.exit(0)
    except ValueError as ve:
        print("Caught ValueError: {}".format(str(ve)))
    except IndexError as ie:
        print("Caught IndexError: {}".format(str(ie)))
    sys.exit(1)

if __name__ == '__main__':
    main()

Try this one later (if you're trying out the program), since it takes 
longer to run. You may observe some interesting behavior:

$ python -O is_prime.py 3 10000 | less

where less is the Unix 'less' command, a text pager. Any command-line text pager 
that can read standard input (from a pipe) will work.

Try some of these below (both normal and error cases) before the one above:

Some error-handling cases:

$ python -O is_prime.py 0 0
Error: Low value must be > 2.

$ python -O is_prime.py 2 0
Error: Low value must be > 2.

$ python -O is_prime.py 3 0
Error: High value must be >= low value.

$ python -O is_prime.py 4 2
Error: High value must be >= low value.

Some normal cases:

$ python -O is_prime.py 3 3
Checking primality of integers between 3 and 3

With nested function calls:
3 prime

With a list of divisors and early termination:
3 prime

With a generator of divisors and early termination:
3 prime

To show that the early termination logic works, run the program without 
the -O option. 
Here is one such run. Due to more debugging output, I've only checked 
two numbers, 4 and 5. But you can try with any number of values if you 
page the output, or redirect it to a file.

$ python is_prime.py 4 5
Checking primality of integers between 4 and 5

With nested function calls:
4 not prime, 5 prime

With a list of divisors and early termination:
in is_prime_3, x: 4
  in loop, div: 2
in is_prime_3, x: 5
  in loop, div: 2
  in loop, div: 3
  in loop, div: 4
4 not prime, 5 prime

With a generator of divisors and early termination:
in is_prime_4, x: 4
  in loop, div: 2
in is_prime_4, x: 5
  in loop, div: 2
  in loop, div: 3
  in loop, div: 4
4 not prime, 5 prime

You can see from the above run that for 4, the checking stops early, 
at the first divisor (2), in fact, because it evenly divides 4.
But for 5, all divisors from 2 to 4 are checked, because 5 has 
no prime factors (except itself and 1).

And here is a run checking for primes between 3 and 30:

$ python -O is_prime.py 3 30
Checking primality of integers between 3 and 30

With nested function calls:
3 prime, 4 not prime, 5 prime, 6 not prime, 7 prime, 8 not prime, 9 not prime, 
10 not prime, 11 prime, 12 not prime, 13 prime, 14 not prime, 15 not prime, 
16 not prime, 17 prime, 18 not prime, 19 prime, 20 not prime, 21 not prime, 
22 not prime, 23 prime, 24 not prime, 25 not prime, 26 not prime, 27 not prime, 
28 not prime, 29 prime, 30 not prime

With a list of divisors and early termination:
3 prime, 4 not prime, 5 prime, 6 not prime, 7 prime, 8 not prime, 9 not prime, 
10 not prime, 11 prime, 12 not prime, 13 prime, 14 not prime, 15 not prime, 
16 not prime, 17 prime, 18 not prime, 19 prime, 20 not prime, 21 not prime, 
22 not prime, 23 prime, 24 not prime, 25 not prime, 26 not prime, 27 not prime, 
28 not prime, 29 prime, 30 not prime

With a generator of divisors and early termination:
3 prime, 4 not prime, 5 prime, 6 not prime, 7 prime, 8 not prime, 9 not prime, 
10 not prime, 11 prime, 12 not prime, 13 prime, 14 not prime, 15 not prime, 
16 not prime, 17 prime, 18 not prime, 19 prime, 20 not prime, 21 not prime, 
22 not prime, 23 prime, 24 not prime, 25 not prime, 26 not prime, 27 not prime, 
28 not prime, 29 prime, 30 not prime

You can see that all three functions (is_prime_2 to 4) give the same results. 
(I commented out the call to the naive function version, is_prime_1, after 
a few runs (not shown), so none of these outputs shows its results, but they 
are the same as the others, except for minor formatting differences, due to 
the slightly different output statements used.

I also timed the program for finding primes up to 1000 and 10,000 
(using my own simple command timing program written in Python - not shown).

Command: python -O is_prime.py 3 1000
Time taken: 2.79 seconds
Return code: 0

Command: python -O is_prime.py 3 10000
Time taken: 66.28 seconds
Return code: 0

The development of Tryton is back on the road after the release of the 4.8 series. Here are the major changes of this last month.

Changes for the user

Allow to edit account move maturity date

The maturity date represents information which has no impact on the accounting but on business only. So it is more flexible to allow to edit it.

Invoice shipment costs only, if there is a shipment

When the cost method is on order and the invoice method is on shipment, the shipment cost is only invoiced when a shipment is done. Before there was an issue with this configuration.

Same input and storage location for customer return shipment

Following similar changes for customer and supplier shipments, the customer return shipments support also to use the same input and storage location. In such case, on reception the shipment is directly marked as done.

URL in web client

The web client sao now supports URLs. This allows users to communicate information by sharing URLs. The syntax is quite similar to the one of the desktop client since the 'index.html' has been removed but uses a URL-hash instead of the full path due to the single page design. The URL is updated to correspond to the active tab. The hash part can be changed by the user (using copy/paste) to open the corresponding tab. This allows to navigate back in the history of the browser. The client parses the URL hash on login and launches the corresponding action.

Web client session

The sessions are now stored in the localStorage of the browser. This means that the session per server and database can be shared between tabs and survive a reload of the page.

Better space management in web client

We have improved the navigation bar to provide more spaces to the tab list. We use an icon for the logout instead of a longer text. Now the favorites menu is a drop down of the global search. The toggle menu and Tryton brand are merged. Also it is now possible to toggle the menu for any size of the screen.

Open Many2One and Reference in new web client tab

It is now possible to open the target record in a new tab instead of a popup by pressing the CTRL key when clicking on the "open" button. This gives access to the actions, relates and reports for this record like the right-click on desktop client does.

Removal of single windows executable

Since we have the web client sao, the need of a Windows executable without installation is very low. More over it required administration rights to be run. So we have decided to stop publishing it in future versions.

Uninstall previous version

On windows, the installer required to uninstall previous version before allowing to install a new one. In order to simplify the process, now it asks to uninstall the previous version of the same series. In silent mode, this is done automatically without any required intervention of the user which ease automatic deployment.

Export CSV

The export CSV window has been improved by moving the predefined exports under the buttons that manage them. This way the view is easier to understand and the predefined list is not the default focused widget (which created inconsistent selection state).

Display current date on document

Draft documents, that are sent to customer, have not yet a date. For example a sale quotation does not have the sale date filled but the document should have at least the date when it was printed to define when starts the validity of the quotation. So we changed the templates to print the current date if no date is yet filled.

Changes for the developer

Add sql_cast on Field

The new method Field.sql_cast(expression) provides the database specific cast of the field values in SQL expressions.

Recursive common table expression for child_of/parent_of operators

For the evaluation of the child_of and parent_of domain operator, we used recursive ORM loops (or Modified Preorder Tree Traversal when available). Now all supported databases support the recursive common table expression (CTE), so we could use it as default implementation. Regarding performance recursive CTE is not better than MPTT but still avoid many round-trip compared to the recursive ORM loop.

Tree Mixin

We added a new Mixin as core feature. It provides all the needed features to manage record names in a tree structure and to check against loops. All the modules having such design have been updated to benefit from this generic implementation that is fully tested.

Limit database per host name

The Tryton server works on multiple database names (one way to support multi-company). But in shared environment, the list of database names can be very large and can leak information. For that, we added a new feature which allows to filter the list of database names per host name.

Default language in report

Until now, almost all reports were using English as fallback language because the call to set_lang required a language code and it was easier to hard-code 'en' than retrieving the default language of the database. Now the set_lang method allow as argument None or an instance of ir.lang. When the value is None, it uses the default language of the database. This is a more consistent behavior with the rest of the application. We have updated all the reports to use this new feature.

Extending depends on methods

The depends on methods was limited to only on_change or on_change_with methods. But it showed some limitations when trying the share common code between different on_change/_with methods. So we extended and generalized the behavior. Now we can define dependencies on any methods and use the @fields.depends decorator on any method.

Today we've issued the 2.0.6 bugfix release.

The release package and checksums are available from our downloads page, as well as from the Python Package Index. The PGP key ID used for this release is Carlton Gibson: E17DF5C82B4F9D00.

Using Appium with Python Appium is an open source test automation framework for use with native, hybrid and mobile web apps. It drives iOS, Android, and Windows apps using the WebDriver protocol. While the main purpose of appium is to perform automation testing, it can be utilized for variety of other things too. Appium has […]

The post Appium Python Client appeared first on The Tara Nights.

In the first three parts of this series of posts we developed together a calculator using a pure TDD methodology. In the previous post we added support for variables.

In this new post we will first add the exponentiation operation. The operator will be challenging because it has a high priority, so we will need to spice up the peek functions to look at multiple tokens.

Then I will show you how I performed a refactoring of the code introducing a new version of the lexer that greatly simplifies the code of the parser.

Level 15 - Exponentiation

That is power.

The exponentiation operation is simple, and Python uses the double star operator to represent it

>>> 2**3
8

The main problem that we will face implementing it is the priority of such an operation. Traditionally, this operator has precedence on the basic arithmetic operations (sum, difference, multiplication, division). So if I write

>>> 1 + 2 ** 3
9

Python correctly computes 1 + (2 ** 3) and not (1 + 2) ** 3. As we did with multiplication and division, then, we will need to create a specific step to parse this operation.

NOTE: I realised too late that the class called PowerNode should have been called ExponentiationNode, the former being a leftover of a previous incorrect nomenclature. I will eventually fix it in one of the refactoring steps, trying to convert a mistake into a good example of TDD.

Lexer

The lexer has a simple task, that of recognising the symbol '^' as a LITERAL token. The test goes into tests/test_calc_lexer.py

deftest_get_tokens_understands_exponentiation():l=clex.CalcLexer()l.load('2 ^ 3')assertl.get_tokens()==[token.Token(clex.INTEGER,'2'),token.Token(clex.LITERAL,'^'),token.Token(clex.INTEGER,'3'),token.Token(clex.EOL),token.Token(clex.EOF)]

Does your code already pass the test? If yes, why?

Parser

It's time to test the proper parsing of the exponentiation operation. Add this test to tests/test_calc_parser.py

deftest_parse_exponentiation():p=cpar.CalcParser()p.lexer.load("2 ^ 3")node=p.parse_exponentiation()assertnode.asdict()=={'type':'exponentiation','left':{'type':'integer','value':2},'right':{'type':'integer','value':3},'operator':{'type':'literal','value':'^'}}

As you can see this test checks directly the parse_exponentiation() method, so you just need to properly implement that, at this stage.

To allow the use of the exponentiation operator '^' in the calculator, however, we have to integrate it with the rest of the parse functions, so we will add three tests to the same file. The first one tests that the natural priority of the exponentiation operator is higher than the multiplication

deftest_parse_exponentiation_with_other_operators():p=cpar.CalcParser()p.lexer.load("3 * 2 ^ 3")node=p.parse_term()assertnode.asdict()=={'type':'binary','operator':{'type':'literal','value':'*'},'left':{'type':'integer','value':3},'right':{'type':'exponentiation','left':{'type':'integer','value':2},'right':{'type':'integer','value':3},'operator':{'type':'literal','value':'^'}}}

The second one checks that the parenthesis still change the priority

deftest_parse_exponentiation_with_parenthesis():p=cpar.CalcParser()p.lexer.load("(3 + 2) ^ 3")node=p.parse_term()assertnode.asdict()=={'type':'exponentiation','operator':{'type':'literal','value':'^'},'left':{'type':'binary','operator':{'type':'literal','value':'+'},'left':{'type':'integer','value':3},'right':{'type':'integer','value':2}},'right':{'type':'integer','value':3}}

And the third one checks that unary operators still have a higher priority than the exponentiation operator

deftest_parse_exponentiation_with_negative_base():p=cpar.CalcParser()p.lexer.load("-2 ^ 2")node=p.parse_exponentiation()assertnode.asdict()=={'type':'exponentiation','operator':{'type':'literal','value':'^'},'left':{'type':'unary','operator':{'type':'literal','value':'-'},'content':{'type':'integer','value':2}},'right':{'type':'integer','value':2}}

My advice is to add the first test and to try and pass that one changing the code. If your change doesn't touch too much of the existing parse methods, chances are that the following two tests will pass as well.

Visitor

Last, we need to properly expose the exponentiation operation in the CLI, which means to change the Visitor in order to support nodes of type 'exponentiation. The test that we need to add to tests/test_calc_visitor.py is

deftest_visitor_exponentiation():ast={'type':'exponentiation','left':{'type':'integer','value':2},'right':{'type':'integer','value':3},'operator':{'type':'literal','value':'^'}}v=cvis.CalcVisitor()assertv.visit(ast)==(8,'integer')

And the change to the CalcVisitor class should be very easy as we need to simply process a new type of node.

Intermezzo - Refactoring with tests

See? You just had to see it in context.

So our little project is growing, and the TDD methodology we are following gives us plenty of confidence in what we did. There as for sure bugs we are not aware of, we we are sure that the cases that we tested are correctly handled by our code.

As happens in many projects at a certain point it's time for refactoring. We implemented solutions to the problems that we found along the way, but are we sure we avoided duplicating code, that we chose the best solution for some algorithms, or more trivially that the names we chose for the variables are clear?

Refactoring means basically to change the internal structure of something without changing its external behaviour, and tests are a priceless help in this phase. The tests we wrote are there to ensure that the previous behaviour does not change. Or, if it changes, that we are perfectly aware of it.

In this section, thus, I want to guide you through a refactoring guided by tests. If you are following this series and writing your own code this section will not add anything to the project, but I recommend that you read it anyway, as it shows why tests are so important in a software project.

If you want to follow the refactoring on the repository you can create a branch on the tag context-manager-refactoring and work there. From that commit I implemented the steps you will find in the next sections.

Context managers

The main issue the current code has is that the lexer cannot automatically recover a past status, that is we cannot easily try to parse something and, when we discover that the initial guess is wrong, go back in time and start over.

Let's consider the parse_line() method

defparse_line(self):try:self.lexer.stash()returnself.parse_assignment()exceptclex.TokenError:self.lexer.pop()returnself.parse_expression()

Since a line can contain either an assignment or an expression the first thing this function does is to save the lexer status with stash() and try to parse an assignment. If the code is not an assignment somewhere is the code the TokenError exception is raised, and parse_line() restores the previous status of the lexer and tries to parse an expression.

The same thing happens in other methods like parse_term()

defparse_term(self):left=self.parse_exponentiation()next_token=self.lexer.peek_token()whilenext_token.type==clex.LITERAL\
                andnext_token.valuein['*','/']:operator=self._parse_symbol()right=self.parse_exponentiation()left=BinaryNode(left,operator,right)next_token=self.lexer.peek_token()returnleft

where the use of lexer.peek_token() and the while loop show that the lexer class requires too much control from the upper level.

Back to parse_line(), it's clear that the code works, but it is not immediately easy to understand what the function does and when the old status is recovered. I'd really prefer something like

# PSEUDOCODE PSEUDOCODE PSEUDOCODE defparse_line(self):ATTEMPTreturnself.parse_assignment()returnself.parse_expression()

where I used a pseudo-keyword ATTEMPT to signal that somehow the lexer status is automatically stored at the beginning and retrieved at the end of it.

There's a very powerful concept in Python that allows us to write code like this, and it is called context manager. I won't go into the theory and syntax of context managers here, please refer to the Python documentation or your favourite course/book/website to discover how context managers work.

If I can add context manager features to the lexer the code of parse_line() might become

defparse_line(self):withself.lexer:returnself.parse_assignment()returnself.parse_expression()

Lexer

The first move is to transform the lexer into a context manager that does nothing. The test in tests/test_calc_lexer.py is

deftest_lexer_as_context_manager():l=clex.CalcLexer()l.load('abcd')withl:assertl.get_token()==token.Token(clex.NAME,'abcd')

When this works we have to be sure that the lexer does not restore the previous state outside the with statement if the code inside the statement ended without errors. The new test is

deftest_lexer_as_context_manager_does_not_restore_the_status_if_no_error():l=clex.CalcLexer()l.load('3 * 5')withl:assertl.get_token()==token.Token(clex.INTEGER,3)assertl.get_token()==token.Token(clex.LITERAL,'*')

Conversely, we need to be sure that the status is restored when the code inside the with statement fails, which is the whole point of the context manager. This is tested by

deftest_lexer_as_context_manager_restores_the_status_if_token_error():l=clex.CalcLexer()l.load('3 * 5')withl:l.get_token()l.get_token()raiseclex.TokenErrorassertl.get_token()==token.Token(clex.INTEGER,3)

When these three tests pass, we have a fully working context manager lexer, that reacts to TokenError exceptions going back to the previously stored status.

Parser

If the context manager lexer works as intended we should be able to replace the code of the parser without changing any test. The new code for parse_line() is the one I showed before

defparse_line(self):withself.lexer:returnself.parse_assignment()returnself.parse_expression()

and it works flawlessly.

The context manager part of the lexer, however, works if the code inside the with statement raises a TokenError exception when it fails. That exception is a signal to the context manager that the parsing path is not leading anywhere and it shall go back to the previous state.

Manage literals

The _parse_symbol() method is often used after some checks like

ifnext_token.type==clex.LITERALandnext_token.valuein['-','+']:operator=self._parse_symbol()

I would prefer to include the checks in the method itself, so that it might be included in a with statement. First of all the method can be renamed to _parse_literal(), and being an internal method I don't expect any test to fail

def_parse_literal(self):t=self.lexer.get_token()returnLiteralNode(t.value)

The method should also raise a TokenError when the token is not a LITERAL, and when the values are not the expected ones

def_parse_literal(self,values=None):t=self.lexer.get_token()ift.type!=clex.LITERAL:raiseclex.TokenErrorifvaluesandt.valuenotinvalues:raiseclex.TokenErrorreturnLiteralNode(t.value)

Note that using the default value for the values parameter I didn't change the current behaviour. The whole battery of tests still passes without errors.

Parsing factors

The next method that we can start changing is parse_factor(). The first pattern this function tries to parse is an unary node

ifnext_token.type==clex.LITERALandnext_token.valuein['-','+']:operator=self._parse_literal()factor=self.parse_factor()returnUnaryNode(operator,factor)

which may be converted to

withself.lexer:operator=self._parse_literal(['+','-'])content=self.parse_factor()returnUnaryNode(operator,content)

while still passing the whole test suite.

The second pattern are expressions surrounded by parenthesis

ifnext_token.type==clex.LITERALandnext_token.value=='(':self.lexer.discard_type(clex.LITERAL)expression=self.parse_expression()self.lexer.discard_type(clex.LITERAL)returnexpression

and this is easily converted to the new syntax as well

withself.lexer:self._parse_literal(['('])expression=self.parse_expression()self._parse_literal([')'])returnexpression

Parsing exponentiation operations

To change the parse_exponentiation() method we need first to make the _parse_variable() return a TokenError in case of wrong token

def_parse_variable(self):t=self.lexer.get_token()ift.type!=clex.NAME:raiseclex.TokenErrorreturnVariableNode(t.value)

then we can change the method we are interested in

defparse_exponentiation(self):left=self.parse_factor()withself.lexer:operator=self._parse_literal(['^'])right=self.parse_exponentiation()returnPowerNode(left,operator,right)returnleft

Doing this last substitution I also notice that there is some duplicated code in parse_factor(), and I replace it with a call to _parse_variable(). The test suite keeps passing, giving me the certainty that what I did does not change the behaviour of the code (at least the behaviour that is covered by tests).

Parsing terms

Now, the parse_term() method will be problematic. To implement this function I used a while loop that keeps using the parse_exponentiation() method until the separation token is a LITERAL with value * or /

defparse_term(self):left=self.parse_exponentiation()next_token=self.lexer.peek_token()whilenext_token.type==clex.LITERAL\
                andnext_token.valuein['*','/']:operator=self._parse_literal()right=self.parse_exponentiation()left=BinaryNode(left,operator,right)next_token=self.lexer.peek_token()returnleft

This is not a pure recursive call, then, and replacing the code with the context manager lexer would result in errors, because the context manage doesn't loop. The same situation is replicated in parse_expression().

This is another typical situation that we face when refactoring code. We realise that the required change is made of multiple steps and that multiple tests will fail until all the steps are completed.

There is no single solution to this problem, but TDD gives you some hints to deal with it. The most important "rule" that I follow when I work in a TDD environment is that there should be maximum one failing test at a time. And when a code change makes multiple tests fail there is a simple way to reach this situation: comment out tests.

Yes, you should temporarily get rid of tests, so you can concentrate in writing code that passes the subset of active tests. Then you will add one test at a time, fixing the code or the tests according to your needs. When you refactor it might be necessary to change the tests as well, as sometimes we test part of the code that are not exactly an external boundary.

I can now change the code of the parse_term() function introducing the context manager

defparse_term(self):left=self.parse_exponentiation()withself.lexer:operator=self._parse_literal(['*','/'])right=self.parse_exponentiation()returnBinaryNode(left,operator,right)returnleft

and the test suite runs with one single failing test, test_parse_term_with_multiple_operations(). I have now to work on it and try to understand why the test fails.

I decided to go for a pure recursive approach (no more while loops), which is what standard language parsers do. After working on it the new version of parse_term() is

defparse_term(self):left=self.parse_factor()withself.lexer:operator=self._parse_literal(['*','/'])right=self.parse_term()returnBinaryNode(left,operator,right)returnleft

And this change makes 3 tests fail. The same test_parse_term_with_multiple_operations() that was failing before, plus test_parse_exponentiation_with_other_operators(), and test_parse_exponentiation_with_parenthesis(). The last two actually test the method parse_exponentiation(), which uses parse_term(). This means that I can temporarily comment them and concentrate on the first one.

What I discover is that changing the code to use the recursive approach changes the output of the functions. The previous output of parse_term() applied to 2 * 3 / 4 was

{'type':'binary','left':{'type':'binary','left':{'type':'integer','value':2},'right':{'type':'integer','value':3},'operator':{'type':'literal','value':'*'}},'right':{'type':'integer','value':4},'operator':{'type':'literal','value':'/'}}

that is, the multiplication was stored first. Moving to a recursive approach makes the parse_term() function return this

{'type':'binary','left':{'type':'integer','value':2},'right':{'type':'binary','left':{'type':'integer','value':3},'right':{'type':'integer','value':4},'operator':{'type':'literal','value':'/'}},'operator':{'type':'literal','value':'*'}}

I think it is pretty clear that this structure, once visited, will return the same value as the previous one, as multiplication and division can be swapped. We will have to pay attention to this swap when operators with different priority are involved, like sum and multiplication, but for this test we can agree the result is no different.

This means that we may change the test and make it pass. Let me stress it once more: we have to understand why the test doesn't pass, and once we understood the reason, and decided it is acceptable, we can change the test.

Tests are not immutable, they are mere safeguards that raise alarms when you change the behaviour. It's up to you to deal with the alarm and to decide how what to do.

Once the test has been modified and the test suite passes, it's time to uncomment the first of the two remaining tests, test_parse_exponentiation_with_other_operators(). This test uses parse_term() to parse a string that contains an exponentiation, but the new code of the parse_term() method doesn't call the parse_exponentiation() function. So the test fails.

Parsing exponentiation

That tests tries to parse a string that contains a multiplication and an exponentiation, so the method that we should use to process it is parse_term(). The current version of parse_term(), however, doesn't consider exponentiation, so the new code is

defparse_term(self):left=self.parse_exponentiation()withself.lexer:operator=self._parse_literal(['*','/'])right=self.parse_term()returnBinaryNode(left,operator,right)returnleft

which makes all the active tests pass.

There is still one commented test, test_parse_exponentiation_with_parenthesis, that now passes with the new code.

Parsing expressions

The new version of parse_expression() has the same issue we found with parse_term(), that is the recursive approach changes the output.

defparse_expression(self):left=self.parse_term()withself.lexer:operator=self._parse_literal(['+','-'])right=self.parse_expression()left=BinaryNode(left,operator,right)returnleft

As before, we have to decide if the change is acceptable or if it is a real error. As happened for parse_term() the test can be safely changes to match the new code output.

Level 16 - Float numbers

So far, our calculator can handle only integer values, so it's time to add support for float numbers. This change shouldn't be complex: floating point numbers are easy to parse as they are basically two integer numbers separated by a dot.

Lexer

To test if the lexer understands floating point numbers it's enough to add this to tests/test_calc_lexer.py

deftest_get_tokens_understands_floats():l=clex.CalcLexer()l.load('3.6')assertl.get_tokens()==[token.Token(clex.FLOAT,'3.6'),token.Token(clex.EOL),token.Token(clex.EOF)]

Parser

To support float numbers it's enough to add that feature the method that we use to parse integers. We can rename parse_integer() to parse_number(), fix the test test_parse_integer(), and add test_parse_float()

deftest_parse_integer():p=cpar.CalcParser()p.lexer.load("5")node=p.parse_number()assertnode.asdict()=={'type':'integer','value':5}deftest_parse_float():p=cpar.CalcParser()p.lexer.load("5.8")node=p.parse_number()assertnode.asdict()=={'type':'float','value':5.8}

Visitor

The same thing that we did for the parser is valid for the visitor. We just need to copy the test for the integer numbers and adapt it

deftest_visitor_float():ast={'type':'float','value':12.345}v=cvis.CalcVisitor()assertv.visit(ast)==(12.345,'float')

This however leaves a bug in the visitor. The Test-Driven Development methodology can help you writing and changing your code, but cannot completely avoid bugs in the code. Actually, if you don't test cases, TDD can't do anything to find and remove bugs.

The bug I noticed after a while is that the visitor doesn't correctly manage an operation between integers and floats, returning an integer result. For example, if you sum 4 with 5.1 you should get 9.1 with type float. To test this behaviour we can add this code

deftest_visitor_expression_sum_with_float():ast={'type':'binary','left':{'type':'float','value':5.1},'right':{'type':'integer','value':4},'operator':{'type':'literal','value':'+'}}v=cvis.CalcVisitor()assertv.visit(ast)==(9.1,'float')

Conclusion

This post showed you how powerful the TDD methodology is when it comes to refactoring, or in general when the code has to be changed. Remember that tests can be changed if there are good reasons, and that the main point is to understand what's happening in your code and in the cases that you already tested.

Titles

The section quotes come from some good films: "Conan the Barbarian" (1982) and "Scrooged" (1988).

Feedback

Feel free to reach me on Twitter if you have questions. The GitHub issues page is the best place to submit corrections.

This is my solution to the fourth part of "A Game of Tokens", which can be found here.

You can find the tests and the code presented here in this repository in the branch called part4.

Level 15 - Exponentiation

Lexer

The lexer can process the exponentiation operator ^ out of the box as a LITERAL token, so no changes to the code are needed.

Parser

The test test_parse_exponentiation() can be passed adding a PowerNode class

classPowerNode(BinaryNode):node_type='exponentiation'

and a parse_exponentiation() method to the parser

defparse_exponentiation(self):left=self.parse_factor()next_token=self.lexer.peek_token()ifnext_token.type==clex.LITERALandnext_token.value=='^':operator=self._parse_symbol()right=self.parse_exponentiation()returnPowerNode(left,operator,right)returnleft

This allows the parser to explicitly parse the exponentiation operation, but when the operation is mixed with others the parser doesn't know how to deal with it, as parse_exponentiation() is not called by any other method.

To pass the test_parse_exponentiation_with_other_operators() test we need to change the parse_term() method and call parse_exponentiation() instead of parse_factor()

defparse_term(self):left=self.parse_exponentiation()next_token=self.lexer.peek_token()whilenext_token.type==clex.LITERAL\
                andnext_token.valuein['*','/']:operator=self._parse_symbol()right=self.parse_exponentiation()left=BinaryNode(left,operator,right)

the full code of the CalcParser class is now

classCalcParser:def__init__(self):self.lexer=clex.CalcLexer()def_parse_symbol(self):t=self.lexer.get_token()returnLiteralNode(t.value)defparse_integer(self):t=self.lexer.get_token()returnIntegerNode(t.value)def_parse_variable(self):t=self.lexer.get_token()returnVariableNode(t.value)defparse_factor(self):next_token=self.lexer.peek_token()ifnext_token.type==clex.LITERALandnext_token.valuein['-','+']:operator=self._parse_symbol()factor=self.parse_factor()returnUnaryNode(operator,factor)ifnext_token.type==clex.LITERALandnext_token.value=='(':self.lexer.discard_type(clex.LITERAL)expression=self.parse_expression()self.lexer.discard_type(clex.LITERAL)returnexpressionifnext_token.type==clex.NAME:t=self.lexer.get_token()returnVariableNode(t.value)returnself.parse_integer()defparse_exponentiation(self):left=self.parse_factor()next_token=self.lexer.peek_token()ifnext_token.type==clex.LITERALandnext_token.value=='^':operator=self._parse_symbol()right=self.parse_exponentiation()returnPowerNode(left,operator,right)returnleftdefparse_term(self):left=self.parse_exponentiation()next_token=self.lexer.peek_token()whilenext_token.type==clex.LITERAL\
                andnext_token.valuein['*','/']:operator=self._parse_symbol()right=self.parse_exponentiation()left=BinaryNode(left,operator,right)next_token=self.lexer.peek_token()returnleftdefparse_expression(self):left=self.parse_term()next_token=self.lexer.peek_token()whilenext_token.type==clex.LITERAL\
                andnext_token.valuein['+','-']:operator=self._parse_symbol()right=self.parse_term()left=BinaryNode(left,operator,right)next_token=self.lexer.peek_token()returnleftdefparse_assignment(self):variable=self._parse_variable()self.lexer.discard(token.Token(clex.LITERAL,'='))value=self.parse_expression()returnAssignmentNode(variable,value)defparse_line(self):try:self.lexer.stash()returnself.parse_assignment()exceptclex.TokenError:self.lexer.pop()returnself.parse_expression()

Visitor

The given test test_visitor_exponentiation() requires the CalcVisitor to parse nodes of type exponentiation. The code required to do this is

ifnode['type']=='exponentiation':lvalue,ltype=self.visit(node['left'])rvalue,rtype=self.visit(node['right'])returnlvalue**rvalue,ltype

as a final case for the CalcVisitor class. The full code of the class is is now

classCalcVisitor:def__init__(self):self.variables={}defisvariable(self,name):returnnameinself.variablesdefvalueof(self,name):returnself.variables[name]['value']deftypeof(self,name):returnself.variables[name]['type']defvisit(self,node):ifnode['type']=='integer':returnnode['value'],node['type']ifnode['type']=='variable':returnself.valueof(node['value']),self.typeof(node['value'])ifnode['type']=='unary':operator=node['operator']['value']cvalue,ctype=self.visit(node['content'])ifoperator=='-':return-cvalue,ctypereturncvalue,ctypeifnode['type']=='binary':lvalue,ltype=self.visit(node['left'])rvalue,rtype=self.visit(node['right'])operator=node['operator']['value']ifoperator=='+':returnlvalue+rvalue,rtypeelifoperator=='-':returnlvalue-rvalue,rtypeelifoperator=='*':returnlvalue*rvalue,rtypeelifoperator=='/':returnlvalue//rvalue,rtypeifnode['type']=='assignment':right_value,right_type=self.visit(node['value'])self.variables[node['variable']]={'value':right_value,'type':right_type}returnNone,None

Level 16 - Float numbers

Lexer

The first thing the lexer need is a label to identify FLOAT tokens

FLOAT='FLOAT'

then the method _process_integer() cna be extended to process float numbers as well. To do this the method is renamed to _process_number(), the regular expression is modified, and the token_type is managed according to the presence of the dot.

def_process_number(self):regexp=re.compile('[\d\.]+')match=regexp.match(self._text_storage.tail)ifnotmatch:returnNonetoken_string=match.group()token_type=FLOATif'.'intoken_stringelseINTEGERreturnself._set_current_token_and_skip(token.Token(token_type,token_string))

Remember that the get_token() function has to be modified to use the new name of the method. The new code is

defget_token(self):eof=self._process_eof()ifeof:returneofeol=self._process_eol()ifeol:returneolself._process_whitespace()name=self._process_name()ifname:returnnameinteger=self._process_number()ifinteger:returnintegerliteral=self._process_literal()ifliteral:returnliteral

Parser

First we need to add a new type of node

classFloatNode(ValueNode):node_type='float'

The new version of parse_integer(), renamed parse_number(), shall deal with both cases but also raise the TokenError exception if the parsing fails

defparse_number(self):t=self.lexer.get_token()ift.type==clex.INTEGER:returnIntegerNode(int(t.value))elift.type==clex.FLOAT:returnFloatNode(float(t.value))raiseclex.TokenError

Visitor

The change to support float nodes is trivial, we just need to include it alongside with the integer case

defvisit(self,node):ifnode['type']in['integer','float']:returnnode['value'],node['type']

To fix the missing type promotion when dealing with integers and floats it's enough to add

ifltype=='float':rtype=ltype

just before evaluating the operator in the binary nodes. The full code of the visit() method is then

defvisit(self,node):ifnode['type']in['integer','float']:returnnode['value'],node['type']ifnode['type']=='variable':returnself.valueof(node['value']),self.typeof(node['value'])ifnode['type']=='unary':operator=node['operator']['value']cvalue,ctype=self.visit(node['content'])ifoperator=='-':return-cvalue,ctypereturncvalue,ctypeifnode['type']=='binary':lvalue,ltype=self.visit(node['left'])rvalue,rtype=self.visit(node['right'])operator=node['operator']['value']ifltype=='float':rtype=ltypeifoperator=='+':returnlvalue+rvalue,rtypeelifoperator=='-':returnlvalue-rvalue,rtypeelifoperator=='*':returnlvalue*rvalue,rtypeelifoperator=='/':returnlvalue//rvalue,rtypeifnode['type']=='assignment':right_value,right_type=self.visit(node['value'])self.variables[node['variable']]={'value':right_value,'type':right_type}returnNone,Noneifnode['type']=='exponentiation':lvalue,ltype=self.visit(node['left'])rvalue,rtype=self.visit(node['right'])returnlvalue**rvalue,ltype

Final words

I bet at this point of the challenge the addition of exponentiation and float numbers wasn't that hard. The refactoring might have been a bit thougher, however, but I hope that it showed you the real power of TDD.

Feedback

Feel free to reach me on Twitter if you have questions. The GitHub issues page is the best place to submit corrections.

# debug1.py

from __future__ import print_function

# A simple debugging function for Python programs.
# How to use it:
# If the -O option is not given on the Python command line (the more common case 
# during development), the in-built special variable __debug__ is defined as True, 
# and the debug1 function displays debugging messages, i.e. it prints the message 
# and all other (optional) values passed to it.
# If the -O option is given, the variable __debug__ is defined as False, 
# and the debug1 function does nothing is defined as a no-op, so does nothing.

import os

if __debug__:
    def debug1(message, *values):
        if len(values) == 0:
            print(message)
        else:
            print("{}:".format(message), end="")
            print("".join([repr(value) for value in values]))
else:
    def debug1(message, *values):
        # Do nothing.
        pass

def main():
    # Test the debug1 function with some calls.
    debug1('message only')
    debug1('message with int', 1)
    debug1('message with int, float', 1, 2.3)
    debug1('message with long, string', 4L, "hi")
    debug1('message with boolean, tuple, set', True, (1, 2), { 3, 4} )
    debug1('message with string, boolean, list', "hi", True, [2, 3])
    debug1('message with complex, dict', 1 + 2j, {'a': 'apple', 'b': 'banana'})
    class Foo: pass
    foo = Foo()
    debug1('message with object', foo)
    debug1('message with class', Foo)
    debug1('message with xrange', xrange(3))
    debug1('message with listiterator', iter(range(4)))

if __name__ == '__main__':
    main()

from debug1 import debug1

def factorial(n):
    # This line prints the message and the value of n when debug1 is enabled.
    debug1("entered factorial, n", n)
    if n  0:
        raise Exception("Factorial argument must be integer, 0 or greater.")
    if n == 0:
        return 1
    p = 1
    for i in range(1, n + 1):
        p *= i
        # This line prints the message and the changing values 
        # of i and p when debug1 is enabled.
        debug1("in for loop, i, p", i, p)
    return p

print "i\tfactorial(i)"
for i in range(6):
    print "{}\t{}".format(i, factorial(i))

$ python factorial.py
i       factorial(i)
0       1
in for loop, i, p: 1 1
1       1
in for loop, i, p: 1 1
in for loop, i, p: 2 2
2       2
in for loop, i, p: 1 1
in for loop, i, p: 2 2
in for loop, i, p: 3 6
3       6
in for loop, i, p: 1 1
in for loop, i, p: 2 2
in for loop, i, p: 3 6
in for loop, i, p: 4 24
4       24
in for loop, i, p: 1 1
in for loop, i, p: 2 2
in for loop, i, p: 3 6
in for loop, i, p: 4 24
in for loop, i, p: 5 120
5       120

$ python -O factorial.py
i       factorial(i)
0       1
1       1
2       2
3       6
4       24
5       120

Hi everyone! I recently decided to step into YouTube video making. This is my first video about 14 of my most favourite and most famous Python libraries and frameworks. Please take a look and if you have ANY suggestions as to how I can improve the quality of the videos in the future please let me know in the comments below.  

Libraries covered:

1. requests
2. tqdm
3. pillow
4. scrapy
5. numpy
6. pandas
7. scapy
8. matplotlib
9. kivy
10. nltk
11. keras
12. SQLAlchemy
13. Django
14. Twisted

Extra libraries:

1. flask
2. scipy
3. pytorch
4. DjangoCMS
5. click
6. astropy
7. scikit-learn
8. opencv
9. PyQt
10. tensorflow
11. wagtail

Till next time, take care!

In 2015 I was introduced by my friend Roberto Ciatti to the concept of Clean Architecture, as it is called by Robert Martin. The well-known Uncle Bob talks a lot about this concept at conferences and wrote some very interesting posts about it. What he calls "Clean Architecture" is a way of structuring a software system, a set of consideration (more than strict rules) about the different layers and the role of the actors in it.

As he clearly states in a post aptly titled The Clean Architecture, the idea behind this design is not new, being built on a set of concepts that have been pushed by many software engineers over the last 3 decades. One of the first implementations may be found in the Boundary-Control-Entity model proposed by Ivar Jacobson in his masterpiece "Object-Oriented Software Engineering: A Use Case Driven Approach" published in 1992, but Martin lists other more recent versions of this architecture.

I will not repeat here what he had already explained better than I can do, so I will just point out some resources you may check to start exploring these concepts:

• The Clean Architecture a post by Robert Martin that concisely describes the goals of the architecture. It also lists resources that describe similar architectures.
• The Open Closed Principle a post by Robert Martin not strictly correlated with the Clean Architecture concept but important for the separation concept.
• Hakka Labs: Robert "Uncle Bob" Martin - Architecture: The Lost Years a video of Robert Martin from Hakka Labs.
• DDD & Testing Strategy by Lauri Taimila
• (upcoming) Clean Architecture by Robert Martin, published by Prentice Hall.

The purpose of this post is to show how to build a web service in Python from scratch using a clean architecture. One of the main advantages of this layered design is testability, so I will develop it following a TDD approach. The project was initially developed from scratch in around 3 hours of work. Given the toy nature of the project some choices have been made to simplify the resulting code. Whenever meaningful I will point out those simplifications and discuss them.

If you want to know more about TDD in Python read the posts in this category.

Project overview

The goal of the "Rent-o-matic" project (fans of Day of the Tentacle may get the reference) is to create a simple search engine on top of a dataset of objects which are described by some quantities. The search engine shall allow to set some filters to narrow the search.

The objects in the dataset are storage rooms for rent described by the following quantities:

• An unique identifier
• A size in square meters
• A renting price in Euro/day
• Latitude and longitude

As pushed by the clean architecture model, we are interested in separating the different layers of the system. The architecture is described by four layers, which however can be implemented by more than four actual code modules. I will give here a brief description of those layers.

Entities

This is the level in which the domain models are described. Since we work in Python, I will put here the class that represent my storage rooms, with the data contained in the database, and whichever data I think is useful to perform the core business processing.

It is very important to understand that the models in this layer are different from the usual models of framework like Django. These models are not connected with a storage system, so they cannot be directly saved or queried using methods of their classes. They may however contain helper methods that implement code related to the business rules.

Use cases

This layer contains the use cases implemented by the system. In this simple example there will be only one use case, which is the list of storage rooms according to the given filters. Here you would put for example a use case that shows the detail of a given storage room or every business process you want to implement, such as booking a storage room, filling it with goods, etc.

Interface Adapters

This layer corresponds to the boundary between the business logic and external systems and implements the APIs used to exchange data with them. Both the storage system and the user interface are external systems that need to exchange data with the use cases and this layer shall provide an interface for this data flow. In this project the presentation part of this layer is provided by a JSON serializer, on top of which an external web service may be built. The storage adapter shall define here the common API of the storage systems.

External interfaces

This part of the architecture is made by external systems that implement the interfaces defined in the previous layer. Here for example you will find a web server that implements (REST) entry points, which access the data provided by use cases through the JSON serializer. You will also find here the storage system implementation, for example a given database such as MongoDB.

API and shades of grey

The word API is of uttermost importance in a clean architecture. Every layer may be accessed by an API, that is a fixed collection of entry points (methods or objects). Here "fixed" means "the same among every implementation", obviously an API may change with time. Every presentation tool, for example, will access the same use cases, and the same methods, to obtain a set of domain models, which are the output of that particular use case. It is up to the presentation layer to format data according to the specific presentation media, for example HTML, PDF, images, etc. If you understand plugin-based architectures you already grasped the main concept of a separate, API-driven component (or layer).

The same concept is valid for the storage layer. Every storage implementation shall provide the same methods. When dealing with use cases you shall not be concerned with the actual system that stores data, it may be a MongoDB local installation, a cloud storage system or a trivial in-memory dictionary.

The separation between layers, and the content of each layer, is not always fixed and immutable. A well-designed system shall also cope with practical world issues such as performances, for example, or other specific needs. When designing an architecture it is very important to know "what is where and why", and this is even more important when you "bend" the rules. Many issues do not have a black-or-white answer, and many decisions are "shades of grey", that is it is up to you to justify why you put something in a given place.

Keep in mind however, that you should not break the structure of the clean architecture, in particular you shall be inflexible about the data flow (see the "Crossing boundaries" section in the original post of Robert Martin). If you break the data flow, you are basically invalidating the whole structure. Let me stress it again: never break the data flow. A simple example of breaking the data flow is to let a use case output a Python class instead of a representation of that class such as a JSON string.

Project structure

Let us take a look at the final structure of the project

The global structure of the package has been built with Cookiecutter, and I will run quickly through that part. The rentomatic directory contains the following subdirectories: domain, repositories, REST, serializers, use_cases. Those directories reflect the layered structure introduced in the previous section, and the structure of the tests directory mirrors this structure so that tests are easily found.

Source code

You can find the source code in this GitHub repository. Feel free to fork it and experiment, change it, and find better solutions to the problem I will discuss in this post. The source code contains tagged commits to allow you to follow the actual development as presented in the post. You can find the current tag in the Git tag: <tag name> label under the section titles. The label is actually a link to the tagged commit on GitHub, if you want to see the code without cloning it.

Project initialization

Git tag: step01

Update: this Cookiecutter package creates an environment like the one I am creating in this section. I will keep the following explanation so that you can see how to manage requirements and configurations, but for your next project consider using this automated tool.

I usually like maintaining a Python virtual environment inside the project, so I will create a temporary virtualenv to install cookiecutter, create the project, and remove the virtualenv. Cookiecutter is going to ask you some questions about you and the project, to provide an initial file structure. We are going to build our own testing environment, so it is safe to answer no to use_pytest. Since this is a demo project we are not going to need any publishing feature, so you can answer no to use_pypi_deployment_with_travis as well. The project does not have a command line interface, and you can safely create the author file and use any license.

virtualenv venv3 -p python3
source venv3/bin/activate
pip install cookiecutter
cookiecutter https://github.com/audreyr/cookiecutter-pypackage

Now answer the questions, then finish creating the project with the following code

deactivate
rm -fR venv3
cd rentomatic
virtualenv venv3 -p python3
source venv3/bin/activate

Get rid of the requirements_dev.txt file that Cookiecutter created for you. I usually store virtualenv requirements in different hierarchical files to separate production, development and testing environments, so create the requirements directory and the relative files

mkdir requirements
touch requirements/prod.txt
touch requirements/dev.txt
touch requirements/test.txt

The test.txt file will contain specific packages used to test the project. Since to test the project you also need to install the packages for the production environment the file will first include the production one.

-r prod.txt

pytest
tox
coverage
pytest-cov

The dev.txt file will contain packages used during the development process and shall install also test and production package

-r test.txt

pip
wheel
flake8
Sphinx

(taking advantage of the fact that test.txt already includes prod.txt).

Last, the main requirements.txt file of the project will just import requirements/prod.txt

-r prod.txt

Obviously you are free to find the project structure that better suits your need or preferences. This is the structure we are going to use in this project but nothing forces you to follow it in your personal projects.

This separation allows you to install a full-fledged development environment on your machine, while installing only testing tools in a testing environment like the Travis platform and to further reduce the amount of dependencies in the production case.

As you can see, I am not using version tags in the requirements files. This is because this project is not going to be run in a production environment, so we do not need to freeze the environment.

Remember at this point to install the development requirements in your virtualenv

$ pip install -r requirements/dev.txt

Miscellaneous configuration

The pytest testing library needs to be configured. This is the pytest.ini file that you can create in the root directory (where the setup.py file is located)

[pytest]
minversion = 2.0
norecursedirs = .git .tox venv* requirements*
python_files = test*.py

To run the tests during the development of the project just execute

$ py.test -sv

If you want to check the coverage, i.e. the amount of code which is run by your tests or "covered", execute

$ py.test --cov-report term-missing --cov=rentomatic

If you want to know more about test coverage check the official documentation of the Coverage.py and the pytest-cov packages.

I strongly suggest the use of the flake8 package to check that your Python code is PEP8 compliant. This is the flake8 configuration that you can put in your setup.cfg file

[flake8]
ignore = D203
exclude = .git, venv*, docs
max-complexity = 10

To check the compliance of your code with the PEP8 standard execute

$ flake8

Flake8 documentation is available here.

Note that every step in this post produces tested code and a of coverage of 100%. One of the benefits of a clean architecture is the separation between layers, and this guarantees a great degree of testability. Note however that in this tutorial, in particular in the REST sections, some tests have been omitted in favour of a simpler description of the architecture.

Domain models

Git tag: step02

Let us start with a simple definition of the StorageRoom model. As said before, the clean architecture models are very lightweight, or at least they are lighter than their counterparts in a framework.

Following the TDD methodology the first thing that I write are the tests. Create the tests/domain/test_storageroom.py and put this code inside it

importuuidfromrentomatic.domainimportmodelsasmdeftest_storageroom_model_init():code=uuid.uuid4()storageroom=m.StorageRoom(code,size=200,price=10,longitude='-0.09998975',latitude='51.75436293')assertstorageroom.code==codeassertstorageroom.size==200assertstorageroom.price==10assertstorageroom.longitude==-0.09998975assertstorageroom.latitude==51.75436293deftest_storageroom_model_from_dict():code=uuid.uuid4()storageroom=m.StorageRoom.from_dict({'code':code,'size':200,'price':10,'longitude':'-0.09998975','latitude':'51.75436293'})assertstorageroom.code==codeassertstorageroom.size==200assertstorageroom.price==10assertstorageroom.longitude==-0.09998975assertstorageroom.latitude==51.75436293

With these two tests we ensure that our model can be initialized with the correct values and that can be created from a dictionary. In this first version all the parameters of the model are required. Later we could want to make some of them optional, and in that case we will have to add the relevant tests.

Now let's write the StorageRoom class in the rentomatic/domain/storageroom.py file. Do not forget to create the __init__.py file in the subdirectories of the project, otherwise Python will not be able to import the modules.

fromrentomatic.shared.domain_modelimportDomainModelclassStorageRoom(object):def__init__(self,code,size,price,latitude,longitude):self.code=codeself.size=sizeself.price=priceself.latitude=float(latitude)self.longitude=float(longitude)@classmethoddeffrom_dict(cls,adict):room=StorageRoom(code=adict['code'],size=adict['size'],price=adict['price'],latitude=adict['latitude'],longitude=adict['longitude'],)returnroomDomainModel.register(StorageRoom)

The model is very simple, and requires no further explanation. One of the benefits of a clean architecture is that each layer contains small pieces of code that, being isolated, shall perform simple tasks. In this case the model provides an initialization API and stores the information inside the class.

The from_dict method comes in handy when we have to create a model from data coming from another layer (such as the database layer or the query string of the REST layer).

One could be tempted to try to simplify the from_dict function, abstracting it and providing it through a Model class. Given that a certain level of abstraction and generalization is possible and desirable, the initialization part of the models shall probably deal with various different cases, and thus is better off being implemented directly in the class.

The DomainModel abstract base class is an easy way to categorize the model for future uses like checking if a class is a model in the system. For more information about this use of Abstract Base Classes in Python see this post.

Serializers

Git tag: step03

Our model needs to be serialized if we want to return it as a result of an API call. The typical serialization format is JSON, as this is a broadly accepted standard for web-based API. The serializer is not part of the model, but is an external specialized class that receives the model instance and produces a representation of its structure and values.

To test the JSON serialization of our StorageRoom class put in the tests/serializers/test_storageroom_serializer.py file the following code

importjsonfromrentomatic.serializersimportstorageroom_serializerassrsfromrentomatic.domainimportmodelsdeftest_serialize_domain_storageroom():room=models.StorageRoom('f853578c-fc0f-4e65-81b8-566c5dffa35a',size=200,price=10,longitude='-0.09998975',latitude='51.75436293')expected_json="""        {"code": "f853578c-fc0f-4e65-81b8-566c5dffa35a","size": 200,"price": 10,"longitude": -0.09998975,"latitude": 51.75436293        }"""assertjson.loads(json.dumps(room,cls=srs.StorageRoomEncoder))==json.loads(expected_json)

Put in the rentomatic/serializers/storageroom_serializer.py file the code that makes the test pass

importjsonclassStorageRoomEncoder(json.JSONEncoder):defdefault(self,o):try:to_serialize={'code':o.code,'size':o.size,'price':o.price,"latitude":o.latitude,"longitude":o.longitude,}returnto_serializeexceptAttributeError:returnsuper().default(o)

Providing a class that inherits from json.JSONEncoder let us use the json.dumps(room, cls=StorageRoomEncoder) syntax to serialize the model.

There is a certain degree of repetition in the code we wrote, and this is the annoying part of a clean architecture. Since we want to isolate layers as much as possible and create lightweight classes we end up somehow repeating certain types of actions. For example the serialization code that assigns attributes of a StorageRoom to JSON attributes is very similar to that we use to create the object from a dictionary. Not exactly the same, obviously, but the two functions are very close.

Use cases (part 1)

Git tag: step04

It's time to implement the actual business logic our application wants to expose to the outside world. Use cases are the place where we implement classes that query the repository, apply business rules, logic, and whatever transformation we need for our data, and return the results.

With those requirements in mind, let us start to build a use case step by step. The simplest use case we can create is one that fetches all the storage rooms from the repository and returns them. Please note that we did not implement any repository layer yet, so our tests will mock it.

This is the skeleton for a basic test of a use case that lists all the storage rooms. Put this code in the tests/use_cases/test_storageroom_list_use_case.py

importpytestfromunittestimportmockfromrentomatic.domainimportmodelsfromrentomatic.use_casesimportstorageroom_use_casesasuc@pytest.fixturedefdomain_storagerooms():storageroom_1=models.StorageRoom(code='f853578c-fc0f-4e65-81b8-566c5dffa35a',size=215,price=39,longitude='-0.09998975',latitude='51.75436293',)storageroom_2=models.StorageRoom(code='fe2c3195-aeff-487a-a08f-e0bdc0ec6e9a',size=405,price=66,longitude='0.18228006',latitude='51.74640997',)storageroom_3=models.StorageRoom(code='913694c6-435a-4366-ba0d-da5334a611b2',size=56,price=60,longitude='0.27891577',latitude='51.45994069',)storageroom_4=models.StorageRoom(code='eed76e77-55c1-41ce-985d-ca49bf6c0585',size=93,price=48,longitude='0.33894476',latitude='51.39916678',)return[storageroom_1,storageroom_2,storageroom_3,storageroom_4]deftest_storageroom_list_without_parameters(domain_storagerooms):repo=mock.Mock()repo.list.return_value=domain_storageroomsstorageroom_list_use_case=uc.StorageRoomListUseCase(repo)result=storageroom_list_use_case.execute()repo.list.assert_called_with()assertresult==domain_storagerooms

The test is straightforward. First we mock the repository so that is provides a list() method that returns the list of models we created above the test. Then we initialize the use case with the repo and execute it, collecting the result. The first thing we check is if the repository method was called without any parameter, and the second is the effective correctness of the result.

This is the implementation of the use case that makes the test pass. Put the code in the rentomatic/use_cases/storageroom_use_case.py

classStorageRoomListUseCase(object):def__init__(self,repo):self.repo=repodefexecute(self):returnself.repo.list()

With such an implementation of the use case, however, we will soon experience issues. For starters, we do not have a standard way to transport the call parameters, which means that we do not have a standard way to check for their correctness either. The second problem is that we miss a standard way to return the call results and consequently we lack a way to communicate if the call was successful of if it failed, and in the latter case what are the reasons of the failure. This applies also to the case of bad parameters discussed in the previous point.

We want thus to introduce some structures to wrap input and outputs of our use cases. Those structures are called request and response objects.

Requests and responses

Git tag: step05

Request and response objects are an important part of a clean architecture, as they transport call parameters, inputs and results from outside the application into the use cases layer.

More specifically, requests are objects created from incoming API calls, thus they shall deal with things like incorrect values, missing parameters, wrong formats, etc. Responses, on the other hand, have to contain the actual results of the API calls, but shall also be able to represent error cases and to deliver rich information on what happened.

The actual implementation of request and response objects is completely free, the clean architecture says nothing about them. The decision on how to pack and represent data is up to us.

For the moment we just need a StorageRoomListRequestObject that can be initialized without parameters, so let us create the file tests/use_cases/test_storageroom_list_request_objects.py and put there a test for this object.

fromrentomatic.use_casesimportrequest_objectsasrodeftest_build_storageroom_list_request_object_without_parameters():req=ro.StorageRoomListRequestObject()assertbool(req)isTruedeftest_build_file_list_request_object_from_empty_dict():req=ro.StorageRoomListRequestObject.from_dict({})assertbool(req)isTrue

While at the moment this request object is basically empty, it will come in handy as soon as we start having parameters for the list use case. The code of the StorageRoomListRequestObject is the following and goes into the rentomatic/use_cases/request_objects.py file

classStorageRoomListRequestObject(object):@classmethoddeffrom_dict(cls,adict):returnStorageRoomListRequestObject()def__nonzero__(self):returnTrue

The response object is also very simple, since for the moment we just need a successful response. Unlike the request, the response is not linked to any particular use case, so the test file can be named tests/shared/test_response_object.py

fromrentomatic.sharedimportresponse_objectasrodeftest_response_success_is_true():assertbool(ro.ResponseSuccess())isTrue

and the actual response object is in the file rentomatic/shared/response_object.py

classResponseSuccess(object):def__init__(self,value=None):self.value=valuedef__nonzero__(self):returnTrue__bool__=__nonzero__

Use cases (part 2)

Git tag: step06

Now that we have implemented the request and response object we can change the test code to include those structures. Change the tests/use_cases/test_storageroom_list_use_case.py to contain this code

importpytestfromunittestimportmockfromrentomatic.domainimportmodelsfromrentomatic.use_casesimportrequest_objectsasrofromrentomatic.use_casesimportstorageroom_use_casesasuc@pytest.fixturedefdomain_storagerooms():storageroom_1=models.StorageRoom(code='f853578c-fc0f-4e65-81b8-566c5dffa35a',size=215,price=39,longitude='-0.09998975',latitude='51.75436293',)storageroom_2=models.StorageRoom(code='fe2c3195-aeff-487a-a08f-e0bdc0ec6e9a',size=405,price=66,longitude='0.18228006',latitude='51.74640997',)storageroom_3=models.StorageRoom(code='913694c6-435a-4366-ba0d-da5334a611b2',size=56,price=60,longitude='0.27891577',latitude='51.45994069',)storageroom_4=models.StorageRoom(code='eed76e77-55c1-41ce-985d-ca49bf6c0585',size=93,price=48,longitude='0.33894476',latitude='51.39916678',)return[storageroom_1,storageroom_2,storageroom_3,storageroom_4]deftest_storageroom_list_without_parameters(domain_storagerooms):repo=mock.Mock()repo.list.return_value=domain_storageroomsstorageroom_list_use_case=uc.StorageRoomListUseCase(repo)request_object=ro.StorageRoomListRequestObject.from_dict({})response_object=storageroom_list_use_case.execute(request_object)assertbool(response_object)isTruerepo.list.assert_called_with()assertresponse_object.value==domain_storagerooms

The new version of the rentomatic/use_case/storageroom_use_cases.py file is the following

fromrentomatic.sharedimportresponse_objectasroclassStorageRoomListUseCase(object):def__init__(self,repo):self.repo=repodefexecute(self,request_object):storage_rooms=self.repo.list()returnro.ResponseSuccess(storage_rooms)

Let us consider what we have achieved with our clean architecture up to this point. We have a very lightweight model that can be serialized to JSON and which is completely independent from other parts of the system. The code also contains a use case that, given a repository that exposes a given API, extracts all the models and returns them contained in a structured object.

We are missing some objects, however. For example, we have not implemented any unsuccessful response object or validated the incoming request object.

To explore these missing parts of the architecture let us improve the current use case to accept a filters parameter that represents some filters that we want to apply to the extracted list of models. This will generate some possible error conditions for the input, forcing us to introduce some validation for the incoming request object.

Requests and validation

Git tag: step07

I want to add a filters parameter to the request. Through that parameter the caller can add different filters by specifying a name and a value for each filter (for instance {'price_lt': 100} to get all results with a price lesser than 100).

The first thing to do is to change the request object, starting from the test. The new version of the tests/use_cases/test_storageroom_list_request_objects.py file is the following

fromrentomatic.use_casesimportrequest_objectsasrodeftest_build_storageroom_list_request_object_without_parameters():req=ro.StorageRoomListRequestObject()assertreq.filtersisNoneassertbool(req)isTruedeftest_build_file_list_request_object_from_empty_dict():req=ro.StorageRoomListRequestObject.from_dict({})assertreq.filtersisNoneassertbool(req)isTruedeftest_build_storageroom_list_request_object_with_empty_filters():req=ro.StorageRoomListRequestObject(filters={})assertreq.filters=={}assertbool(req)isTruedeftest_build_storageroom_list_request_object_from_dict_with_empty_filters():req=ro.StorageRoomListRequestObject.from_dict({'filters':{}})assertreq.filters=={}assertbool(req)isTruedeftest_build_storageroom_list_request_object_with_filters():req=ro.StorageRoomListRequestObject(filters={'a':1,'b':2})assertreq.filters=={'a':1,'b':2}assertbool(req)isTruedeftest_build_storageroom_list_request_object_from_dict_with_filters():req=ro.StorageRoomListRequestObject.from_dict({'filters':{'a':1,'b':2}})assertreq.filters=={'a':1,'b':2}assertbool(req)isTruedeftest_build_storageroom_list_request_object_from_dict_with_invalid_filters():req=ro.StorageRoomListRequestObject.from_dict({'filters':5})assertreq.has_errors()assertreq.errors[0]['parameter']=='filters'assertbool(req)isFalse

As you can see I added the assert req.filters is None check to the original two tests, then I added 5 tests to check if filters can be specified and to test the behaviour of the object with an invalid filter parameter.

To make the tests pass we have to change our StorageRoomListRequestObject class. There are obviously multiple possible solutions that you can come up with, and I recommend you to try to find your own. This is the one I usually employ. The file rentomatic/use_cases/request_object.py becomes

importcollectionsclassInvalidRequestObject(object):def__init__(self):self.errors=[]defadd_error(self,parameter,message):self.errors.append({'parameter':parameter,'message':message})defhas_errors(self):returnlen(self.errors)>0def__nonzero__(self):returnFalse__bool__=__nonzero__classValidRequestObject(object):@classmethoddeffrom_dict(cls,adict):raiseNotImplementedErrordef__nonzero__(self):returnTrue__bool__=__nonzero__classStorageRoomListRequestObject(ValidRequestObject):def__init__(self,filters=None):self.filters=filters@classmethoddeffrom_dict(cls,adict):invalid_req=InvalidRequestObject()if'filters'inadictandnotisinstance(adict['filters'],collections.Mapping):invalid_req.add_error('filters','Is not iterable')ifinvalid_req.has_errors():returninvalid_reqreturnStorageRoomListRequestObject(filters=adict.get('filters',None))

Let me review this new code bit by bit.

First of all, two helper objects have been introduced, ValidRequestObject and InvalidRequestObject. They are different because an invalid request shall contain the validation errors, but both can be converted to booleans.

Second, the StorageRoomListRequestObject accepts an optional filters parameter when instantiated. There are no validation checks in the __init__() method because this is considered to be an internal method that gets called when the parameters have already been validated.

Last, the from_dict() method performs the validation of the filters parameter, if it is present. I leverage the collections.Mapping abstract base class to check if the incoming parameter is a dictionary-like object and return either an InvalidRequestObject or a ValidRequestObject instance.

Since we can now tell bad requests from good ones we need to introduce a new type of response as well, to manage bad requests or other errors in the use case.

Responses and failures

Git tag: step08

What happens if the use case encounter an error? Use cases can encounter a wide set of errors: validation errors, as we just discussed in the previous section, but also business logic errors or errors that come from the repository layer. Whatever the error, the use case shall always return an object with a known structure (the response), so we need a new object that provides a good support for different types of failures.

As happened for the requests there is no unique way to provide such an object, and the following code is just one of the possible solutions.

The first thing to do is to expand the tests/shared/test_response_object.py file, adding tests for failures.

importpytestfromrentomatic.sharedimportresponse_objectasresfromrentomatic.use_casesimportrequest_objectsasreq@pytest.fixturedefresponse_value():return{'key':['value1','value2']}@pytest.fixturedefresponse_type():return'ResponseError'@pytest.fixturedefresponse_message():return'This is a response error'

This is some boilerplate code, basically pytest fixtures that we will use in the following tests.

deftest_response_success_is_true(response_value):assertbool(res.ResponseSuccess(response_value))isTruedeftest_response_failure_is_false(response_type,response_message):assertbool(res.ResponseFailure(response_type,response_message))isFalse

Two basic tests to check that both the old ResponseSuccess and the new ResponseFailure objects behave consistently when converted to boolean.

deftest_response_success_contains_value(response_value):response=res.ResponseSuccess(response_value)assertresponse.value==response_value

The ResponseSuccess object contains the call result in the value attribute.

deftest_response_failure_has_type_and_message(response_type,response_message):response=res.ResponseFailure(response_type,response_message)assertresponse.type==response_typeassertresponse.message==response_messagedeftest_response_failure_contains_value(response_type,response_message):response=res.ResponseFailure(response_type,response_message)assertresponse.value=={'type':response_type,'message':response_message}

These two tests ensure that the ResponseFailure object provides the same interface provided by the successful one and that the type and message parameter are accessible.

deftest_response_failure_initialization_with_exception():response=res.ResponseFailure(response_type,Exception('Just an error message'))assertbool(response)isFalseassertresponse.type==response_typeassertresponse.message=="Exception: Just an error message"deftest_response_failure_from_invalid_request_object():response=res.ResponseFailure.build_from_invalid_request_object(req.InvalidRequestObject())assertbool(response)isFalsedeftest_response_failure_from_invalid_request_object_with_errors():request_object=req.InvalidRequestObject()request_object.add_error('path','Is mandatory')request_object.add_error('path',"can't be blank")response=res.ResponseFailure.build_from_invalid_request_object(request_object)assertbool(response)isFalseassertresponse.type==res.ResponseFailure.PARAMETERS_ERRORassertresponse.message=="path: Is mandatory\npath: can't be blank"

We sometimes want to create responses from Python exceptions that can happen in the use case, so we test that ResponseFailure objects can be initialized with a generic exception.

And last we have the tests for the build_from_invalid_request_object() method that automate the initialization of the response from an invalid request. If the request contains errors (remember that the request validates itself), we need to put them into the response message.

The last test uses a class attribute to classify the error. The ResponseFailure class will contain three predefined errors that can happen when running the use case, namely RESOURCE_ERROR, PARAMETERS_ERROR, and SYSTEM_ERROR. This categorization is an attempt to capture the different types of issues that can happen when dealing with an external system through an API. RESOURCE_ERROR contains all those errors that are related to the resources contained in the repository, for instance when you cannot find an entry given its unique id. PARAMETERS_ERROR describes all those errors that occur when the request parameters are wrong or missing. SYSTEM_ERROR encompass the errors that happen in the underlying system at operating system level, such as a failure in a filesystem operation, or a network connection error while fetching data from the database.

The use case has the responsibility to manage the different error conditions arising from the Python code and to convert them into an error description made of one of the three types I just described and a message.

Let's write the ResponseFailure class that makes the tests pass. This can be the initial definition of the class. Put it in rentomatic/shared/response_object.py

classResponseFailure(object):RESOURCE_ERROR='ResourceError'PARAMETERS_ERROR='ParametersError'SYSTEM_ERROR='SystemError'def__init__(self,type_,message):self.type=type_self.message=self._format_message(message)def_format_message(self,msg):ifisinstance(msg,Exception):return"{}: {}".format(msg.__class__.__name__,"{}".format(msg))returnmsg

Through the _format_message() method we enable the class to accept both string messages and Python exceptions, which is very handy when dealing with external libraries that can raise exceptions we do not know or do not want to manage.

@propertydefvalue(self):return{'type':self.type,'message':self.message}

This property makes the class comply with the ResponseSuccess API, providing the value attribute, which is an aptly formatted dictionary.

def__nonzero__(self):returnFalse__bool__=__nonzero__@classmethoddefbuild_from_invalid_request_object(cls,invalid_request_object):message="\n".join(["{}: {}".format(err['parameter'],err['message'])forerrininvalid_request_object.errors])returncls(cls.PARAMETERS_ERROR,message)

As explained before, the PARAMETERS_ERROR type encompasses all those errors that come from an invalid set of parameters, which is the case of this function, that shall be called whenever the request is wrong, which means that some parameters contain errors or are missing.

Since building failure responses is a common activity it is useful to have helper methods, so I add three tests for the building functions to the tests/shared/test_response_object.py file

deftest_response_failure_build_resource_error():response=res.ResponseFailure.build_resource_error("test message")assertbool(response)isFalseassertresponse.type==res.ResponseFailure.RESOURCE_ERRORassertresponse.message=="test message"deftest_response_failure_build_parameters_error():response=res.ResponseFailure.build_parameters_error("test message")assertbool(response)isFalseassertresponse.type==res.ResponseFailure.PARAMETERS_ERRORassertresponse.message=="test message"deftest_response_failure_build_system_error():response=res.ResponseFailure.build_system_error("test message")assertbool(response)isFalseassertresponse.type==res.ResponseFailure.SYSTEM_ERRORassertresponse.message=="test message"

We add the relevant methods to the class and change the build_from_invalid_request_object() method to leverage the build_parameters_error() new method. Change the rentomatic/shared/response_object.py file to contain this code

@classmethoddefbuild_resource_error(cls,message=None):returncls(cls.RESOURCE_ERROR,message)@classmethoddefbuild_system_error(cls,message=None):returncls(cls.SYSTEM_ERROR,message)@classmethoddefbuild_parameters_error(cls,message=None):returncls(cls.PARAMETERS_ERROR,message)@classmethoddefbuild_from_invalid_request_object(cls,invalid_request_object):message="\n".join(["{}: {}".format(err['parameter'],err['message'])forerrininvalid_request_object.errors])returncls.build_parameters_error(message)

Use cases (part 3)

Git tag: step09

Our implementation of responses and requests is finally complete, so now we can implement the last version of our use case. The use case correctly returns a ResponseSuccess object but is still missing a proper validation of the incoming request.

Let's change the test in the tests/use_cases/test_storageroom_list_use_case.py file and add two more tests. The resulting set of tests (after the domain_storagerooms fixture) is the following

importpytestfromunittestimportmockfromrentomatic.domain.storageroomimportStorageRoomfromrentomatic.sharedimportresponse_objectasresfromrentomatic.use_casesimportrequest_objectsasreqfromrentomatic.use_casesimportstorageroom_use_casesasuc@pytest.fixturedefdomain_storagerooms():[...]deftest_storageroom_list_without_parameters(domain_storagerooms):repo=mock.Mock()repo.list.return_value=domain_storageroomsstorageroom_list_use_case=uc.StorageRoomListUseCase(repo)request_object=req.StorageRoomListRequestObject.from_dict({})response_object=storageroom_list_use_case.execute(request_object)assertbool(response_object)isTruerepo.list.assert_called_with(filters=None)assertresponse_object.value==domain_storagerooms

This is the test we already wrote, but the assert_called_with() method is called with filters=None to reflect the added parameter. The import line has slightly changed as well, given that we are now importing both response_objects and request_objects. The domain_storagerooms fixture has not changed and has been omitted from the code snippet to keep it short.

deftest_storageroom_list_with_filters(domain_storagerooms):repo=mock.Mock()repo.list.return_value=domain_storageroomsstorageroom_list_use_case=uc.StorageRoomListUseCase(repo)qry_filters={'a':5}request_object=req.StorageRoomListRequestObject.from_dict({'filters':qry_filters})response_object=storageroom_list_use_case.execute(request_object)assertbool(response_object)isTruerepo.list.assert_called_with(filters=qry_filters)assertresponse_object.value==domain_storagerooms