Mikko Ohtamaa: Python standard logging pattern

(this article originally appeared in Websauna documentation)

1. Introduction

Python standard library provides logging module as a de facto solution for libraries and applications to log their behavior. logging is extensively used by Websauna, Pyramid, SQLAlchemy and other Python packages.

• Python logging subsystem can be configured using external configuration file and the logging configuration format is specified in Python standard library.
• Python logger can be individually turned on, off and their verbosity adjusted on per module basis. For example by default, Websauna development server sets SQLALchemy logging level to INFO instead of DEBUG to avoid flooding the console with verbose SQL logs. However if you are debugging issues related to a database you might want to set the SQLAlchemy logging back to INFO.
• Logging is preferred diagnose method over print statements cluttered around source code.. Well designed logging calls can be left in the source code and later turned back on if the problems must be diagnosed further.
• Python logging output can be directed to console, file, rotating file, syslog, remote server, email, etc.

2. Log colorization

• Websauna uses rainbow_logging_handler which colorizes the logs, making it easier to read them in the console of the development web server.

3. Standard logging pattern

A common logging pattern in Python is:

import logging


logger = logging.getLogger(__name__)


def my_view(request):
    logger.debug("my_view got request: %s", request)
    logger.info("my_view got request: %s", request)
    logger.error("my_view got request: %s and BAD STUFF HAPPENS", request)

    try:
        raise RuntimeError("OH NOES")
    except Exception as e:
        # Let's log full traceback even when we ignore this exception
        # and it's not risen again
        logger.exception(e)

• This names a logger based on a module so you can switch logger on/off on module basis.
• Pass logged objects to logging.Logger.debug() and co. as full and let the logger handle the string formatting. This allows intelligent display of logged objects when using non-console logging solutions like Sentry.
• Use logging.Logger.exception() to report exceptions. This will record the full traceback of the exception and not just the error message.

Please note that although this logging pattern is common, it’s not a universal solution. For example if you are creating third party APIs, you might want to pass the logger to a class instance of an API, so that the API consumer can take over the logger setup and there is no inversion of control.

4. Changing logging level using INI settings

Websauna defines development web server log levels in its core development.ini. Your Websauna application inherits settings from this file and can override them for each logger in the conf/development.ini file of your application.

For example to set SQLAlchemy and transaction logging level to more verbose you can do:

[logger_sqlalchemy]level=DEBUG[logger_transaction]level=DEBUG

Now console is flooded with very verbose logging:

[2016-05-2220:39:55,429][sqlalchemy.engine.base.Engine_begin_impl]BEGIN(implicit)[2016-05-2220:39:55,429][txn.123145312813056__init__]newtransaction[2016-05-2220:39:55,429][sqlalchemy.engine.base.Engine_execute_context]SELECTusers.passwordASusers_password,users.idASusers_id,users.uuidASusers_uuid,users.usernameASusers_username,users.emailASusers_email,users.created_atASusers_created_at,users.updated_atASusers_updated_at,users.activated_atASusers_activated_at,users.enabledASusers_enabled,users.last_login_atASusers_last_login_at,users.last_login_ipASusers_last_login_ip,users.user_dataASusers_user_data,users.last_auth_sensitive_operation_atASusers_last_auth_sensitive_operation_at,users.activation_idASusers_activation_id

5. Initialization loggers from INI file

If you need to initialize loggers in your own applications see websauna.system.devop.cmdline.setup_logging() for how Websauna picks up loggers from INI configuration file.

6. More information

How Websauna logs username and email for every internal server error. It’s impressive service if your devops teams calls a customer on a second an error happens and guide the customer around the error. As a bonus if using Sentry you will see the Gravatar profile image of the user when viewing the exception.

Logbook is an alternative for Python standard library logging if performance is critical or the application has more complex logging requirements .

Discussion about log message formatting and why we are still using old style string formatting.

structlog package– add context to your logged messages like user id or HTTP request URL.