Introduction
Somewhere in between getting your python project to run and getting to the point where even a debugger won’t help you find a bug, you might realize that creating a log file of what your program is doing could be really beneficial. I’ve typically found that once you get a working version of your program, you’ll want to really understand what is happening during subsequent executions of the program. The simplest (note: not best) way to accomplish this is to put lots of print statements in your code. However, this is really a bad idea as you’re going to be getting lots of print out in your console and you are likely going to be deleting these print statements once you get a bug resolved. So what’s a better approach? Logging!
Python has a built-in library called ‘logging’, which is a great library for logging your program’s execution to a file. The tutorial for ‘logging’ provides a good range of examples from basic to more advanced uses of the library. However, I want to illustrate how I got the ‘logging’ module to work successfully in this blog post.
The source code for these examples can be found on GitLab, organized by tags:
https://gitlab.com/patkennedy79/python_logging
Example #1 – Single File Application
tag: v0.1 – https://gitlab.com/patkennedy79/python_logging/tree/v0.1
The first example is a simple program to help illustrate the basic capabilities of the ‘logging’ module. This program is comprised of a single file named app.py that contains a single class:
class FirstClass(object):
def __init__(self):
self.current_number = 0
def increment_number(self):
self.current_number += 1
def decrement_number(self):
self.current_number -= 1
def clear_number(self):
self.current_number = 0
number = FirstClass()
number.increment_number()
number.increment_number()
print "Current number: %s" % str(number.current_number)
number.clear_number()
print "Current number: %s" % str(number.current_number)
You can run this program by executing:
$ python run.py Current number: 2 Current number: 0
This program uses two print statements to print to the console. This is fine for getting a program to work, but switching over to logging messages is a better long-term approach.
Configuring the Logging Module
The configuration of the logging module can get complex as you start to specify more and more details of how the logging should be performed. I’ve found the following sequence provides a good configuration for:
- setting the logging severity level
- setting the file to log messages to
- setting the format of the log messages
In order to configure the logging module, the constructor for the FirstClass class should be updated to:
def __init__(self):
self.current_number = 0
self.logger = logging.getLogger(__name__)
# Create the Logger
self.logger = logging.getLogger(__name__)
self.logger.setLevel(logging.WARNING)
# Create the Handler for logging data to a file
logger_handler = logging.FileHandler('python_logging.log')
logger_handler.setLevel(logging.WARNING)
# Create a Formatter for formatting the log messages
logger_formatter = logging.Formatter('%(name)s - %(levelname)s - %(message)s')
# Add the Formatter to the Handler
logger_handler.setFormatter(logger_formatter)
# Add the Handler to the Logger
self.logger.addHandler(logger_handler)
self.logger.info('Completed configuring logger()!')
The constructor configures the usage of the logging module and finishes by logging a message that the configuration of the logger is completed.
Adding Log Messages
In order to add log messages, you can utilize one of the methods from the logging module to log different severity log messages:
def increment_number(self):
self.current_number += 1
self.logger.warning('Incrementing number!')
self.logger.info('Still incrementing number!!')
def clear_number(self):
self.current_number = 0
self.logger.warning('Clearing number!')
self.logger.info('Still clearing number!!')
If you run the program again, you’ll still see the same console output.
$ python run.py Current number: 2 Current number: 0
Logging Severity Levels
To see what the logging module is doing, check the log file that was created:
$ cat python_logging.log __main__ - WARNING - Incrementing number! __main__ - WARNING - Incrementing number! __main__ - WARNING - Clearing number!
Interesting, is that what you expected? I initially thought that there should be two “Still incrementing number!!” statements, but they don’t get displayed. Why? Well, the ‘logging’ module has 5 severity levels:
- DEBUG (lowest)
- INFO
- WARNING
- ERROR
- CRITICAL (highest)
Our program is using the default setting (WARNING) for the logging severity, which means that any log message with a lower severity than WARNING will not be display. Hence, the INFO messages are not displayed.
Change the following lines in __init__():
# Create the Logger
self.logger = logging.getLogger(__name__)
self.logger.setLevel(logging.WARNING)
# Create the Handler for logging data to a file
logger_handler = logging.FileHandler('python_logging.log')
logger_handler.setLevel(logging.WARNING)
Now check the log file to see that all of the log messages are included:
$ cat python_logging.log __main__ - WARNING - Incrementing number! __main__ - WARNING - Incrementing number! __main__ - WARNING - Clearing number! __main__ - INFO - Completed configuring logger()! __main__ - WARNING - Incrementing number! __main__ - INFO - Still incrementing number!! __main__ - WARNING - Incrementing number! __main__ - INFO - Still incrementing number!! __main__ - WARNING - Clearing number! __main__ - INFO - Still clearing number!!
Example #2 – Logging in a Module
tag: v0.2 – https://gitlab.com/patkennedy79/python_logging/tree/v0.2
The second example is slightly more complex, as it updates the structure of the program to include a package with a single module:
python_logging python_logging __init__.py first_class.py run.py
The first_class.py file basically contains the FirstClass class that was created in the first example:
import logging
class FirstClass(object):
def __init__(self):
self.current_number = 0
self.logger = logging.getLogger(__name__)
# Create the Logger
self.logger = logging.getLogger(__name__)
self.logger.setLevel(logging.DEBUG)
# Create the Handler for logging data to a file
logger_handler = logging.FileHandler('python_logging.log')
logger_handler.setLevel(logging.DEBUG)
# Create a Formatter for formatting the log messages
logger_formatter = logging.Formatter('%(name)s - %(levelname)s - %(message)s')
# Add the Formatter to the Handler
logger_handler.setFormatter(logger_formatter)
# Add the Handler to the Logger
self.logger.addHandler(logger_handler)
self.logger.info('Completed configuring logger()!')
def increment_number(self):
self.current_number += 1
self.logger.warning('Incrementing number!')
self.logger.info('Still incrementing number!!')
def decrement_number(self):
self.current_number -= 1
def clear_number(self):
self.current_number = 0
self.logger.warning('Clearing number!')
self.logger.info('Still clearing number!!')
In order to utilize this module, update the run.py file in the top-level directory to import the FirstClass class to utilize it:
from python_logging.first_class import FirstClass number = FirstClass() number.increment_number() number.increment_number() print "Current number: %s" % str(number.current_number) number.clear_number() print "Current number: %s" % str(number.current_number)
Check that the log file is unchanged.
Example #3: Logging in a Package (Part I)
tag: v0.3 – https://gitlab.com/patkennedy79/python_logging/tree/v0.3
The third example adds a second class to the python_logging package to show how to configure the logging module for a whole package. Here is the structure of this example:
python_logging
python_logging
__init__.py
first_class.py
second_class.py
run.py
Here is the basic version of the second_class.py file:
class SecondClass(object):
def __init__(self):
self.enabled = False
def enable_system(self):
self.enabled = True
def disable_system(self):
self.enabled = False
Your first inclination might be to duplicate the configuration of the logger in the constructor of this class (copy from first_class.py). This would result in a lot of unnecessary, repetitive code. A better option is to move the configuration of the logger to the __init__.py file:
from os import path, remove
import logging
import logging.config
from .first_class import FirstClass
from .second_class import SecondClass
# If applicable, delete the existing log file to generate a fresh log file during each execution
if path.isfile("python_logging.log"):
remove("python_logging.log")
# Create the Logger
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
# Create the Handler for logging data to a file
logger_handler = logging.FileHandler('python_logging.log')
logger_handler.setLevel(logging.DEBUG)
# Create a Formatter for formatting the log messages
logger_formatter = logging.Formatter('%(name)s - %(levelname)s - %(message)s')
# Add the Formatter to the Handler
logger_handler.setFormatter(logger_formatter)
# Add the Handler to the Logger
logger.addHandler(logger_handler)
logger.info('Completed configuring logger()!')
Depending on the type of program you are creating, you might find it beneficial to delete any existing log files prior to logging any new messages from this execution of the program. One option to consider if you want to maintain an on-going log of an application is the RotatingFileHandler within the logging module.
Now that the configuration of the logging module is done in __init__.py, the second_class.py file can be greatly simplified to utilize the logger, instead of having to worry about configuring it first:
import logging
class SecondClass(object):
def __init__(self):
self.enabled = False
self.logger = logging.getLogger(__name__)
def enable_system(self):
self.enabled = True
self.logger.warning('Enabling system!')
self.logger.info('Still enabling system!!')
def disable_system(self):
self.enabled = False
self.logger.warning('Disabling system!')
self.logger.info('Still disabling system!!')
There are similar updates to the first_class.py file.
Finally, the updates to __init__.py result in needing the following updates to run.py:
from python_logging import FirstClass, SecondClass number = FirstClass() number.increment_number() number.increment_number() print "Current number: %s" % str(number.current_number) number.clear_number() print "Current number: %s" % str(number.current_number) system = SecondClass() system.enable_system() system.disable_system() print "Current system configuration: %s" % str(system.enabled)
Try running the program again and looking at the log file:
$ cat python_logging.log python_logging - INFO - Completed configuring logger()! python_logging.first_class - WARNING - Incrementing number! python_logging.first_class - INFO - Still incrementing number!! python_logging.first_class - WARNING - Incrementing number! python_logging.first_class - INFO - Still incrementing number!! python_logging.first_class - WARNING - Clearing number! python_logging.first_class - INFO - Still clearing number!! python_logging.second_class - WARNING - Enabling system! python_logging.second_class - INFO - Still enabling system!! python_logging.second_class - WARNING - Disabling system! python_logging.second_class - INFO - Still disabling system!!
Notice how the module names are printed! This is a really handy feature to quickly identify where specific operations are happening.
Example #4: Logging in a Package (Part II)
tag: v0.4 – https://gitlab.com/patkennedy79/python_logging/tree/v0.4
The fourth (and final) example expands upon the logging capability that was added to a package by adding an input file (JSON) to configure the logger. Keep an eye out for how the log messages are unaffected by this configuration change…
The first change for this example is to __init__.py to change the configuration of the logger to utilize a JSON input file:
from os import path, remove
import logging
import logging.config
import json
from .first_class import FirstClass
from .second_class import SecondClass
# If applicable, delete the existing log file to generate a fresh log file during each execution
if path.isfile("python_logging.log"):
remove("python_logging.log")
with open("python_logging_configuration.json", 'r') as logging_configuration_file:
config_dict = json.load(logging_configuration_file)
logging.config.dictConfig(config_dict)
# Log that the logger was configured
logger = logging.getLogger(__name__)
logger.info('Completed configuring logger()!')
Now that we have the code to process the input file, let’s define the input file (python_logging_configuration.json). Make sure to include this file in your top-level folder so that it can be easily identified by the python interpreter. You should be running this program from the top-level folder, so that will make the input JSON file accessible by the python interpreter since it is in the current/working directory.
Here is the configuration file:
{
"version": 1,
"disable_existing_loggers": false,
"formatters": {
"simple": {
"format": "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
}
},
"handlers": {
"file_handler": {
"class": "logging.FileHandler",
"level": "DEBUG",
"formatter": "simple",
"filename": "python_logging.log",
"encoding": "utf8"
}
},
"root": {
"level": "DEBUG",
"handlers": ["file_handler"]
}
}
Run the program again and take a look at the log file:
$ cat python_logging.log 2017-03-09 22:32:01,846 - python_logging - INFO - Completed configuring logger()! 2017-03-09 22:32:01,847 - python_logging.first_class - WARNING - Incrementing number! 2017-03-09 22:32:01,847 - python_logging.first_class - INFO - Still incrementing number!! 2017-03-09 22:32:01,847 - python_logging.first_class - WARNING - Incrementing number! 2017-03-09 22:32:01,847 - python_logging.first_class - INFO - Still incrementing number!! 2017-03-09 22:32:01,847 - python_logging.first_class - WARNING - Clearing number! 2017-03-09 22:32:01,847 - python_logging.first_class - INFO - Still clearing number!! 2017-03-09 22:32:01,848 - python_logging.second_class - WARNING - Enabling system! 2017-03-09 22:32:01,848 - python_logging.second_class - INFO - Still enabling system!! 2017-03-09 22:32:01,848 - python_logging.second_class - WARNING - Disabling system! 2017-03-09 22:32:01,848 - python_logging.second_class - INFO - Still disabling system!!
The log file is almost the same, but the date/time stamp has been added to each log message. I like this format of logging messages: date/time – package.module – log message
If JSON is not your favorite, the input file can also be defined as a YAML file. Here’s an example: Good Logging Practice in Python
Conclusion
The idea of moving beyond print statements to actually logging messages is made so easy thanks to the logging module that is built-in to python. The logging module requires a bit of configuration, but this is a small price to pay to such an easy-to-use module.
I’ve found that having a log file for a program, especially command-line applications, provides a great way to understand what the program is doing. You might go weeks without looking at the log file if things are working well, but having a lot of data about your program’s execution readily available is so beneficial when things go wrong.
References
Python Logging Module Documentation: https://docs.python.org/3/howto/logging.html
Blog Post on Good Logging Practice in Python: https://fangpenlin.com/posts/2012/08/26/good-logging-practice-in-python/
Blog Post on Logging in Python: https://www.relaxdiego.com/2014/07/logging-in-python.html