Introduction
This blog post provides an introduction to unit testing a Flask application. I’ve been on a big unit testing kick at work recently which is spilling over into updating the unit tests for my personal projects. Therefore, I thought it would be a good time to document the basics of unit testing a Flask application.
There seems to be a lot of opinions about unit testing amongst software developers, but I believe the following:
- 100% code coverage using unit tests does not mean your code works, but it’s a good indicator that the code has been developed well
- No unit tests mean that the code does not work and cannot be expanded upon
The second belief might be a bit extreme, but if I inherit a software module that does not have unit tests, I’m already assuming that it does not run properly and has not been maintained. Simply put, the lack of unit tests leads to a negative impression of the software module.
I believe that you should create an environment for developing unit test that is both easy to use and fun to use. These may sound like very fluffy words, but they have meaning for me:
- create helper methods to allow quick generation of unit tests
- provide quick feedback on progress of unit testing
- create an environment that allows for tests to be added quickly
- make sure that the unit tests can be executed with a single command
- provide visual feedback of path coverage
As with so many things in software development, if it’s easy to use, it will be used. If it’s tough to use, it won’t be used as much as it should be.
Unit Test Frameworks
The two unit test frameworks for python that I’ve used are:
- unittest– built-in unit test framework that is based on the xUnit framework
- py.test– module for building unit tests
Both work really well, but I tend to prefer unittest since it’s built-in to the python library (there is a wealth of incredible modules built-in to python!). I’ll be using unittest throughout this blog post.
If you want to see just how many options there are for tools to help with unit testing in python, check out this listing of python testing tools. An incredible number of tools available!
Where to Store Your Unit Tests
Based on the flexibility that using unit test runners gives you, you could probably store your unit test files in any location in your Flask application. However, I find it best to store the unit tests (in this case located in the ‘tests’ directory) at the same level as the files that you are going to be testing:
$ tree ├── instance ├── migrations ├── project │ ├── __init__.py │ ├── models.py │ ├── recipes │ ├── static │ ├── templates │ ├── tests │ │ ├── test_basic.py │ │ ├── test_recipes.py │ │ └── test_users.py │ └── users ├── requirements.txt └── run.py
In this structure, the unit tests are stored in the ‘tests’ directory and the unit tests are going to be focused on testing the functionality in the ‘recipes’ and ‘users’ modules, which are at the same level.
Creating a Basic Unit Test File
There are a lot of important principles to follow when writing unit tests, but I really believe it’s important to create an environment that makes writing unit tests easy. I’m a fan of writing unit tests, but I know a lot of software developers that just don’t like it. Therefore, I think it’s important to develop a unit testing structure that has a lot ‘helper’ functions to facilitate writing unit tests.
With that in mind, let’s create a simple unit test file that is not testing our Flask application, but is simply showing the structure of a unit test:
# project/test_basic.py
import os
import unittest
from project import app, db, mail
TEST_DB = 'test.db'
class BasicTests(unittest.TestCase):
############################
#### setup and teardown ####
############################
# executed prior to each test
def setUp(self):
app.config['TESTING'] = True
app.config['WTF_CSRF_ENABLED'] = False
app.config['DEBUG'] = False
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///' + \
os.path.join(app.config['BASEDIR'], TEST_DB)
self.app = app.test_client()
db.drop_all()
db.create_all()
# Disable sending emails during unit testing
mail.init_app(app)
self.assertEqual(app.debug, False)
# executed after each test
def tearDown(self):
pass
###############
#### tests ####
###############
def test_main_page(self):
response = self.app.get('/', follow_redirects=True)
self.assertEqual(response.status_code, 200)
if __name__ == "__main__":
unittest.main()
This unit test file creates a class, BasicTests, based on the unittest.TestCase class. Within this class, the setUp() and tearDown() methods are defined. This is really critical, as the setUp() method is called prior to each unit test executing and the tearDown() method is called after each unit test finishes executing.
Since we want to test out the functionality of the web application, this unit test uses a SQLite database for storing data instead of the typical Postgres database that we’ve been using. Why I’ve seen lots of discussions about whether or not this is a good idea, I feel like it really simplifies the development of the unit tests, so I’m in favor of it.
Running the Unit Tests
If you want to run the unit test that was just created, the easiest way is to just execute that file:
$ python project/tests/test_basic.py . ---------------------------------------------------------------------- Ran 1 test in 0.122s OK
Please take note of the directory that the unit test was run from (top-level folder of the Flask application). Why can’t we just run from the directory where the unit test file is located? We would have this option available for simple unit test, but since we’ll importing the ‘app’, ‘db’, and ‘mail’ objects, we need to be at a location where those are discoverable. Therefore, running at the top-level directory of the Flask application allows the python interpreter to find these objects within the ‘project’ directory.
Taking it one step further, I’d highly recommend using Nose2 as the unit test runner. A unit test runner provides the ability to easily detect the unit tests in your project and then execute them.
The easiest way to run Nose2 is simply to call the executable from the top-level directory:
$ nose2
This command will find all of the unit tests (as long as the files start with test_*.py) and execute them. I’m a bit preferential to using the verbose mode:
$ nose2 -v
If you just want to run a single unit test file, you can still use Nose2:
$ nose2 -v project.tests.test_basic test_main_page2 (project.tests.test_basic.BasicTests) ... ok ---------------------------------------------------------------------- Ran 1 test in 0.130s OK
I’ve found Nose2 to be so easy to use and I highly recommend using it for running your unit tests.
Adding Helper Methods to Facilitate Developing More Unit Tests
For the unit testing in this blog post, we’re going to be focused on testing some of the user management aspects of the Flask application. As such, we’re likely going to be doing a lot of logging in, logging out, and registering for a new account. Let’s simplify the development of future unit tests by creating some helper methods to perform these steps:
########################
#### helper methods ####
########################
def register(self, email, password, confirm):
return self.app.post(
'/register',
data=dict(email=email, password=password, confirm=confirm),
follow_redirects=True
)
def login(self, email, password):
return self.app.post(
'/login',
data=dict(email=email, password=password),
follow_redirects=True
)
def logout(self):
return self.app.get(
'/logout',
follow_redirects=True
)
The ‘register’ method registers a new user by sending a POST command to the ‘/register’ URL of the Flask application. The ‘login’ method logs a user in by sending a POST command to the ‘/login’ URL of the Flask application. The ‘logout’ method logs a user out of the application by sending a GET command to the ‘/logout’ URL of the Flask application. These will come in handy when we…
…add a test to make sure we can register a new user:
def test_valid_user_registration(self):
response = self.register('patkennedy79@gmail.com', 'FlaskIsAwesome', 'FlaskIsAwesome')
self.assertEqual(response.status_code, 200)
self.assertIn(b'Thanks for registering!', response.data)
This unit test checks that we can successfully register a new user and receive a positive confirmation. We’re testing a nominal situation here, but it’s also good to check what happens when off-nominal data is provided to the application. Let’s try registering a new user where the confirmation password does not match the original password:
def test_invalid_user_registration_different_passwords(self):
response = self.register('patkennedy79@gmail.com', 'FlaskIsAwesome', 'FlaskIsNotAwesome')
self.assertIn(b'Field must be equal to password.', response.data)
Try running the unit tests and you should see that all three unit tests pass:
$ nose2 -v project.tests.test_basic test_invalid_user_registration_different_passwords (project.tests.test_basic.BasicTests) ... ok test_main_page (project.tests.test_basic.BasicTests) ... ok test_valid_user_registration (project.tests.test_basic.BasicTests) ... ok ---------------------------------------------------------------------- Ran 3 tests in 3.544s OK
One of the important checks that is in the application is to prevent duplicate emails being used for registration. Let’s check that this functionality works:
def test_invalid_user_registration_duplicate_email(self):
response = self.register('patkennedy79@gmail.com', 'FlaskIsAwesome', 'FlaskIsAwesome')
self.assertEqual(response.status_code, 200)
response = self.register('patkennedy79@gmail.com', 'FlaskIsReallyAwesome', 'FlaskIsReallyAwesome')
self.assertIn(b'ERROR! Email (patkennedy79@gmail.com) already exists.', response.data)
Running the unit tests:
$ nose2 -v project.tests.test_basic test_invalid_user_registration_different_passwords (project.tests.test_basic.BasicTests) ... ok test_invalid_user_registration_duplicate_email (project.tests.test_basic.BasicTests) ... ok test_main_page (project.tests.test_basic.BasicTests) ... ok test_valid_user_registration (project.tests.test_basic.BasicTests) ... ok ---------------------------------------------------------------------- Ran 4 tests in 10.442s OK
Excellent!
Please take note that the order that the unit tests are listed in test_*.py does not determine the order in which the unit tests are run. This is really important to understand, as each unit test needs to be self-contained. Don’t count on unit test #1 registering a new user and unit test #2 logging that user in.
Code Coverage
In order to check the code coverage of your unit tests, I’d recommend using the Coverage module. This module works great with unit tests that have been written using the unittest model.
Start by installing the Coverage module and updating your list of modules for your project:
$ pip install coverage $ pip freeze > requirements.txt
The order of commands to execute when using the coverage module is:
- coverage run …
- coverage report …
The ‘run’ command runs the unit tests and collects the data for determining the code coverage. The ‘report’ command shows a basic text output of the code coverage.
For example, I’ve written some unit tests for the application and the here is how I use the Coverage module:
$ coverage run project/tests/test_basic.py $ coverage report project/users/*.py Name Stmts Miss Cover ----------------------------------------------- project/users/__init__.py 0 0 100% project/users/forms.py 14 0 100% project/users/views.py 177 118 33% ----------------------------------------------- TOTAL 191 118 38%
If you add the -m flag, you can see which lines are not being tested:
$ coverage report -m project/users/*.py Name Stmts Miss Cover Missing --------------------------------------------------------- project/users/__init__.py 0 0 100% project/users/forms.py 14 0 100% project/users/views.py 177 118 33% 33-35, 69-80, 109-124, 130-136, 141-159, 164-179, 184-206, 212, 218-238, 244-254, 260-266, 272-277 --------------------------------------------------------- TOTAL 191 118 38%
This text-based output is nice to get a summary of the code coverage, but just seeing line numbers that are not being testing in a file is just not that beneficial. Luckily, the Coverage module is able to generate HTML to provide better insight into which lines are being tested and which are not. You can generate the HTML output by running:
$ coverage html project/users/*.py
Now you can go to navigate to your project’s folder, open the newly created ‘htmlcov’ directory, and open the index.html file. In your web browser, you should see a summary of the path coverage for this directory:
By clicking on views.py, you get a detailed view of the file contents including a color-coded line-by-line view of which lines are tested by the unit tests and which are not:
Conclusion
This blog post was intended to show you how to develop unit tests for a Flask application with a focus on using the right tools to make the process of writing and running unit tests as easy as possible. The recommended modules for unit testing a Flask application are:
– unittest – built-in python module for developing unit tests
– nose2 – runner for identifying and running the unit tests
– Coverage – seeing the code coverage of your unit tests
To recap, I think that the following aspects are needed to have a functioning setup for writing and running unit tests:
– create helper methods to allow quick generation of unit tests
– provide quick feedback on progress of unit testing
– create an environment that allows for tests to be added quickly
– make sure that the unit tests can be executed with a single command
– provide visual feedback of path coverage
I’ve found that unit tests are critical to testing out any software module that I write. Writing unit tests should be an efficient process that can be greatly simplified by setting up a proper infrastructure.
For the source code utilized in this blog post, please see my GitLab repository.