Here are 10 things we learned writing pytest code that might come in handy:
1. Testing package structure
People new to pytest are often thrown off by this:
$ tree
.├── src│ ├── __init__.py│ └── script.py└── tests└── test_script.py2 directories, 3 files$ more src/script.py
def hello(): return 'hello'$ more tests/test_script.py
from src.script import hellodef test_hello(): assert hello() == "hello"$ pytest
...ImportError while importing test module '/Users/bobbelderbos/Downloads/demo/tests/test_script.py'.Hint: make sure your test modules/packages have valid Python names.Traceback:/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/importlib/__init__.py:127: in import_module return _bootstrap._gcd_import(name[level:], package, level)tests/test_script.py:1: in <module> from src.script import helloE ModuleNotFoundError: No module named 'src'============================================================================================= short test summary info ==============================================================================================ERROR tests/test_script.py!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!================================================================================================= 1 error in 0.42s =================================================================================================$ touch tests/__init__.py
$ pytest
...tests/test_script.py . [100%]================================================================================================ 1 passed in 0.19s =================================================================================================So the tests directory needs an __init__.py file as well.
Setting your project up with Poetry makes this a lot easier / automatic.
If you don't turn your code directory into a package (so not including an __init__.py file), you might want to use pytest-pythonpath:
... a py.test plugin for adding to the PYTHONPATH from the pytests.ini file before tests run.
Thanks Martin for telling us about this plugin.
2. Organize your fixtures
You can use a conftest.py file to create your fixtures (setup and tear down code) for reuse across your test modules.
See more info in the documentation and a practical example in one of our projects. This will definitely make your test modules leaner.
3. Filter out particular tests
You can use pytest's -k switch to filter tests by expression:
-kEXPRESSIONonlyruntestswhichmatchthegivensubstringexpression. Anexpressionisapythonevaluatableexpressionwhereallnamesaresubstring-matchedagainsttestnamesandtheirparentclasses. Example:
-k'test_method or test_other'matchesalltestfunctionsandclasseswhosenamecontains'test_method'or'test_other', while-k'nottest_method' matches those that don'tcontain'test_method'intheirnames. ...
Or you can mark them with @pytest.mark, for example:
@pytest.mark.slowdeftest_func_slow():passThen target those "marked" tests individually. The docs show a good example of how to do this.
This can be useful if you want to target fast vs. slow tests for example.
Another cool use case is @pytest.mark.skipif to skip a test based on a condition:
# comments.py (code with a syntax error)deftime_printer():thislineshouldbecommented# test_comments.pyimportpytestdef_can_import():try:importcomments# noqa F401returnTrueexceptIndentationError:returnFalsedeftest_import_fails_because_not_all_garbage_commented():ifnot_can_import():raisepytest.fail(…@pytest.mark.skipif(not_can_import(),reason="Only run if import works")deftest_output_time_printer_with_time_arg_returns_string(capfd):# tests past successful import ...# nicer output + skip other test$pytest[outputtruncated]EFailed:comments.pyraisedanIndentationError,didyoucommentitproperly?===1failed,1skippedin0.05seconds===4. Testing floats
Ever hit this when testing floats?
Eassert0.30000000000000004==0.3E+where0.30000000000000004=sum_numbers(0.1,0.2)Yikes!
No worries though, pytest's approx has your back, this passes:
assertsum_numbers(0.1,0.2)==approx(0.3)5. Working with temporary files
Creating and cleaning up temporary files can be a lot of work, but pytest makes this quite effortlessly.
In this example taken from Bite 161 we create 5 files in a temporary directory and assert that count_dirs_and_files returns a tuple of counts (0 directories and 5 files):
deftest_only_files(tmp_path):
foriinrange(1, 6):
path=tmp_path/f'{i}.txt'withopen(path, 'w')asf:
f.write('hello')assertcount_dirs_and_files(tmp_path)==(0, 5)The files were created in a temporary directory and I did not have to clean anything up manually.
6. Testing exceptions
Here is an example from Intro Bite #10 that uses pytest.raises(...) to test an exception:
@pytest.mark.parametrize("numerator, denominator",[ (2, 's'), ('s', 2), ('v', 'w'),])deftest_divide_numbers_raises_value_error(numerator,denominator):withpytest.raises(ValueError):divide_numbers(numerator,denominator)7. Enhance your parametrized tests
For this tip I changed divide_numbers to have test_divide_numbers_raises_value_error fail:
FAILEDtest_division.py::test_divide_numbers_raises_value_error[2-s] -TypeError: unsupportedoperandtype(s)for/: 'int'and'str'This is ok, but we can make the [2-s] part a bit more readable.
We can wrap the parametrize list arguments inside pytest.param giving it test IDs (see here):
@pytest.mark.parametrize("numerator, denominator",[ pytest.param(2, 's', id="denominator_wrong_type"), pytest.param('s', 2, id="numerator_wrong_type"), pytest.param('v', 'w', id="both_numerator_denominator_wrong_type"),])deftest_divide_numbers_raises_value_error(numerator,denominator):withpytest.raises(ValueError):divide_numbers(numerator,denominator)Now this string will show up in the failing test:
FAILEDtest_division.py::test_divide_numbers_raises_value_error[denominator_wrong_type] -TypeError: unsupportedoperandtype(s)for/: 'int'and'str'And we can target these strings with pytest -k as well, for example pytest -k both_numerator runs only the third test of test_divide_numbers_raises_value_error, pytest -k numerator would run two tests.
8. Drop into the debugger upon failure
This is one the most useful tips in my opinion: when something breaks you want to be able to debug right then and there.
So in the previous failing example if we run the tests with pytest --pdb it drops into the debugger:
> /Users/bobbelderbos/code/bitesofpy/110/division.py(9)divide_numbers()-> return int(numerator)/denominator(Pdb)For more variations check out the docs.
And in order to debug a hanging test, check out our related article.
9. Test logging
You can test logging with pytest's caplog fixture:
# script.pyimportloggingdeffunc():logging.debug("a debug message to ignore")logging.info("an info message")try:1/0exceptZeroDivisionError:logging.exception("cannot divide by 0")# test_script.pyimportloggingfromscriptimportfuncdeftest_func(caplog):caplog.set_level(logging.INFO)func()record1,record2=caplog.recordsassertrecord1.levelname=="INFO"# no debugassertrecord1.message=="an info message"assertrecord2.message=="cannot divide by 0"assertrecord2.exc_info[0]isZeroDivisionErrorHere we made a function called func that logs 3 messages: DEBUG, INFO and ERROR (by the way, logging.exception is really useful, it adds exception info the logging message!)
In the test we use the caplog fixture to grab those logging messages and test them.
10. Test standard output
How to test a function that prints to standard output (as opposed to returning something)?
You can use the capsys / capfd fixtures for this.
Here is an example from Intro Bite #01.
Code (spoiler alert!):
MIN_DRIVING_AGE=18defallowed_driving(name, age):
"""Print '{name} is allowed to drive' or '{name} is not allowed to drive'checkingthepassedinageagainsttheMIN_DRIVING_AGEconstant"""is_allowed='is allowed'ifage>=MIN_DRIVING_AGEelse'is not allowed'print(f'{name} {is_allowed} to drive')Tests:
fromdrivingimportallowed_drivingdeftest_not_allowed_to_drive(capfd):allowed_driving('tim',17)output=capfd.readouterr()[0].strip()assertoutput=='tim is not allowed to drive'...I hope you learned something new and that you can use any of this when you are writing pytest code.
If you want to share other cool pytest tips, please comment below ...
Keep Calm and Write more Tests!
-- Bob