Here is a quick summary of the steps required to prepare a build of Python 2.7 for use with valgrind memcheck tool.
Valgrind is an indispensable tool for discovering memory safety issues, and for quickly tracking down the source of many kinds of memory corruption, especially in C code the user may not be familiar with. If your Python program is crashing with a segmentation fault, executing it using an instrumented build running under valgrind can very often pinpoint the exact cause of the fault, if the fault is due to a memory safety violation.
I needed this setup as today I decided to finalize an old incomplete patch to py-lmdb, to allow passing fixed-size integer keys and values to the database, useful for reducing storage and performance overhead. Today valgrind is useful since with the patch enabled, LMDB appears to disable some of its sanity checks around the size of key/value pointers passed to it, and so in order to maintain memory safety in the binding, I’d like to discover any places where I’ve forgotten to add a corresponding size check.
Grab dependencies
Grab a copy of the Python source tree, and check out the 2.7 branch:
$ sudo apt-get install build-essential valgrind $ pip install Mercurial $ cd /home/dmw/src $ hg clone https://hg.python.org/cpython/ $ cd cpython $ hg checkout 2.7
Build Python
Build a copy of Python with pymalloc disabled. This is required
since when pymalloc is enabled, Python asks for large chunks of memory at a
time from the C library, and so valgrind is unable to discover allocation
boundaries using its magical instrumented malloc and
free implementations that it injects into the target process.
Additionally build with Py_DEBUG to cause reference counts to be
more thoroughly checked. Do not --enable-shared, as this produces
a Python binary that will link to your system-installed
libpython.so unless you remember to set
LD_LIBRARY_PATH every time you run the debug build.
There is also a --with-valgrind option that prepares a build where
pymalloc is disabled at runtime when running under valgrind, but means the
Python binary will change behaviour depending on whether it’s running directly,
under valgrind, or under gdb, a fabulous source of seeming heisenbugs when the
user (me) is half asleep, so I prefer to avoid it.
Finally, set --prefix to a subdirectory so that when we come to
install pip and setuptools, these tools are
not installed into /usr/local or any other global filesystem path.
$ ./configure --without-pymalloc --with-pydebug --prefix=`pwd`/build $ make -j 16
Test The Build
We do not need to install the built tree in order to make use of it, simply
invoke ./python to run it in-place:
$ ./python Python 2.7.12+ (2.7:11a9bca71528, Oct 17 2016, 16:21:52) [GCC 4.9.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> [59613 refs] [27063 refs]
The presence of [59613 refs] indicates the binary you are running
was built with --with-pydebug.
Try It Under Valgrind
Now let’s try running the binary by itself under valgrind, to ensure our configuration is sane, and that valgrind is not reporting errors because we screwed up some step.
$ time valgrind ./python -S -c 'print 123' ==12914== Memcheck, a memory error detector ==12914== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al. ==12914== Using Valgrind-3.10.0 and LibVEX; rerun with -h for copyright info ==12914== Command: ./python -S -c print\ 123 ==12914== 123 [10871 refs] ==12914== ==12914== HEAP SUMMARY: ==12914== in use at exit: 470,972 bytes in 2,963 blocks ==12914== total heap usage: 9,362 allocs, 6,399 frees, 1,626,675 bytes allocated ==12914== ==12914== LEAK SUMMARY: ==12914== definitely lost: 0 bytes in 0 blocks ==12914== indirectly lost: 0 bytes in 0 blocks ==12914== possibly lost: 77,979 bytes in 310 blocks ==12914== still reachable: 392,993 bytes in 2,653 blocks ==12914== suppressed: 0 bytes in 0 blocks ==12914== Rerun with --leak-check=full to see details of leaked memory ==12914== ==12914== For counts of detected and suppressed errors, rerun with: -v ==12914== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0) real 0m2.201s user 0m2.168s sys 0m0.032s
We see already valgrind is reporting memory has been leaked. This is normal, as there are a variety of small structures allocated by CPython at startup that it never frees, usually because doing so would be too difficult.
You may also notice this Python build is much slower! Not only is valgrind
slowing performance quite a lot, but the absense of pymalloc
vastly hurts performance of the built interpreter. Why is this? Well at least,
pymalloc knows more about the allocation styles that Python needs in comparison
to the C library allocator, and so can be specialized for it, but perhaps more
importantly, unlike the C allocator, pymalloc knows it will always run under
the Global Interpreter Lock, so it does not require any additional mutexes for
its data structures.
Running with --leak-check=full should indicate most of these
“leaks” have PyType_Ready somewhere in the call stack, indicating
they are heap allocations to support the static Py_Type structures
that implement a variety of built-in functionality (such as the
str object).
Install pip and setuptools
$ ./python -m ensurepip
We can now install any Python packages we need to support our test environment.
If your application is complex, it might be desirable to also ./python -m
pip install virtualenv along with make install to get a
working virtualenv. I do not need it now so we will skip this step.
Build your application with the debug Python build
Since we built a completely new Python interpreter with debugging enabled, its
header definitions will no longer match the headers that were in use by any C
extensions you may have previously built. This is important, since the critical
macros Py_INCREF and Py_DECREF change their
implementation significantly in debug mode. For that reason, you must rebuild
any third party modules your application is using with the new Python build.
In my case, I am not debugging an application, just an extension module, so I
will only rebuild it, and add a little symlink so I do not need to install it
to --prefix every time I rebuild it.
$ cd /home/dmw/src/py-lmdb $ git clean -dfx $ /home/dmw/src/cpython/python setup.py build running build running build_py ... [91959 refs] $ cd lmdb $ ln -s ../build/lib.linux-x86_64-2.7-pydebug/lmdb/cpython.so
Trigger behaviour leading to the scenario you wish to debug
Self-explanatory. In my case, running py-lmdb’s tests is sufficient to reach the code paths I suspect might be damaged, so I’ll install py.test and run the tests under vaglrind.
To demonstrate what a valgrind error looks like, I’ve injected some obviously
buggy code into cpython.c:
{
printf("hello\n");
char *x = malloc(3);
setenv("X", x, 0);
}
This allocates some uninitialized memory, then attempts to set the
X environment variable to the content of that memory. It is a bug
since the memory could contain any random junk, and accessing uninitialized
memory in C is “undefined behaviour”, i.e. the specification allows for your
program to crash, your computer to catch fire, or the moon to come crashing
into the earth.
Now let’s see how the error looks under valgrind:
$ /home/dmw/src/cpython/python -m pip install pytest $ PYTHONPATH=. valgrind /home/dmw/src/cpython/python -m pytest tests/txn_test.py ==29422== Memcheck, a memory error detector ==29422== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al. ==29422== Using Valgrind-3.10.0 and LibVEX; rerun with -h for copyright info ==29422== Command: /home/dmw/src/cpython/python -m pytest tests/txn_test.py ==29422== =============================================================================== test session starts ================================================================================ platform linux2 -- Python 2.7.12+, pytest-3.0.3, py-1.4.31, pluggy-0.4.0 rootdir: /home/dmw/src/py-lmdb, inifile: collected 46 items tests/txn_test.py ==29422== Conditional jump or move depends on uninitialised value(s) ==29422== at 0x4C30759: setenv (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==29422== by 0xD1A152F: trans_put (cpython.c:3271) ==29422== by 0x562798: PyCFunction_Call (methodobject.c:85) ==29422== by 0x4D6457: call_function (ceval.c:4350) ==29422== by 0x4D1281: PyEval_EvalFrameEx (ceval.c:2987) ==29422== by 0x4D3C57: PyEval_EvalCodeEx (ceval.c:3582) ==29422== by 0x4D6A31: fast_function (ceval.c:4445) ==29422== by 0x4D6632: call_function (ceval.c:4370) ==29422== by 0x4D1281: PyEval_EvalFrameEx (ceval.c:2987) ==29422== by 0x4D3C57: PyEval_EvalCodeEx (ceval.c:3582) ==29422== by 0x55FE44: function_call (funcobject.c:523) ==29422== by 0x420F28: PyObject_Call (abstract.c:2547) ==29422==
From the stack trace, we can see cpython.c:3271 is mentioned,
corresponding to the source line where our erroneous setenv() call
appears. It is very obvious from this output that uninitialized data was
accessed, and who accessed it.
valgrind –db-attach
There is one final crucial flag to know about for valgrind, and that is
--db-attach. This causes valgrind to immediately halt the Python
interpreter when an error is detected, and drop you into a gdb
debugger shell, where you can inspect the program using a regular debugger with
the exact state of memory preserved at the point where the error occurred.
Don’t forget to apt-get install gdb before trying it out.
Happy debugging!
Valgrind catches a huge variety of memory violations, this simple example only demonstrates one. A full tutorial on these errors would be hard to write, as I doubt I’ve even seen them all. As always, better understanding comes through practice, so next you catch a Python segmentation fault, perhaps try this route to see if you can diagnose or fix it for yourself.