Fixtures are one of the innovations I’m most happy with.
A Fixture is an enhanced context manager. The enhancements are:
- There’s an API for gathering debugging information from the fixture (rather than depending on side effects such as the logging module or stdout). This makes it easy to attach log files from servers (for instance rabbitfixture does this).
- There is glue to support composing other fixtures while still exposing errors from any fixture in the composed set.
OpenStack’s Neutron has been using fixtures in its test suite for some time, but is finding that writing correct fixtures is hard. In particular, they were leaking processes when a fixture would fail during setUp / __enter__ – and then not be cleaned up by the testtools / fixtures useFixture function.
There are several things we can do to improve the situation.
- We could make the convenience APIs like useFixture add a try:/finally: and call cleanUp() when setUp fails. This involves making cleanUp() be callable in more situations than it is today.
- We could make setUp itself do that, advising users to override a different function; this would hide the failure interactions internally, but wouldn’t benefit existing fixtures until they are rewritten to not override setUp.
- We could provide a decorator that folk with fragile setUp’s (e.g. those that involve IO) could use to robustify their fixtures.
The highest leverage change is the first, but is it safe and suitable? Lets look at PEP-343.
In PEP-343 we see the following translation of with expressions:
with EXPR as VAR:
BLOCK
....
mgr = (EXPR)
exit = type(mgr).__exit__
value = type(mgr).__enter__(mgr)
exc = True
try:
try:
VAR = value
BLOCK
except:
exc = False
if not exit(mgr, *sys.exc_info()):
raise
finally:
if exc:
exit(mgr, None, None, None)
This means that using a Fixture which may leak external resources when setUp fails is unsafe via with. Therefore we can’t use the first solution.
Decorators are nice, but somewhat noisy and opt-in. Both decorators and a different setUp in the base class will require extending the protocol to specify when cleanUp can be called more precisely.
If we make the documentation advise users to override a specific method, and setUp does this in the event of failure, I think we’ll have somewhat more uptake. So – thats the route I’m going to head down.
There’s one more thing to consider, which is access to debugging information of failures in setUp. Since the object will have been cleaned up, accessing logs etc will be hard. I think if we raise an additional exception into the MultiException with the details objects, it will be possible for fixtures to provide those details, though they will need buffering in memory (or some sophisticated lazy-delete logic such as holding a reference to an unlinked fd).
