Every conversation I hear about testing seems to go thus:
"Here's a useful way to test!"
"But if you only test that way, you won't catch all the bugs."
I would like us, the open source community, to move on from this conversation. Each testing method is useful for some kinds of software, and each accomplishes some goals. But none is plenipotent. There is no silver bullet.
So the next time someone proposes unit testing, please don't object: "Unit tests can't catch integration bugs." Nor, when you hear about integration tests, should you object: "Integration tests don't reveal bugs as early as unit tests." Instead, consider whether each method is being applied where appropriate, or if some other method would be better in this particular circumstance.
It's the same with this series of articles I've written about "black pipe" testing. I make no claim that my testing method is perfect, certain, and complete. Please don't replace your other tests with black pipe tests wholesale. I have merely identified a color in the spectrum that ranges between unit tests and integration tests, and named it. And I have distinguished two scenarios where I think black pipe testing is a superior way to test connected applications.
The first scenario is when your code's observable behavior is correct, even if it misbehaves at the network layer. In this case, a black pipe test can watch the wire and check that your code sends the messages you expect. I describe a case like this in Testing PyMongo As A Black Pipe: PyMongo would appear correct even if it sent outdated wire protocol messages to MongoDB. I need a black pipe test to validate that PyMongo uses the latest MongoDB wire protocol.
Second, black pipe tests can reliably test rare events, like network timeouts, hangups, and resets. These events are hard to trigger during integration testing. If you want to continuously test your code's reaction to a timeout, for example, it may not be practical to stop a network interface at exactly the moment when your test is exercising the relevant code. But a black pipe test impersonates the server and reacts predictably to each message your code sends.
Black pipe tests are a useful complement to your existing methods. If you have bad tests or untested code that falls into one of these two scenarios, I hope my articles offer you the tool you did not know you lacked.
I hope you apply this method to servers besides MongoDB. Take inspiration from MockupDB's API in Python, or the mock_server_t functions in C, but impersonate some other server: a mock PostgreSQL server, for example, could be used to test thousands of different client libraries and applications. A mock LDAP or SMTP server would be similarly useful. And if you write a RESTful HTTP server with testing methods as expressive and convenient as MockupDB's, the number of Python client libraries you could help test would be practically numberless.
Further reading:
- My six-part black pipe testing series
- Dan Crosta's "Good Test, Bad Test"
- Nick Coghlan's "Client and Server in Python 3.5"
