Machinalis: Controlling uncertainty on web applications and APIs

“But it works fine on my machine!”

How many times have your heard that phrase? I’ve been working as a developer for over a decade and I can swear I heard it in every project I worked on. No matter how well you design your software and plan its deployments, almost always something unexpected happens.

Over the years, the industry has developed techniques and technologies to minimize the difference between the development and production environments. Right now we have scripted deployments, several container and virtualization solutions, and continuous integration, all working together to make sure that every step of the process is predictable. But even with all this in our toolbox, things still go wrong from time to time.

Here at Machinalis we provide services and work on projects with small startups and huge corporations (and everything in between), and although the resources and corporate structure might change, the goal remains the same: producing efficient high-quality products that fulfills its throughout its lifespan. Hence, the policies and techniques for deployment are pretty similar regardless of the size of the client, and it usually works just as we planned it...for a time.

Whether a multi-million dollar storage solution that’s about to give up the ghost or a clueless operator that uses a mission critical network to transfer several terabytes of media right in the middle of the launch of a product (both true stories, by the way); there are things that are beyond our control.

How can we make sure that our applications and APIs are robust enough to handle these situations? What can we do to test theories regarding these transient difficulties in infrastructure without access to the real environments?

The first answer is to design robust software. It’s not particularly hard and several ways of doing it have been thoroughly researched and documented. The next thing is setup logging properly. It’s the easiest thing to do but it is also frequently overlooked.

The next thing is test for less than ideal situations. Enter chaos.

When we need to introduce simulated network problems into a project, we rely on chaotic proxies. These proxies sit in between components that need to communicate through a network, and they cause predictable problems like delays, disconnections and, in the case of TCP proxies, packet dropouts and other low-level network problems.

So far we’ve used mostly Vaurien due to its flexibility, ease of use, and the fact that is written in Python (which is alwasy a plus). Others we can mention are comcast, toxiproxy, and clumsy.

Let’s illustrate the process with a simple example. We have a very simple RESTful web API that serves a one-page web application. We want to make sure the application can handle transient problems with the API, so we need to set-up the proxy between the app and the API.

First we’ll hit the API using ApacheBench to get an idea of how it works under normal circumstances:

ServerSoftware:nginx/1.4.6ServerHostname:localhostServerPort:8000DocumentPath:/api/items/DocumentLength:2817bytesConcurrencyLevel:1Timetakenfortests:3.909secondsCompleterequests:500Failedrequests:0Totaltransferred:1519000bytesHTMLtransferred:1408500bytesRequestspersecond:127.91[#/sec] (mean)Timeperrequest:7.818[ms](mean)Timeperrequest:7.818[ms](mean,acrossallconcurrentrequests)Transferrate:379.49[Kbytes/sec]receivedConnectionTimes(ms)minmean[+/-sd]medianmaxConnect:000.000Processing:780.5811Waiting:780.5711Total:780.5811ERROR:Themedianandmeanforthewaitingtimearemorethantwicethestandarddeviationapart.TheseresultsareNOTreliable.Percentageoftherequestsservedwithinacertaintime(ms)50%866%875%880%890%895%998%999%10100%11(longestrequest)

Now we want 20% of the calls to fail, so we set-up vaurien:

$ vaurien --protocol http --proxy 0.0.0.0:8005 --backend localhost:8000 --behavior 10:error
2016-08-08 11:54:58 [18580] [INFO] Starting the Chaos TCP Server
2016-08-08 11:54:58 [18580] [INFO] Options:
2016-08-08 11:54:58 [18580] [INFO] * proxies from 0.0.0.0:8005 to localhost:8000
2016-08-08 11:54:58 [18580] [INFO] * timeout: 30
2016-08-08 11:54:58 [18580] [INFO] * stay_connected: 0
2016-08-08 11:54:58 [18580] [INFO] * pool_max_size: 100
2016-08-08 11:54:58 [18580] [INFO] * pool_timeout: 30
2016-08-08 11:54:58 [18580] [INFO] * async_mode: 1

And run ab against the proxy port:

ServerSoftware:ServerHostname:localhostServerPort:8005DocumentPath:/api/items/DocumentLength:188bytesConcurrencyLevel:1Timetakenfortests:3.854secondsCompleterequests:500Failedrequests:471(Connect:0,Receive:0,Length:471,Exceptions:0)Non-2xxresponses:102Totaltransferred:1241946bytesHTMLtransferred:1144942bytesRequestspersecond:129.74[#/sec] (mean)Timeperrequest:7.707[ms](mean)Timeperrequest:7.707[ms](mean,acrossallconcurrentrequests)Transferrate:314.72[Kbytes/sec]receivedConnectionTimes(ms)minmean[+/-sd]medianmaxConnect:000.000Processing:183.8936Waiting:183.8936Total:183.8936Percentageoftherequestsservedwithinacertaintime(ms)50%966%975%980%1090%1095%1198%1299%13100%36(longestrequest)

So we got 106 out 500 abnormal responses (close to our 20% target).

Vuarien supports other protocols as well, which makes it ideal for the task of introducing uncertainty into any modern project that integrates multiple back-ends, but it is not without its problems; it only supports Python 2, so you would have to use separate virtualenvs for Python 3 project, it lacks flexibility for some use cases, and it is another component that you have to integrate into your testing architecture which adds complexity.

What if all you wanted was a simple way to introduce uncertainty into an API to test a single-page app? If you’re using Django, we’ve got a solution for you.

Last week Django 1.10 was released, and with it, a new middleware system. While reviewing the release notes we realized that this was a good opportunity to fill a gap in our toolbox and learn to use the new features in the process; so we decided to create django_uncertainty.

django_uncertainty is a Django 1.10 middleware that allows developers to introduced the kinds of problems we described above to allow testing less than favorable conditions in the local development environment.

What makes it different than Vaurien or any other proxy?

• It is dead simple and can be easily extended.
• It has no external dependencies.
• You can define behaviors based on knowledge of the internal structure of the application.
• It only works with Django 1.10 or later.
• It only supports HTTP.
• It can only be placed in front of the target application.

As you can see, it has its limitations, but we think it can be useful under certain conditions. Let’s see it in action.

Same as before, we want 20% of the requests to fail, but now we want 10% of those to be Internal Server Errors (500), 5% to be Forbidden (403) and 5% to be Not Found (404). Assuming that the middleware has been properly set up (it’s explained in the package documentation), you can declare this particular behavior with the DJANGO_UNCERTAINTY settings variable:

importuncertaintyasuDJANGO_UNCERTAINTY=u.random_choice([(u.server_error(),0.1),(u.forbidden(),0.05),(u.not_found(),0.05)])

Let us fire up ab once again to see how the middleware performs:

ServerSoftware:nginx/1.4.6ServerHostname:localhostServerPort:8000DocumentPath:/api/items/DocumentLength:2817bytesConcurrencyLevel:1Timetakenfortests:3.401secondsCompleterequests:500Failedrequests:97(Connect:0,Receive:0,Length:97,Exceptions:0)Non-2xxresponses:97Totaltransferred:1242483bytesHTMLtransferred:1135251bytesRequestspersecond:147.01[#/sec] (mean)Timeperrequest:6.802[ms](mean)Timeperrequest:6.802[ms](mean,acrossallconcurrentrequests)Transferrate:356.76[Kbytes/sec]receivedConnectionTimes(ms)minmean[+/-sd]medianmaxConnect:000.000Processing:172.7833Waiting:172.7733Total:172.7833Percentageoftherequestsservedwithinacertaintime(ms)50%866%875%880%890%895%998%1099%10100%33(longestrequest)

The count for each specific error code is not shown, but we can extract the values from the log file. Here’s the final tally.

The numbers don’t match our specification exactly due to the usage of a random number to chose which behavior to activate.

Let us change the specification a bit. Now we want to delay requests by half a second, but only those which use PUT or PATCH requests and go through under the /api/ path and we want the requests to go through as normal

importuncertaintyasuDJANGO_UNCERTAINTY=u.cond(u.path_is('^/api')&(u.is_post|u.is_method('PATCH')),u.delay(u.default(),0.5))

You can read the documentation to see what other behaviors and conditions can be used to control the outcome of the requests. We tried our best to provide the tools to describe the most common scenarios, but if you think there are others that we can cover, feel free to create a ticket in the GitHub repository with a description of the situation you want to specify.

If you want to try a live demo of the middleware, we’ve created a sample that exposes a RESTful API using Django Rest Framework. You can get the code on GitHub and there’s also a Vagrantfile with everything already set-up for you.

Drop us a comment with your take on this matter. We’re interested to learn about other situations were the unexpected happened and how was the problem deal with. Feedback on the blogpost and the code is always welcomed as well.