I have written before about Flask and obtaining test coverage results here and with an update here. This is pretty trivial if you're writing unit tests that directly call the application, but if you actually want to write tests which animate a browser, for example with selenium, then it's a little more complicated, because the browser/test code has to run concurrently with the server code.
Previously I would have the Flask server run in a separate process and run 'coverage' over that process. This was slightly unsatisfying, partly because you sometimes want coverage analysis of your actual tests. Test suites, just like application code, can grow in size with many utility functions and imports etc. which may eventually end up not actually being used. So it is good to know that you're not needlessly maintaining some test code which is not actually invoked.
We could probably get around this restriction by running coverage in both the server process and the test-runner's process and combine the results (or simply view them separately). However, this was unsatisfying simply because it felt like something that should not be necessary. Today I spent a bit of time setting up the scheme to test a Flask application without the need for a separate process.
I solved this now, by not using Flask's included Werkzeug server and instead using the WSGI server included in the standard-library wsgiref.simple_server module. Here is, a minimal example:
importflaskclassConfiguration(object):TEST_SERVER_PORT=5001application=flask.Flask(__name__)application.config.from_object(Configuration)@application.route("/")deffrontpage():ifFalse:pass# Should not be coveredelse:return'I am the lizard queen!'# Should be in coverage.# Now for some testing.fromseleniumimportwebdriverfromselenium.webdriver.common.action_chainsimportActionChainsimportpytest# Currently just used for the temporary hack to quit the phantomjs process# see below in quit_driver.importsignalimportthreadingimportwsgiref.simple_serverclassServerThread(threading.Thread):defsetup(self):application.config['TESTING']=Trueself.port=application.config['TEST_SERVER_PORT']defrun(self):self.httpd=wsgiref.simple_server.make_server('localhost',self.port,application)self.httpd.serve_forever()defstop(self):self.httpd.shutdown()classBrowserClient(object):"""Interacts with a running instance of the application via animating a browser."""def__init__(self,browser="phantom"):driver_class={'phantom':webdriver.PhantomJS,'chrome':webdriver.Chrome,'firefox':webdriver.Firefox}.get(browser)self.driver=driver_class()self.driver.set_window_size(1200,760)deffinalise(self):self.driver.close()# A bit of hack this but currently there is some bug I believe in# the phantomjs code rather than selenium, but in any case it means that# the phantomjs process is not being killed so we do so explicitly here# for the time being. Obviously we can remove this when that bug is# fixed. See: https://github.com/SeleniumHQ/selenium/issues/767self.driver.service.process.send_signal(signal.SIGTERM)self.driver.quit()deflog_current_page(self,message=None,output_basename=None):content=self.driver.page_source# This is frequently what we really care about so I also output it# here as well to make it convenient to inspect (with highlighting).basename=output_basenameor'log-current-page'file_name=basename+'.html'withopen(file_name,'w')asoutfile:ifmessage:outfile.write("<!-- {} --> ".format(message))outfile.write(content)filename=basename+'.png'self.driver.save_screenshot(filename)defmake_url(endpoint,**kwargs):withapplication.app_context():returnflask.url_for(endpoint,**kwargs)# TODO: Ultimately we'll need a fixture so that we can have multiple# test functions that all use the same server thread and possibly the same# browser client.deftest_server():server_thread=ServerThread()server_thread.setup()server_thread.start()client=BrowserClient()driver=client.drivertry:port=application.config['TEST_SERVER_PORT']application.config['SERVER_NAME']='localhost:{}'.format(port)driver.get(make_url('frontpage'))assert'I am the lizard queen!'indriver.page_sourcefinally:client.finalise()server_thread.stop()server_thread.join()
To run this you will of course need flask as well as pytest, pytest-cov, and selenium:
$ pip install flask pytest pytest-cov
In addition you will need the phantomjs to run:
$ npm install phantomjs $ exportPATH=$PATH:./node_modules/.bin/
Then to run it, the command is:
$ py.test --cov=./ app.py
$ coverage html
The coverage html is of course optional and only if you wish to view the results in friendly HTML format.
Notes
I've not used this extensively myself yet, so there may be some problems when using a more interesting flask application.
Don't put your virtual environment directory in the same directory as app.py because in that case it will perform coverage analysis over the standard library and dependencies.
In a real application you will probably want to make a pytest fixture out of the server thread and browser client. So that you can use each for multiple separate test functions. Essentially your test function should just be the part inside the try clause.
I have used the log_current_page method but I frequently find it quite useful so included it here nonetheless.