Testing
Chapter 15. Testing
There are two very good reasons for writing unit tests. When implementing new functionality, unit tests are used to confirm that the new code is working in the expected way. The same result can be obtained by testing manually, but of course automated tests save time and effort because they can be repeated easily.
A second, more important reason is that each time the application is modified, all the unit tests built around it can be executed to ensure that there are no regressions in the existing code; in other words, that the new changes did not affect the way the older code works.
Unit tests have been a part of Flasky since the very beginning, with tests designed to exercise specific features of the application implemented in the database model classes. These classes are easy to test outside of the context of a running application, so given that it takes little effort, implementing unit tests for all the features that exist in the database models is the best way to ensure at least that part of the application starts robust and stays that way.
This chapter discusses ways to improve and extend unit testing to other areas of the application.
Obtaining Code Coverage Reports
Having a test suite is important, but it is equally important to know how good or bad it is. Code coverage tools measure how much of the application is exercised by unit tests and can provide a detailed report that indicates which parts of the application code are not being tested. This information is invaluable, because it can be used to direct the effort of writing new tests to the areas that need it most.
Python has an excellent code coverage tool appropriately called coverage. You can install it with pip:
This tool comes as a command-line script that can launch any Python application with code coverage enabled, but it also provides more convenient scripting access to start the coverage engine programmatically. To have coverage metrics nicely integrated into the flask test command added in Chapter 7, a --coverage option can be added. The implementation of this option is shown in Example 15-1.
Example 15-1. flasky.py: coverage metrics
The code coverage support is enabled by passing the --coverage option to the flask test command. To add the Boolean option to the test custom command, the click.option decorator is used. Click then passes the value of the Boolean flag as an argument to the function.
But integrating code coverage in the flasky.py script presents a small problem. By the time the --coverage option is received in the test() function, it is already too late to enable coverage metrics; by that time all the code in the global scope has already executed. So, to get accurate metrics, the script recursively restarts itself after setting the FLASK_COVERAGE environment variable. In the second run, the top of the script finds that the environment variable is set and turns on coverage from the start, even before all the application imports.
The coverage.coverage() function starts the coverage engine. The branch=True option enables branch coverage analysis, which, in addition to tracking which lines of code execute, checks whether for every conditional both the True and False cases have executed. The include option is used to limit coverage analysis to the files that are inside the application package, which is the only code that needs to be measured. Without the include option, all the extensions installed in the virtual environment and the code for the tests itself would be included in the coverage reports—and that would add a lot of noise to the report.
After all the tests have executed, the test() function writes a report to the console and also writes a nicer HTML report to disk. The HTML version shows all the source code annotated with colors that indicate the lines that are covered by the tests and the ones that are not.
Tip
If you have cloned the application’s Git repository on GitHub, you can run git checkout 15a to check out this version of the application. To ensure that you have all the dependencies installed, also run pip install -r requirements/dev.txt.
An example of the text-based report follows:
(venv) $ flask test --coverage
...
.----------------------------------------------------------------------
Ran 23 tests in 6.337s
OK
Coverage Summary:
Name Stmts Miss Branch BrPart Cover
----------------------------------------------------------------
app/__init__.py 32 0 0 0 100%
app/api_v1/__init__.py 3 0 0 0 100%
app/api_v1/authentication.py 29 18 10 0 28%
app/api_v1/comments.py 40 30 12 0 19%
app/api_v1/decorators.py 11 3 2 0 62%
app/api_v1/errors.py 17 10 0 0 41%
app/api_v1/posts.py 36 24 8 0 27%
app/api_v1/users.py 30 24 12 0 14%
app/auth/__init__.py 3 0 0 0 100%
app/auth/forms.py 45 8 8 0 70%
app/auth/views.py 116 91 42 0 16%
app/decorators.py 14 3 2 0 69%
app/email.py 15 9 0 0 40%
app/exceptions.py 2 0 0 0 100%
app/main/__init__.py 6 1 0 0 83%
app/main/errors.py 20 15 6 0 19%
app/main/forms.py 39 7 6 0 71%
app/main/views.py 178 140 34 0 18%
app/models.py 236 42 42 6 79%
----------------------------------------------------------------
TOTAL 872 425 184 6 45%
HTML version: file:///home/flask/flasky/tmp/coverage/index.html
The report shows an overall coverage of 45%, which is not terrible, but isn’t very good either. The model classes, which have received all the unit testing attention so far, constitute a total of 236 statements, of which 79% are covered in tests. Obviously the views.py files in the main and auth blueprints and the routes in the api_v1 blueprint all have very low coverage, since these are not exercised in any of the existing unit tests. And of course, these coverage metrics are not indicative of how much bug-free code exists in the project, since other factors (such as the quality of the tests) play a big role in that.
Armed with this report, it is easy to determine where tests need to be added to the test suite to improve coverage—but unfortunately, not all parts of the application can be tested as easily as the database models. The next two sections discuss more advanced testing strategies that can be applied to view functions, forms, and templates.
The Flask Test Client
Some portions of the application code rely heavily on the environment that is created by a running application. For example, you can’t simply invoke the code in a view function to test it, since the function may need to access Flask context variables such as request or session, it may be expecting form data provided in a POST request, and it may also require a logged-in user. In short, view functions can run only within the context of a request and a running application.
Flask comes equipped with a test client to try to address this problem, at least to some extent. The test client replicates the environment that exists when an application is running inside a web server, allowing tests to act as clients and send requests.
The view functions do not see any major differences when executed under the test client; requests are received and routed to the appropriate view functions, from which responses are generated and returned. After a view function executes, its response is passed to the test, which can check it for correctness.
Testing Web Applications
Example 15-2 shows a unit testing framework that uses the test client.
Compared to tests/test_basics.py, this module adds a self.client instance variable, which is the Flask test client object. This object exposes methods that issue requests into the application. When the test client is created with the use_cookies option enabled, it will accept and send cookies in the same way browsers do, so functionality that relies on cookies to recall context between requests can be used. In particular, this approach enables the use of user sessions, which are stored in cookies.
The test_home_page() test is a simple example of what the test client can do. In this example, a request for the root URL of the application is issued. The return value of the get() method of the test client is a Flask response object containing the response returned by the invoked view function. To check whether the test was successful, the status code of the response is checked, and then the body of the response, obtained from response.get_data(), is searched for the word "Stranger", which is part of the “Hello, Stranger!” greeting shown to anonymous users. Note that get_data() returns the response body as a byte array by default; passing as_text=True converts it to a string, which is easier to work with.
The test client can also send POST requests that include form data using the post() method, but submitting forms presents a small complication. As discussed in Chapter 4, all forms generated by Flask-WTF have a hidden field with a CSRF token that needs to be submitted along with the form. To be able to send the CSRF token, a test would need to request the page that displays the form, then parse the HTML returned in that response and extract the token, so that it can then send it with the form data. To avoid the hassle of dealing with CSRF tokens in tests, it is better to disable CSRF protection in the testing configuration. This is shown in Example 15-3.
Example 15-3. config.py: disabling CSRF protection in the testing configuration
Example 15-4 shows a more advanced unit test that simulates a new user registering an account, logging in, confirming the account with a confirmation token, and finally logging out.
Example 15-4. tests/test_client.py: simulation of a new user workflow with the Flask test client
class FlaskClientTestCase(unittest.TestCase):
# ...
def test_register_and_login(self):
# register a new account
response = self.client.post('/auth/register', data={
'email': 'john@example.com',
'username': 'john',
'password': 'cat',
'password2': 'cat'
})
self.assertEqual(response.status_code, 302)
# log in with the new account
response = self.client.post('/auth/login', data={
'email': 'john@example.com',
'password': 'cat'
}, follow_redirects=True)
self.assertEqual(response.status_code, 200)
self.assertTrue(re.search('Hello,\s+john!',
response.get_data(as_text=True)))
self.assertTrue(
'You have not confirmed your account yet' in response.get_data(
as_text=True))
# send a confirmation token
user = User.query.filter_by(email='john@example.com').first()
token = user.generate_confirmation_token()
response = self.client.get('/auth/confirm/{}'.format(token),
follow_redirects=True)
user.confirm(token)
self.assertEqual(response.status_code, 200)
self.assertTrue(
'You have confirmed your account' in response.get_data(
as_text=True))
# log out
response = self.client.get('/auth/logout', follow_redirects=True)
self.assertEqual(response.status_code, 200)
self.assertTrue('You have been logged out' in response.get_data(
as_text=True))
The test begins with a form submission to the registration route. The data argument to post() is a dictionary with the form fields, which must exactly match the field names defined in the HTML form. Since CSRF protection is now disabled in the testing configuration, there is no need to send the CSRF token with the form.
The /auth/register route can respond in two ways. If the registration data is valid, a redirect sends the user to the login page. In the case of an invalid registration, the response renders the page with the registration form again, including any appropriate error messages. To validate that the registration was accepted, the test checks that the status code of the response is 302, which is the code for a redirect.
The second section of the test issues a login request to the application using the email and password just registered. This is done with a POST request to the /auth/login route. This time a follow_redirects=True argument is included in the post() call to make the test client work like a browser and automatically issue a GET request for the redirected URL. With this option, status code 302 will not be returned; instead, the response from the redirected URL is returned.
A successful response to the login submission would now have a page that greets the user by their username and then indicates that the account needs to be confirmed to gain access. Two assert statements verify that this is the page returned. Here, it is interesting to note that a search for the string 'Hello, john!' would not work because this string is assembled from static and dynamic portions, so due to the way the Jinja2 template was created the final HTML has extra whitespace in between these two words. To avoid an error in this test due to the whitespace, a regular expression is used.
The next step is to confirm the account, which presents another small obstacle. The confirmation URL is sent to the user by email during registration, so there is no easy way to access it from the test. The solution presented in the test bypasses the token that was generated as part of the registration and generates another one directly from the User instance. Another possibility would have been to extract the token by parsing the email body, which Flask-Mail saves when running in a testing configuration.
With the token at hand, the next step of the test is to simulate the user clicking the confirmation token URL received by email. This is achieved by sending a GET request to the confirmation URL, which includes the token. The response to this request is a redirect to the home page, but once again follow_redirects=True is specified, so the test client requests the redirected page automatically and returns it. The response is checked for the greeting and a flashed message that informs the user that the confirmation was successful.
The final step in this test is to send a GET request to the logout route; to confirm that this has worked, the test searches for the flashed message in the response.
Tip
If you have cloned the application’s Git repository on GitHub, you can run git checkout 15b to check out this version of the application.
Testing Web Services
The Flask test client can also be used to test RESTful web services.Example 15-5 shows an example unit test class with two tests.
Example 15-5. tests/test_api.py: RESTful API testing with the Flask test client
class APITestCase(unittest.TestCase):
# ...
def get_api_headers(self, username, password):
return {
'Authorization':
'Basic ' + b64encode(
(username + ':' + password).encode('utf-8')).decode('utf-8'),
'Accept': 'application/json',
'Content-Type': 'application/json'
}
def test_no_auth(self):
response = self.client.get(url_for('api.get_posts'),
content_type='application/json')
self.assertEqual(response.status_code, 401)
def test_posts(self):
# add a user
r = Role.query.filter_by(name='User').first()
self.assertIsNotNone(r)
u = User(email='john@example.com', password='cat', confirmed=True,
role=r)
db.session.add(u)
db.session.commit()
# write a post
response = self.client.post(
'/api/v1/posts/',
headers=self.get_api_headers('john@example.com', 'cat'),
data=json.dumps({'body': 'body of the *blog* post'}))
self.assertEqual(response.status_code, 201)
url = response.headers.get('Location')
self.assertIsNotNone(url)
# get the new post
response = self.client.get(
url,
headers=self.get_api_headers('john@example.com', 'cat'))
self.assertEqual(response.status_code, 200)
json_response = json.loads(response.get_data(as_text=True))
self.assertEqual('http://localhost' + json_response['url'], url)
self.assertEqual(json_response['body'], 'body of the *blog* post')
self.assertEqual(json_response['body_html'],
'<p>body of the <em>blog</em> post</p>')
The setUp() and tearDown() methods for testing the API are the same as for the regular application, but the cookie support does not need to be configured because the API does not use it. The get_api_headers() method is a helper method that returns the common headers that need to be sent with most API requests. These include the authentication credentials and the MIME type-related headers.
The test_no_auth() test is a simple test that ensures that a request that does not include authentication credentials is rejected with error code 401. The test_posts() test adds a user to the database and then uses the RESTful API to insert a blog post and then read it back. Any requests that send data in the body must encode it with json.dumps(), because the Flask test client does not automatically encode to JSON. Likewise, response bodies are also returned in JSON format and must be decoded with json.loads() before they can be inspected.
Tip
If you have cloned the application’s Git repository on GitHub, you can run git checkout 15c to check out this version of the application.
End-to-End Testing with Selenium
The Flask test client cannot fully emulate the environment of a running application. For example, any application that relies on JavaScript code running in the client browser will not work, as the JavaScript code included in the responses will be returned to the test without being executed.
When tests require the complete environment, there is no other choice than to use a real web browser connected to the application running on a real web server. Fortunately, most web browsers can be automated. Selenium is a web browser automation tool that supports the most popular web browsers in the three major operating systems.
The Python interface for Selenium is installed with pip:
Selenium requires a driver for the desired web browser to be installed separately, in addition to the browser itself. There are drivers for all major web browsers, so an application could set up a sophisticated framework to test several browsers. For this application, however, only the Google Chrome web browser will be used for automated tests, with its corresponding driver, ChromeDriver. If you are using a macOS computer with the brew package installer, you can install ChromeDriver as follows:
For Linux, Microsoft Windows, or a macOS computer without brew, you can download a regular ChromeDriver installer from the ChromeDriver website.
Testing with Selenium requires the application to be running inside a web server that is listening for real HTTP requests. The method that will be shown in this section starts the application with the development server in a background thread while the tests run on the main thread. Under the control of the tests, Selenium launches a web browser and makes it connect to the application to perform the required operations.
A problem with this approach is that after all the tests have completed, the Flask server needs to be stopped, ideally in a graceful way, so that background tasks such as the code coverage engine can cleanly complete their work. The Werkzeug web server has a shutdown option, but because the server is running isolated in its own thread, the only way to ask the server to shut down is by sending a regular HTTP request. Example 15-6 shows the implementation of a server shutdown route.
Example 15-6. _app/main/views.py: server shutdown route
The shutdown route will work only when the application is running in testing mode; invoking it in other configurations will return a 404 status code response. The actual shutdown procedure involves calling a shutdown function that Werkzeug exposes in the environment. After calling this function and returning from the request, the development web server will know that it needs to exit gracefully.
Example 15-7 shows the layout of a test case that is configured to run tests with Selenium.
Example 15-7. tests/test_selenium.py: framework for tests using Selenium
from selenium import webdriver
class SeleniumTestCase(unittest.TestCase):
client = None
@classmethod
def setUpClass(cls):
# start Chrome
options = webdriver.ChromeOptions()
options.add_argument('headless')
try:
cls.client = webdriver.Chrome(chrome_options=options)
except:
pass
# skip these tests if the browser could not be started
if cls.client:
# create the application
cls.app = create_app('testing')
cls.app_context = cls.app.app_context()
cls.app_context.push()
# suppress logging to keep unittest output clean
import logging
logger = logging.getLogger('werkzeug')
logger.setLevel("ERROR")
# create the database and populate with some fake data
db.create_all()
Role.insert_roles()
fake.users(10)
fake.posts(10)
# add an administrator user
admin_role = Role.query.filter_by(permissions=0xff).first()
admin = User(email='john@example.com',
username='john', password='cat',
role=admin_role, confirmed=True)
db.session.add(admin)
db.session.commit()
# start the Flask server in a thread
cls.server_thread = threading.Thread(
target=cls.app.run, kwargs={'debug': 'false',
'use_reloader': False,
'use_debugger': False})
cls.server_thread.start()
@classmethod
def tearDownClass(cls):
if cls.client:
# stop the Flask server and the browser
cls.client.get('http://localhost:5000/shutdown')
cls.client.quit()
cls.server_thread.join()
# destroy database
db.drop_all()
db.session.remove()
# remove application context
cls.app_context.pop()
def setUp(self):
if not self.client:
self.skipTest('Web browser not available')
def tearDown(self):
pass
The setUpClass() and tearDownClass() class methods are invoked before and after the tests in this class execute. The setup involves starting an instance of Chrome through Selenium’s webdriver API, and creating an application and a database with some initial fake data for tests to use. The application is started in a thread using the app.run() method. At the end the application receives a request to /shutdown, which causes the background thread to end. The browser is then closed and the test database removed.
Note
Before the Flask command-line interface based on Click was introduced, you had to start the Flask development web server by calling app.run() from the application’s main script, or else use a third-party extension such as Flask-Script. While using app.run() to start a server is now replaced with the flask run command, the app.run() method continues to be supported, and here you can see how it can still be useful for complex unit testing situations.
Note
Selenium supports many other web browsers besides Chrome. Consult the Selenium documentation if you wish to use another web browser or test additional browsers.
The setUp() method that runs before each test skips tests if Selenium cannot start the web browser in the startUpClass() method. In Example 15-8 you can see an example test built with Selenium.
This test logs in to the application using the administrator account that was created in setUpClass() and then opens the user’s profile page. Note how different the testing methodology is from the Flask test client. When testing with Selenium, tests send commands to the web browser and never interact with the application directly. The commands closely match the actions that a real user would perform with a mouse or keyboard.
The test begins with a call to get() with the home page of the application. In the browser, this causes the URL to be entered in the address bar. To verify this step, the page source is checked for the “Hello, Stranger!” greeting.
To go to the sign-in page, the test looks for the “Log In” link using find_element_by_link_text() and then calls click() on it to trigger a real click in the browser. Selenium provides several find_element_by...() convenience methods that can search for elements within the HTML page in different ways.
To log in to the application, the test locates the email and password form fields by their names using find_element_by_name() and then writes text into them with send_keys(). The form is submitted by calling click() on the submit button. The personalized greeting is checked to ensure that the login was successful and the browser is now on the home page.
The final part of the test locates the “Profile” link in the navigation bar and clicks it. To verify that the profile page was loaded, the heading with the username is searched in the page source.
Tip
If you have cloned the application’s Git repository on GitHub, you can run git checkout 15d to check out this version of the application. This update contains a database migration, so remember to run flask db upgrade after you check out the code. To ensure that you have all the dependencies installed, also run pip install -r requirements/dev.txt.
When you run the unit tests with the flask test command there will be no visible difference. The test_admin_home_page unit test in Example 15-8 will run a headless Chrome instance and perform all the actions on it. If you want to see the actions performed in a real Chrome window, comment out the line options.add_argument('headless') in the setUpClass() method, so that Selenium creates a regular Chrome window.
Is It Worth It?
By now you may be asking yourself if testing using the Flask test client or Selenium is really worth the trouble. It is a valid question, and it does not have a simple answer.
Whether you like it or not, your application will be tested. If you don’t test it yourself, then your users will become the unwilling testers; they will find the bugs, and then you will have to fix them under pressure. Simple and focused tests like the ones that exercise database models and other parts of the application that can be executed outside of the context of an application should always be carried out, as they have a very low cost and ensure the proper functioning of the core pieces of application logic.
End-to-end tests of the type that the Flask test client and Selenium can carry out are sometimes necessary, but due to the increased complexity of writing them, they should be used only for functionality that cannot be tested in isolation. The application code should be organized so that it is possible to push the business logic into application modules that are independent of the context of the application, and thus can be tested more easily. The code that exists in view functions should be simple and just act as a thin layer that accepts requests and invokes the corresponding actions in other classes or functions that encapsulate the application logic.
So yes, testing is absolutely worth it. But it is important to design an efficient testing strategy and write code that can take advantage of it.
Table of contents collapsed