Application Programming Interfaces

Chapter 14. Application Programming Interfaces

In recent years, there has been a trend in web applications to move more and more of the business logic to the client side, producing an architecture that is known as Rich Internet Applications (RIAs). In RIAs, the server’s main (and sometimes only) function is to provide the client application with data retrieval and storage services. In this model, the server becomes a web service or application programming interface (API).

There are several protocols by which RIAs can communicate with a web service. Remote procedure call (RPC) protocols such as XML-RPC or its derivative, the Simplified Object Access Protocol (SOAP), were popular choices a few years ago. More recently, the Representational State Transfer (REST) architecture has emerged as the favorite for web applications due to its being built on the familiar model of the World Wide Web.

Flask is an ideal framework to build RESTful web services, thanks to its lightweight nature. In this chapter, you will learn how to implement a Flask-based RESTful API.

Introduction to REST

Roy Fielding’s PhD dissertation describes the REST architectural style for web services in terms of its six defining characteristics:

Client–server

There must be a clear separation between clients and servers.

Stateless

A client request must contain all the information that is necessary to carry it out. The server must not store any state about the client that persists from one request to the next.

Cache

Responses from the server can be labeled as cacheable or noncacheable so that clients (or intermediaries between clients and servers) can use a cache for optimization purposes.

Uniform interface

The protocol by which clients access server resources must be consistent, well defined, and standardized. This is the most complex aspect of REST, covering the use of unique resource identifiers, resource representations, self-descriptive messages between client and server, and hypermedia.

Layered system

Proxy servers, caches, or gateways can be inserted between clients and servers as necessary to improve performance, reliability, and scalability.

Code-on-demand

Clients can optionally download code from the server to execute in their context.

Resources Are Everything

The concept of resources is core to the REST architectural style. In this context, a resource is an item of interest in the domain of the application. For example, in the blogging application, users, blog posts, and comments are all resources.

Each resource must have a unique identifier that represents it. When working with HTTP, identifiers for resources are URLs. Continuing with the blogging example, a blog post could be represented by the URL /api/posts/12345, where 12345 is the identifier for a post, such as the post’s database primary key. The format or contents of the URL do not really matter; all that matters is that each resource URL uniquely identifies a resource.

A collection of all the resources in a class also has an assigned URL. The URL for the collection of blog posts could be /api/posts/ and the URL for the collection of all comments could be /api/comments/.

An API can also define collection URLs that represent logical subsets of all the resources in a class. For example, the collection of all comments in blog post 12345 could be represented by the URL /api/posts/12345/comments/. It is common to define URLs that represent collections of resources with a trailing slash, as this gives them a “subdirectory” representation.

Tip

Be aware that Flask applies special treatment to routes that end with a slash. If a client requests a URL without a trailing slash and there is a matching route that has a slash at the end, then Flask will automatically respond with a redirect to the trailing-slash URL. No redirects are issued for the reverse case.

Request Methods

The client application sends requests to the server at the established resource URLs and uses the request method to indicate the desired operation. To obtain the list of available blog posts in the blogging API the client would send a GET request to http://www.example.com/api/posts/, and to insert a new blog post it would send a POST request to the same URL, with the contents of the blog post in the request body. To retrieve blog post 12345 the client would send a GET request to http://www.example.com/api/posts/12345. Table 14-1 lists the request methods that are commonly used in RESTful APIs, with their meanings.

Note

The REST architecture does not require that all methods be implemented for a resource. If the client invokes a method that is not supported for a given resource, then a response with the 405 status code (Method Not Allowed) should be returned. Flask handles this error automatically.

The GET, POST, PUT, and DELETE request methods are not the only ones. The HTTP protocol relies on other methods, such as HEAD and OPTIONS, which are automatically implemented by Flask.

Request and Response Bodies

Resources are sent back and forth between client and server in the bodies of requests and responses, but REST does not specify the format to use to encode resources. The Content-Type header in requests and responses is used to indicate the format in which a resource is encoded in the body. The standard content negotiation mechanisms in the HTTP protocol can be used between client and server to agree on a format that both support.

The two formats commonly used with RESTful web services are JavaScript Object Notation (JSON) and Extensible Markup Language (XML). For web-based RIAs, JSON is attractive due to being much more concise than XML, and because of its close ties to JavaScript, the client-side scripting language used by web browsers. Returning to the blog example API, a blog post resource could have the following JSON representation:

Note how the url, author_url, and comments_url fields are fully qualified resource URLs. This is important because these URLs allow the client to discover new resources.

In a well-designed RESTful API, the client knows a short list of top-level resource URLs and then discovers the rest from links included in responses, similar to how you can discover new web pages while browsing the web by clicking on links that appear in pages that you know about.

Versioning

In a traditional server-centric web application, the server has full control of the application. When an application is updated, installing the new version on the server is enough to update all users because even the parts of the application that run in the user’s web browser are downloaded from the server.

The situation with RIAs and web services is more complicated, because often clients are developed independently of the server—maybe even by different people. Consider the case of an application where the RESTful web service is used by a variety of clients including web browsers and native smartphone clients. The web browser client can be updated on the server at any time, but the smartphone apps cannot be updated by force; the smartphone owner needs to allow the update to happen. Even if the smartphone owner is willing to update, it is not possible to orchestrate the upgrade of all existing instances of smartphone applications to coincide exactly with the deployment of the new server version.

For these reasons, web services need to be more tolerant than regular web applications and be able to work with old versions of their clients. Changes to a web service must be done with extreme care, because backward-incompatible changes can cause existing clients to break until they are upgraded. A common practice is to give web services a version, which is added to all URLs defined in that version of the server application. For example, the first release of the blogging web service could expose the collection of blog posts at /api/v1/posts/.

Including the web service version in the URL helps keep old and new features organized so that the server can provide new features to new clients while continuing to support old clients. An update to the blogging service could change the JSON format of blog posts and now expose blog posts as /api/v2/posts/, while keeping the older JSON format for clients that connect to /api/v1/posts/.

Although supporting multiple versions of the server can become a maintenance burden, there are situations in which this is the only way to allow the application to grow without causing problems to existing deployments. Older service versions can be deprecated and later removed, once all clients have migrated to a newer version.

RESTful Web Services with Flask

Flask makes it very easy to create RESTful web services. The familiar route() decorator along with its methods optional argument can be used to declare the routes that handle the resource URLs exposed by the service. Working with JSON data is also simple, as JSON data included with a request can be obtained in dictionary format by calling request.get_json(), and a response that needs to contain JSON can be easily generated from a Python dictionary using Flask’s jsonify() helper function.

The following sections show how Flasky can be extended with a RESTful web service that gives clients access to blog posts and related resources.

Creating an API Blueprint

The routes associated with a RESTful API form a self-contained subset of the application, so putting them in their own blueprint is the best way to keep them well organized. The general structure of the API blueprint within the application is shown in Example 14-1.

Example 14-1. API blueprint structure

Note how the package used for the API includes a version number in its name. If in the future a backward-incompatible version of the API needs to be introduced, it can be added as a separate package with a different version number and both APIs can be included in the application.

The API blueprint implements each resource in a separate module. Modules to take care of authentication and error handling and to provide custom decorators are also included. The blueprint constructor is shown in Example 14-2.

The structure of the blueprint package constructor is similar to that of the other blueprints. Importing all the components of the blueprint is necessary so that routes and other handlers are registered. Since many of these modules need to import the api blueprint referenced here, the imports are done at the bottom to help prevent errors due to circular dependencies.

The registration of the API blueprint is shown in Example 14-3.

The API blueprint is registered with a URL prefix, so that all its routes will have their URLs prefixed with /api/v1. Adding a prefix when registering the blueprint is a good idea because it eliminates the need to hardcode the version number in every blueprint route.

Error Handling

A RESTful web service informs the client of the status of a request by sending the appropriate HTTP status code in the response, plus any additional information in the response body. The typical status codes that a client can expect to see from a web service are listed in Table 14-2.

The handling of status codes 404 and 500 presents a small complication, in that these errors are normally generated by Flask on its own, and will return an HTML response. This can confuse an API client, which will likely expect all responses in JSON format.

One way to generate appropriate responses for all clients is to make the error handlers adapt their responses based on the format requested by the client, a technique called content negotiation. Example 14-4 shows an improved 404 error handler that responds with JSON to web service clients and with HTML to others. The 500 error handler is written in a similar way.

Example 14-4. app/api/errors.py: 404 error handler with HTTP content negotiation

This new version of the error handler checks the Accept request header, which is decoded into request.accept_mimetypes, to determine what format the client wants the response in. Browsers generally do not specify any restrictions on response formats, but API clients typically do. The JSON response is generated only for clients that include JSON in their list of accepted formats, but not HTML.

The remaining status codes are generated explicitly by the web service, so they can be implemented as helper functions inside the blueprint in the errors.py module. Example 14-5 shows the implementation of the 403 error; the others are similar.

Example 14-5. app/api/errors.py: API error handler for status code 403

View functions in the API blueprint can invoke these auxiliary functions to generate error responses when necessary.

User Authentication with Flask-HTTPAuth

Web services, like regular web applications, need to protect information and ensure that it is not given to unauthorized parties. For this reason, RIAs must ask their users for login credentials and pass them to the server for verification.

It was mentioned earlier that one of the characteristics of RESTful web services is that they are stateless, which means that the server is not allowed to “remember” anything about the client between requests. Clients need to provide all the information necessary to carry out a request in the request itself, so all requests must include user credentials.

The current login functionality implemented with the help of Flask-Login stores data in the user session, which Flask stores by default in a client-side cookie, so the server does not store any user-related information; it asks the client to store it instead. It would appear that this implementation complies with the stateless requirement of REST, but the use of cookies in RESTful web services falls into a gray area, as it can be cumbersome for clients that are not web browsers to implement them. For that reason, it is generally seen as a bad design choice to use cookies in APIs.

Note

The stateless requirement of REST may seem overly strict, but it is not arbitrary. Stateless servers can scale very easily. If servers store information about clients, it is necessary to ensure that the same server always gets requests from a given client, or else to use shared storage for client data. Both are complex problems to solve that do not exist when the server is stateless.

Because the RESTful architecture is based on the HTTP protocol, HTTP authentication is the preferred method used to send credentials, either in its Basic or Digest flavor. With HTTP authentication, user credentials are included in an Authorization header with all requests.

The HTTP authentication protocol is simple enough that it can be implemented directly, but the Flask-HTTPAuth extension provides a convenient wrapper that hides the protocol details in a decorator similar to Flask-Login’s login_required.

Flask-HTTPAuth is installed with pip:

To initialize the extension for HTTP Basic authentication, an object of class HTTPBasicAuth must be created. Like Flask-Login, Flask-HTTPAuth makes no assumptions about the procedure required to verify user credentials, so this information is given in a callback function.Example 14-6 shows how the extension is initialized and provided with a verification callback.

Example 14-6. app/api/authentication.py: Flask-HTTPAuth initialization

Because this type of user authentication will be used only in the API blueprint, the Flask-HTTPAuth extension is initialized in the blueprint package, and not in the application package like other extensions.

The email and password are verified using the existing support in the User model. The verification callback returns True when the login is valid and False otherwise. The Flask-HTTPAuth extension also will invoke the callback for requests that carry no authentication, setting both arguments to the empty string. In this case, when email is an empty string, the function immediately returns False to block the request; for certain applications it may be acceptable to allow the anonymous user by returning True. The authentication callback saves the authenticated user in Flask’s g context variable so that the view function can access it later.

Caution

Because user credentials are being exchanged with every request, it is extremely important that the API routes are exposed over secure HTTP so that all requests and responses are encrypted in transit.

When the authentication credentials are invalid, the server returns a 401 status code response to the client. Flask-HTTPAuth generates a response with this status code by default, but to ensure that the response is consistent with other errors returned by the API, the error response can be customized as shown in Example 14-7.

Example 14-7. _app/api/authentication.py: Flask-HTTPAuth error handler

To protect a route, the auth.login_required decorator is used:

But since all the routes in the blueprint need to be protected in the same way, the login_required decorator can be included once in a before_request handler for the blueprint, as shown in Example 14-8.

Now the authentication checks will be done automatically for all the routes in the blueprint. As an additional check, the before_request handler also rejects authenticated users who have not confirmed their accounts.

Token-Based Authentication

Clients must send authentication credentials with every request. To avoid having to constantly transfer sensitive information such as a password, a token-based authentication solution can be used.

In token-based authentication, the client requests an access token by sending a request that includes the login credentials as authentication. The token can then be used in place of the login credentials to authenticate requests. For security reasons, tokens are issued with an associated expiration. When a token expires, the client must reauthenticate to get a new one. The risk of a token getting into the wrong hands is limited due to its short lifespan. Example 14-9 shows the two new methods added to the User model that support generation and verification of authentication tokens using itsdangerous.

Example 14-9. app/models.py: token-based authentication support

The generate_auth_token() method returns a signed token that encodes the user’s id field. An expiration time given in seconds is also used. The verify_auth_token() method takes a token and, if it’s found to be valid, returns the user stored in it. This is a static method, as the user will be known only after the token is decoded.

To authenticate requests that come with a token, the verify_password callback for Flask-HTTPAuth must be modified to accept tokens as well as regular credentials. The updated callback is shown in Example 14-10.

Example 14-10. app/api/authentication.py: improved authentication verification with token support

In this new version, the first authentication argument can be the email address or an authentication token. If this field is blank, an anonymous user is assumed, as before. If the password is blank, then the email_or_token field is assumed to be a token and validated as such. If both fields are nonempty then regular email and password authentication is assumed. With this implementation, token-based authentication is optional; it is up to each client to use it or not. To give view functions the ability to distinguish between the two authentication methods a g.token_used variable is added.

The route that returns authentication tokens to the client is also added to the API blueprint. The implementation is shown in Example 14-11.

Example 14-11. app/api/authentication.py: authentication token generation

Since this route is in the blueprint, the authentication mechanisms added to the before_request handler also apply to it. To prevent clients from authenticating to this route using a previously obtained token instead of an email address and password, the g.token_used variable is checked, and requests authenticated with a token are rejected. The purpose of this is to prevent users from bypassing the token expiration by requesting a new token using the old token as authentication. The function returns a token in the JSON response with a validity period of one hour. The period is also included in the JSON response.

Serializing Resources to and from JSON

A frequent need when writing a web service is to convert internal representations of resources to and from JSON, which is the transport format used in HTTP requests and responses. The process of converting an internal representation to a transport format such as JSON is called serialization. Example 14-12 shows a new to_json() method added to the Post class.

Example 14-12. app/models.py: converting a post to a JSON serializable dictionary

The url, author_url, and comments_url fields need to return the URLs for the respective resources, so these are generated with url_for() calls to other routes that will be defined in the API blueprint.

This example shows how it is possible to return “made-up” attributes in the representation of a resource. The comment_count field returns the number of comments that exist for the blog post. Although this is not a real attribute of the model, it is included in the resource representation as a convenience to the client.

The to_json() method for User models can be constructed in a similar way. This method is shown in Example 14-13.

Example 14-13. app/models.py: converting a user to a JSON serializable dictionary

Note how in this method some of the attributes of the user, such as email and role, are omitted from the response for privacy reasons. This example again demonstrates that the representation of a resource offered to clients does not need to be identical to the internal definition of the corresponding database model.

The inverse of serialization is called deserialization. Deserializing a JSON structure back to a model presents the challenge that some of the data coming from the client might be invalid, wrong, or unnecessary. Example 14-14 shows the method that creates a Post from JSON.

Example 14-14. app/models.py: creating a blog post from JSON

As you can see, this implementation chooses to only use the body attribute from the JSON dictionary. The body_html attribute is ignored since the server-side Markdown rendering is automatically triggered by an SQLAlchemy event whenever the body attribute is modified. The timestamp attribute does not need to be given unless the client is allowed to back- or future-date posts, which is not a feature this application supports. The author_url field is not used because the client has no authority to select the author of a blog post; the only possible value for this field is that of the authenticated user. The comments_url and comment_count attributes are automatically generated from a database relationship, so there is no useful information in them that is needed to create a Post. Finally, the url field is ignored because in this implementation the resource URLs are defined by the server, not the client.

Note how error checking is done. If the body field is missing or empty then a ValidationError exception is raised. Raising an exception is in this case the appropriate way to deal with the error because this method does not have enough knowledge to properly handle the error condition. The exception effectively passes the error up to the caller, enabling higher-level code to do the error handling. The ValidationError class is implemented as a simple subclass of Python’s ValueError. This implementation is shown in Example 14-15.

Example 14-15. app/exceptions.py: ValidationError exception

The application now needs to handle this exception by providing the appropriate response to the client. To avoid having to add exception-catching code in view functions, a global exception handler can be installed using Flask’s errorhandler decorator. A handler for the ValidationError exception is shown in Example 14-16.

Example 14-16. app/api/errors.py: API error handler for ValidationError exceptions

The errorhandler decorator is the same one that is used to register handlers for HTTP status codes, but in this usage it takes an Exception class as an argument. The decorated function will be invoked any time an exception of the given class is raised. Note that the decorator is obtained from the API blueprint, so this handler will be invoked only when the exception is raised while a route from the blueprint is being handled. Using this technique, the code in view functions can be written very cleanly and concisely, without the need to include error checking. For example:

Implementing Resource Endpoints

What remains is to implement the routes that handle the different resources. The GET requests are typically the easiest because they just return information and don’t need to make any changes.Example 14-17 shows the two GET handlers for blog posts.

Example 14-17. app/api/posts.py: GET resource handlers for posts

The first route handles the request for the collection of posts. This function uses a list comprehension to generate the JSON version of all the posts. The second route returns a single blog post and responds with a code 404 error when the given id is not found in the database.

The POST handler for blog post resources inserts a new blog post in the database. This route is shown in Example 14-18.

Example 14-18. app/api/posts.py: POST resource handler for posts

This view function is wrapped in a permission_required decorator (shown in an upcoming example) that ensures that the authenticated user has the permission to write blog posts. The actual creation of the blog post is straightforward due to the error handling support that was implemented previously. A blog post is created from the JSON data and its author is explicitly assigned as the authenticated user. After the model is written to the database, a 201 status code is returned and a Location header is added with the URL of the newly created resource.

Note that as a convenience to clients, the body of the response includes the new resource. This will save the client from having to issue a GET request for it immediately after creating the resource.

The permission_required decorator used to prevent unauthorized users from creating new blog posts is similar to the one used in the application but is customized for the API blueprint. The implementation is shown in Example 14-19.

Example 14-19. app/api/decorators.py: permission_required decorator

The PUT handler for blog posts, used for editing existing resources, is shown in Example 14-20.

Example 14-20. app/api/posts.py: PUT resource handler for posts

The permission checks are more complex in this case. The standard check for permission to write blog posts is done with the decorator, but to allow a user to edit a blog post the function must also ensure that the user is the author of the post or else is an administrator. This check is added explicitly to the view function. If this check had to be added in many view functions, building a decorator for it would be a good way to avoid code repetition.

Since the application does not allow deletion of posts, the handler for the DELETE request method does not need to be implemented.

The resource handlers for users and comments are implemented in a similar way. Table 14-3 lists the set of resources implemented for this application and the HTTP methods each supports. The complete implementation is available for you to study in the GitHub repository for this application.

Note that the resources that were implemented offer only a subset of the functionality that is available through the web application. The list of supported resources could be expanded if necessary, such as to expose followers, to enable comment moderation, and to implement any other features that an API client might need.

Pagination of Large Resource Collections

The GET requests that return a collection of resources can be extremely expensive and difficult to manage for very large collections. Like web applications, web services can choose to paginate collections.

Example 14-21 shows a possible implementation of pagination for the list of blog posts.

Example 14-21. app/api/posts.py: Post pagination

The posts field in the JSON response contains the data items as before, but now it is just a page and not the complete set. The prev_url and next_url items contain the resource URLs for the previous and following pages, or None when a page in that direction is not available. The count value is the total number of items in the collection.

This technique can be applied to all the routes that return collections.

Tip

If you have cloned the application’s Git repository on GitHub, you can run git checkout 14a to check out this version of the application. To ensure that you have all the dependencies installed, also run pip install -r requirements/dev.txt.

Testing Web Services with HTTPie

To test a web service, an HTTP client must be used. The two most used clients for testing Python web services from the command line are cURL and HTTPie. While both are useful tools, the latter has a much more concise and readable command line syntax that is tailored specifically to API requests. HTTPie is installed with pip:

Assuming the development server is running on the default http://127.0.0.1:5000 address, a GET request can be issued from another terminal window as follows:

Note the pagination links included in the response. Since this is the first page, a previous page is not defined, but a URL to obtain the next page and a total count were returned.

The following command sends a POST request to add a new blog post:

To use authentication tokens instead of a username and password, a POST request to /api/v1/tokens/ is sent first:

And now the returned token can be used to make calls into the API for the next hour by passing it along in the username field and leaving the password empty:

When the token expires, requests will be returned with a code 401 error, indicating that a new token needs to be obtained.

Congratulations! This chapter completes Part II, and with that the feature development phase of Flasky is complete. The next step is obviously to deploy it, and that brings a new set of challenges that are the subject of Part III.

Table of contents collapsed