In this article by Kevin Harvey, author of the book Test-Driven Development with Django, we’ll expose the data in our application via a REST API. As we do, we’ll learn:
- The importance of documentation in the API development process
- How to write functional tests for API endpoints
- API patterns and best practices
(For more resources related to this topic, see here.)
It’s an API world, we’re just coding in it
It’s very common nowadays to include a public REST API in your web project. Exposing your services or data to the world is generally done for one of two reasons:
- You’ve got interesting data, and other developers might want to integrate that information into a project they’re working on
- You’re building a secondary system that you expect your users to interact with, and that system needs to interact with your data (that is, a mobile or desktop app, or an AJAX-driven front end)
We’ve got both reasons in our application. We’re housing novel, interesting data in our database that someone might want to access programmatically. Also, it would make sense to build a desktop application that could interact with a user’s own digital music collection so they could actually hear the solos we’re storing in our system.
Deceptive simplicity
The good news is that there are some great options for third-party plugins for Django that allow you to build a REST API into an existing application. The bad news is that the simplicity of adding one of these packages can let you go off half-cocked, throwing an API on top of your project without a real plan for it.
If you’re lucky, you’ll just wind up with a bird’s nest of an API: inconsistent URLs, wildly varying payloads, and difficult authentication. In the worst-case scenario, your bolt-on API exposes data you didn’t intend to make public and wind up with a self-inflicted security issue.
Never forget that an API is sort of invisible. Unlike traditional web pages, where bugs are very public and easy to describe, API bugs are only visible to other developers. Take special care to make sure your API behaves exactly as intended by writing thorough documentation and tests to make sure you’ve implemented it correctly.
Writing documentation first
“Documentation is king.”
– Kenneth Reitz
If you’ve spent any time at all working with Python or Django, you know what good documentation looks like. The Django folks in particular seem to understand this well: the key to getting developers to use your code is great documentation.
In documenting an API, be explicit. Most of your API methods’ docs should take the form of “if you send this, you will get back this”, with real-world examples of input and output.
A great side effect of prewriting documentation is that it makes the intention of your API crystal clear. You’re allowing yourself to conjure up the API from thin air without getting bogged down in any of the details, so you can get a bird’s-eye view of what you’re trying to accomplish. Your documentation will keep you oriented throughout the development process.
Documentation-Driven testing
Once you’ve got your documentation done, testing is simply a matter of writing test cases that match up with what you’ve promised. The actions of the test methods exercise HTTP methods, and your assertions check the responses.
Test-Driven Development really shines when it comes to API development. There are great tools for sending JSON over the wire, but properly formatting JSON can be a pain, and reading it can be worse. Enshrining test JSON in test methods and asserting they match the real responses will save you a ton of headache.
More developers, more problems
Good documentation and test coverage are exponentially more important when two groups are developing in tandem—one on the client application and one on the API. Changes to an API are hard for teams like this to deal with, and should come with a lot of warning (and apologies). If you have to make a change to an endpoint, it should break a lot of tests, and you should methodically go and fix them all. What’s more, no one feels the pain of regression bugs like the developer of an API-consuming client. You really, really, really need to know that all the endpoints you’ve put out there are still going to work when you add features or refactor.
Building an API with Django REST framework
Now that you’re properly terrified of developing an API, let’s get started. What sort of capabilities should we add? Here are a couple possibilities:
- Exposing the Album, Track, and Solo information we have
- Creating new Solos or updating existing ones
Initial documentation
In the Python world it’s very common for documentation to live in docstrings, as it keeps the description of how to use an object close to the implementation. We’ll eventually do the same with our docs, but it’s kind of hard to write a docstring for a method that doesn’t exist yet. Let’s open up a new Markdown file API.md, right in the root of the project, just to get us started. If you’ve never used Markdown before, you can read an introduction to GitHub’s version of Markdown at https://help.github.com/articles/markdown-basics/.
Here’s a sample of what should go in API.md. Have a look at https://github.com/kevinharvey/jmad/blob/master/API.md for the full, rendered version.
...
# Get a Track with Solos
* URL: /api/tracks/<pk>/
* HTTP Method: GET
## Example Response
{
"name": "All Blues",
"slug": "all-blues",
"album": {
"name": "Kind of Blue",
"url": "http://jmad.us/api/albums/2/"
},
"solos": [
{
"artist": "Cannonball Adderley",
"instrument": "saxophone",
"start_time": "4:05",
"end_time": "6:04",
"slug": "cannonball-adderley",
"url": "http://jmad.us/api/solos/281/"
},
...
]
}
# Add a Solo to a Track
* URL: /api/solos/
* HTTP Method: POST
## Example Request
{
"track": "/api/tracks/83/",
"artist": "Don Cherry",
"instrument": "cornet",
"start_time": "2:13",
"end_time": "3:54"
}
## Example Response
{
"url": "http://jmad.us/api/solos/64/",
"artist": "Don Cherry",
"slug": "don-cherry",
"instrument": "cornet",
"start_time": "2:13",
"end_time": "3:54",
"track": "http://jmad.us/api/tracks/83/"
}
There’s not a lot of prose, and there needn’t be. All we’re trying to do is layout the ins and outs of our API. It’s important at this point to step back and have a look at the endpoints in their totality. Is there enough of a pattern that you can sort of guess what the next one is going to look like? Does it look like a fairly straightforward API to interact with? Does anything about it feel clunky? Would you want to work with this API by yourself? Take time to think through any weirdness now before anything gets out in the wild.
$ git commit -am 'Initial API Documentation'
$ git tag -a ch7-1-init-api-docs
Introducing Django REST framework
Now that we’ve got some idea what we’re building, let’s actually get it going. We’ll be using Django REST Framework (http://www.django-rest-framework.org/). Start by installing it in your environment:
$ pip install djangorestframework
Add rest_framework to your INSTALLED_APPS in jmad/settings.py:
INSTALLED_APPS = (
...
'rest_framework'
)
Now we’re ready to start testing.
Writing tests for API endpoints
While there’s no such thing as browser-based testing for an external API, it is important to write tests that cover its end-to-end processing. We need to be able to send in requests like the ones we’ve documented and confirm that we receive the responses our documentation promises.
Django REST Framework (DRF from here on out) provides tools to help write tests for the application functionality it provides. We’ll use rest_framework.tests.APITestCase to write functional tests. Let’s kick off with the list of albums. Convert albums/tests.py to a package, and add a test_api.py file. Then add the following:
from rest_framework.test import APITestCase
from albums.models import Album
class AlbumAPITestCase(APITestCase):
def setUp(self):
self.kind_of_blue = Album.objects.create(
name='Kind of Blue')
self.a_love_supreme = Album.objects.create(
name='A Love Supreme')
def test_list_albums(self):
"""
Test that we can get a list of albums
"""
response = self.client.get('/api/albums/')
self.assertEqual(response.status_code, 200)
self.assertEqual(response.data[0]['name'],
'A Love Supreme')
self.assertEqual(response.data[1]['url'],
'http://testserver/api/albums/1/')
Since much of this is very similar to other tests that we’ve seen before, let’s talk about the important differences:
- We import and subclass APITestCase, which makes self.client an instance of rest_framework.test.APIClient. Both of these subclass their respective django.test counterparts add a few niceties that help in testing APIs (none of which are showcased yet).
- We test response.data, which we expect to be a list of Albums. response.data will be a Python dict or list that corresponds to the JSON payload of the response.
- During the course of the test, APIClient (a subclass of Client) will use http://testserver as the protocol and hostname for the server, and our API should return a host-specific URI.
Run this test, and we get the following:
$ python manage.py test albums.tests.test_api
Creating test database for alias 'default'...
F
=====================================================================
FAIL: test_list_albums (albums.tests.test_api.AlbumAPITestCase)
Test that we can get a list of albums
---------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/kevin/dev/jmad-project/jmad/albums/tests/test_api.py",
line 17, in test_list_albums
self.assertEqual(response.status_code, 200)
AssertionError: 404 != 200
---------------------------------------------------------------------
Ran 1 test in 0.019s
FAILED (failures=1)
We’re failing because we’re getting a 404 Not Found instead of a 200 OK status code. Proper HTTP communication is important in any web application, but it really comes in to play when you’re using AJAX. Most frontend libraries will properly classify responses as successful or erroneous based on the status code: making sure the code are on point will save your frontend developers friends a lot of headache.
We’re getting a 404 because we don’t have a URL defined yet. Before we set up the route, let’s add a quick unit test for routing. Update the test case with one new import and method:
from django.core.urlresolvers import resolve
...
def test_album_list_route(self):
"""
Test that we've got routing set up for Albums
"""
route = resolve('/api/albums/')
self.assertEqual(route.func.__name__, 'AlbumViewSet')
Here, we’re just confirming that the URL routes to the correct view. Run it:
$ python manage.py test
albums.tests.test_api.AlbumAPITestCase.test_album_list_route
...
django.core.urlresolvers.Resolver404: {'path': 'api/albums/',
'tried': [[<RegexURLResolver <RegexURLPattern list> (admin:admin)
^admin/>], [<RegexURLPattern solo_detail_view
^recordings/(?P<album>[w-]+)/(?P<track>[w-]+)/(?P<artist>[w-
]+)/$>], [<RegexURLPattern None ^$>]]}
---------------------------------------------------------------------
Ran 1 test in 0.003s
FAILED (errors=1)
We get a Resolver404 error, which is expected since Django shouldn’t return anything at that path. Now we’re ready to set up our URLs.
API routing with DRF’s SimpleRouter
Take a look at the documentation for routers at http://www.django-rest-framework.org/api-guide/routers/. They’re a very clean way of setting up URLs for DRF-powered views. Update jmad/urls.py like so:
...
from rest_framework import routers
from albums.views import AlbumViewSet
router = routers.SimpleRouter()
router.register(r'albums', AlbumViewSet)
urlpatterns = [
# Admin
url(r'^admin/', include(admin.site.urls)),
# API
url(r'^api/', include(router.urls)),
# Apps
url(r'^recordings/(?P<album>[w-]+)/(?P<track>[w-]+)/
(?P<artist>[w-]+)/$',
'solos.views.solo_detail',
name='solo_detail_view'),
url(r'^$', 'solos.views.index'),
]
Here’s what we changed:
- We created an instance of SimpleRouter and used the register method to set up a route. The register method has two required arguments: a prefix to build the route methods from, and something called a viewset. Here we’ve supplied a non-existent class AlbumViewSet, which we’ll come back to later.
- We’ve added a few comments to break up our urls.py, which was starting to look a little like a rat’s nest.
- The actual API URLs are registered under the ‘^api/’ path using Django’s include function.
Run the URL test again, and we’ll get ImportError for AlbumViewSet. Let’s add a stub to albums/views.py:
class AlbumViewSet():
pass
Run the test now, and we’ll start to see some specific DRF error messages to help us build out our view:
$ python manage.py test
albums.tests.test_api.AlbumAPITestCase.test_album_list_route
Creating test database for alias 'default'...
F
...
File "/Users/kevin/.virtualenvs/jmad/lib/python3.4/sitepackages/
rest_framework/routers.py", line 60, in register
base_name = self.get_default_base_name(viewset)
File "/Users/kevin/.virtualenvs/jmad/lib/python3.4/sitepackages/
rest_framework/routers.py", line 135, in
get_default_base_name
assert queryset is not None, ''base_name' argument not specified,
and could '
AssertionError: 'base_name' argument not specified, and could not
automatically determine the name from the viewset, as it does not
have a '.queryset' attribute.
After a fairly lengthy output, the test runner tells us that it was unable to get base_name for the URL, as we did not specify the base_name in the register method, and it couldn’t guess the name because the viewset (AlbumViewSet) did not have a queryset attribute.
In the router documentation, we came across the optional base_name argument for register (as well as the exact wording of this error). You can use that argument to control the name your URL gets. However, let’s keep letting DRF do its default behavior. We haven’t read the documentation for viewsets yet, but we know that a regular Django class-based view expects a queryset parameter. Let’s stick one on AlbumViewSet and see what happens:
from .models import Album
class AlbumViewSet():
queryset = Album.objects.all()
Run the test again, and we get:
django.core.urlresolvers.Resolver404: {'path': 'api/albums/',
'tried': [[<RegexURLResolver <RegexURLPattern list> (admin:admin)
^admin/>], [<RegexURLPattern solo_detail_view
^recordings/(?P<album>[w-]+)/(?P<track>[w-]+)/(?P<artist>[w-
]+)/$>], [<RegexURLPattern None ^$>]]}
---------------------------------------------------------------------
Ran 1 test in 0.011s
FAILED (errors=1)
Huh? Another 404 is a step backwards. What did we do wrong? Maybe it’s time to figure out what a viewset really is.
Summary
In this article, we covered basic API design and testing patterns, including the importance of documentation when developing an API. In doing so, we took a deep dive into Django REST Framework and the utilities and testing tools available in it.
Resources for Article:
Further resources on this subject:
- Test-driven API Development with Django REST Framework [Article]
- Adding a developer with Django forms [Article]
- Code Style in Django [Article]
