In this article by Jason Krol, the author of Web Development with MongoDB and NodeJS, we will review the following topics:
(For more resources related to this topic, see here.)
- Introducing RESTful APIs
- Installing a few basic tools
- Creating a basic API server and sample JSON data
- Responding to GET requests
- Updating data with POST and PUT
- Removing data with DELETE
- Consuming external APIs from Node
What is an API?
An Application Programming Interface (API) is a set of tools that a computer system makes available that provides unrelated systems or software the ability to interact with each other. Typically, a developer uses an API when writing software that will interact with a closed, external, software system. The external software system provides an API as a standard set of tools that all developers can use. Many popular social networking sites provide developer’s access to APIs to build tools to support those sites. The most obvious examples are Facebook and Twitter. Both have a robust API that provides developers with the ability to build plugins and work with data directly, without them being granted full access as a general security precaution.
As you will see with this article, providing your own API is not only fairly simple, but also it empowers you to provide your users with access to your data. You also have the added peace of mind knowing that you are in complete control over what level of access you can grant, what sets of data you can make read-only, as well as what data can be inserted and updated.
What is a RESTful API?
Representational State Transfer (REST) is a fancy way of saying CRUD over HTTP. What this means is when you use a REST API, you have a uniform means to create, read, and update data using simple HTTP URLs with a standard set of HTTP verbs. The most basic form of a REST API will accept one of the HTTP verbs at a URL and return some kind of data as a response.
Typically, a REST API GET request will always return some kind of data such as JSON, XML, HTML, or plain text. A POST or PUT request to a RESTful API URL will accept data to create or update. The URL for a RESTful API is known as an endpoint, and while working with these endpoints, it is typically said that you are consuming them. The standard HTTP verbs used while interfacing with REST APIs include:
- GET: This retrieves data
- POST: This submits data for a new record
- PUT: This submits data to update an existing record
- PATCH: This submits a date to update only specific parts of an existing record
- DELETE: This deletes a specific record
Typically, RESTful API endpoints are defined in a way that they mimic the data models and have semantic URLs that are somewhat representative of the data models. What this means is that to request a list of models, for example, you would access an API endpoint of /models. Likewise, to retrieve a specific model by its ID, you would include that in the endpoint URL via /models/:Id.
Some sample RESTful API endpoint URLs are as follows:
- GET http://myapi.com/v1/accounts: This returns a list of accounts
- GET http://myapi.com/v1/accounts/1: This returns a single account by Id: 1
- POST http://myapi.com/v1/accounts: This creates a new account (data submitted as a part of the request)
- PUT http://myapi.com/v1/accounts/1: This updates an existing account by Id: 1 (data submitted as part of the request)
- GET http://myapi.com/v1/accounts/1/orders: This returns a list of orders for account Id: 1
- GET http://myapi.com/v1/accounts/1/orders/21345: This returns the details for a single order by Order Id: 21345 for account Id: 1
It’s not a requirement that the URL endpoints match this pattern; it’s just common convention.
Introducing Postman REST Client
Before we get started, there are a few tools that will make life much easier when you’re working directly with APIs. The first of these tools is called Postman REST Client, and it’s a Google Chrome application that can run right in your browser or as a standalone-packaged application. Using this tool, you can easily make any kind of request to any endpoint you want. The tool provides many useful and powerful features that are very easy to use and, best of all, free!
Installation instructions
Postman REST Client can be installed in two different ways, but both require Google Chrome to be installed and running on your system. The easiest way to install the application is by visiting the Chrome Web Store at https://chrome.google.com/webstore/category/apps.
Perform a search for Postman REST Client and multiple results will be returned. There is the regular Postman REST Client that runs as an application built into your browser, and then separate Postman REST Client (packaged app) that runs as a standalone application on your system in its own dedicated window. Go ahead and install your preference. If you install the application as the standalone packaged app, an icon to launch it will be added to your dock or taskbar. If you installed it as a regular browser app, you can launch it by opening a new tab in Google Chrome and going to Apps and finding the Postman REST Client icon.
After you’ve installed and launched the app, you should be presented with an output similar to the following screenshot:
A quick tour of Postman REST Client
Using Postman REST Client, we’re able to submit REST API calls to any endpoint we want as well as modify the type of request. Then, we can have complete access to the data that’s returned from the API as well as any errors that might have occurred. To test an API call, enter the URL to your favorite website in the Enter request URL here field and leave the dropdown next to it as GET. This will mimic a standard GET request that your browser performs anytime you visit a website. Click on the blue Send button. The request is made and the response is displayed at the bottom half of the screen. In the following screenshot, I sent a simple GET request to http://kroltech.com and the HTML is returned as follows:
If we change this URL to that of the RSS feed URL for my website, you can see the XML returned:
The XML view has a few more features as it exposes the sidebar to the right that gives you a handy outline to glimpse the tree structure of the XML data. Not only that, you can now see a history of the requests we’ve made so far along the left sidebar. This is great when we’re doing more advanced POST or PUT requests and don’t want to repeat the data setup for each request while testing an endpoint.
Here is a sample API endpoint I submitted a GET request to that returns the JSON data in its response:
A really nice thing about making API calls to endpoints that return JSON using Postman Client is that it parses and displays the JSON in a very nicely formatted way, and each node in the data is expandable and collapsible.
The app is very intuitive so make sure you spend some time playing around and experimenting with different types of calls to different URLs.
Using the JSONView Chrome extension
There is one other tool I want to let you know about (while extremely minor) that is actually a really big deal. The JSONView Chrome extension is a very small plugin that will instantly convert any JSON you view directly via the browser into a more usable JSON tree (exactly like Postman Client). Here is an example of pointing to a URL that returns JSON from Chrome before JSONView is installed:
And here is that same URL after JSONView has been installed:
You should install the JSONView Google Chrome extension the same way you installed Postman REST Client—access the Chrome Web Store and perform a search for JSONView.
Now that you have the tools to be able to easily work with and test API endpoints, let’s take a look at writing your own and handling the different request types.
Creating a Basic API server
Let’s create a super basic Node.js server using Express that we’ll use to create our own API. Then, we can send tests to the API using Postman REST Client to see how it all works. In a new project workspace, first install the npm modules that we’re going to need in order to get our server up and running:
$ npm init $ npm install --save express body-parser underscore
Now that the package.json file for this project has been initialized and the modules installed, let’s create a basic server file to bootstrap up an Express server. Create a file named server.js and insert the following block of code:
var express = require('express'),
bodyParser = require('body-parser'),
_ = require('underscore'),
json = require('./movies.json'),
app = express();
app.set('port', process.env.PORT || 3500);
app.use(bodyParser.urlencoded());
app.use(bodyParser.json());
var router = new express.Router();
// TO DO: Setup endpoints ...
app.use('/', router);
var server = app.listen(app.get('port'), function() {
console.log('Server up: http://localhost:' + app.get('port'));
});
Most of this should look familiar to you. In the server.js file, we are requiring the express, body-parser, and underscore modules. We’re also requiring a file named movies.json, which we’ll create next.
After our modules are required, we set up the standard configuration for an Express server with the minimum amount of configuration needed to support an API server. Notice that we didn’t set up Handlebars as a view-rendering engine because we aren’t going to be rendering any HTML with this server, just pure JSON responses.
Creating sample JSON data
Let’s create the sample movies.json file that will act as our temporary data store (even though the API we build for the purposes of demonstration won’t actually persist data beyond the app’s life cycle):
[{
"Id": "1",
"Title": "Aliens",
"Director": "James Cameron",
"Year": "1986",
"Rating": "8.5"
},
{
"Id": "2",
"Title": "Big Trouble in Little China",
"Director": "John Carpenter",
"Year": "1986",
"Rating": "7.3"
},
{
"Id": "3",
"Title": "Killer Klowns from Outer Space",
"Director": "Stephen Chiodo",
"Year": "1988",
"Rating": "6.0"
},
{
"Id": "4",
"Title": "Heat",
"Director": "Michael Mann",
"Year": "1995",
"Rating": "8.3"
},
{
"Id": "5",
"Title": "The Raid: Redemption",
"Director": "Gareth Evans",
"Year": "2011",
"Rating": "7.6"
}]
This is just a really simple JSON list of a few of my favorite movies. Feel free to populate it with whatever you like. Boot up the server to make sure you aren’t getting any errors (note we haven’t set up any routes yet, so it won’t actually do anything if you tried to load it via a browser):
$ node server.js Server up: http://localhost:3500
Responding to GET requests
Adding a simple GET request support is fairly simple, and you’ve seen this before already in the app we built. Here is some sample code that responds to a GET request and returns a simple JavaScript object as JSON. Insert the following code in the routes section where we have the // TO DO: Setup endpoints … waiting comment:
router.get('/test', function(req, res) {
var data = {
name: 'Jason Krol',
website: 'http://kroltech.com'
};
res.json(data);
});
Let’s tweak the function a little bit and change it so that it responds to a GET request against the root URL (that is /) route and returns the JSON data from our movies file. Add this new route after the /test route added previously:
router.get('/', function(req, res) {
res.json(json);
});
The res (response) object in Express has a few different methods to send data back to the browser. Each of these ultimately falls back on the base send method, which includes header information, statusCodes, and so on. res.json and res.jsonp will automatically format JavaScript objects into JSON and then send using res.send. res.render will render a template view as a string and then send it using res.send as well.
With that code in place, if we launch the server.js file, the server will be listening for a GET request to the / URL route and will respond with the JSON data of our movies collection. Let’s first test it out using the Postman REST Client tool:
GET requests are nice because we could have just as easily pulled that same URL via our browser and received the same result:
However, we’re going to use Postman for the remainder of our endpoint testing as it’s a little more difficult to send POST and PUT requests using a browser.
Receiving data – POST and PUT requests
When we want to allow our users using our API to insert or update data, we need to accept a request from a different HTTP verb. When inserting new data, the POST verb is the preferred method to accept data and know it’s for an insert. Let’s take a look at code that accepts a POST request and data along with the request, and inserts a record into our collection and returns the updated JSON. Insert the following block of code after the route you added previously for GET:
router.post('/', function(req, res) {
// insert the new item into the collection (validate first)
if(req.body.Id && req.body.Title && req.body.Director && req.body.Year && req.body.Rating) {
json.push(req.body);
res.json(json);
} else {
res.json(500, { error: 'There was an error!' });
}
});
You can see the first thing we do in the POST function is check to make sure the required fields were submitted along with the actual request. Assuming our data checks out and all the required fields are accounted for (in our case every field), we insert the entire req.body object into the array as is using the array’s push function. If any of the required fields aren’t submitted with the request, we return a 500 error message instead. Let’s submit a POST request this time to the same endpoint using the Postman REST Client. (Don’t forget to make sure your API server is running with node server.js.):
First, we submitted a POST request with no data, so you can clearly see the 500 error response that was returned.
Next, we provided the actual data using the x-www-form-urlencoded option in Postman and provided each of the name/value pairs with some new custom data. You can see from the results that the STATUS was 200, which is a success and the updated JSON data was returned as a result. Reloading the main GET endpoint in a browser yields our original movies collection with the new one added.
PUT requests will work in almost exactly the same way except traditionally, the Id property of the data is handled a little differently. In our example, we are going to require the Id attribute as a part of the URL and not accept it as a parameter in the data that’s submitted (since it’s usually not common for an update function to change the actual Id of the object it’s updating). Insert the following code for the PUT route after the existing POST route you added earlier:
router.put('/:id', function(req, res) {
// update the item in the collection
if(req.params.id && req.body.Title && req.body.Director && req.body.Year && req.body.Rating) {
_.each(json, function(elem, index) {
// find and update:
if (elem.Id === req.params.id) {
elem.Title = req.body.Title;
elem.Director = req.body.Director;
elem.Year = req.body.Year;
elem.Rating = req.body.Rating;
}
});
res.json(json);
} else {
res.json(500, { error: 'There was an error!' });
}
});
This code again validates that the required fields are included with the data that was submitted along with the request. Then, it performs an _.each loop (using the underscore module) to look through the collection of movies and find the one whose Id parameter matches that of the Id included in the URL parameter. Assuming there’s a match, the individual fields for that matched object are updated with the new values that were sent with the request. Once the loop is complete, the updated JSON data is sent back as the response. Similarly, in the POST request, if any of the required fields are missing, a simple 500 error message is returned. The following screenshot demonstrates a successful PUT request updating an existing record.
The response from Postman after including the value 1 in the URL as the Id parameter, which provides the individual fields to update as x-www-form-urlencoded values, and finally sending as PUT shows that the original item in our movies collection is now the original Alien (not Aliens, its sequel as we originally had).
Removing data – DELETE
The final stop on our whirlwind tour of the different REST API HTTP verbs is DELETE. It should be no surprise that sending a DELETE request should do exactly what it sounds like. Let’s add another route that accepts DELETE requests and will delete an item from our movies collection. Here is the code that takes care of DELETE requests that should be placed after the existing block of code from the previous PUT:
router.delete('/:id', function(req, res) {
var indexToDel = -1;
_.each(json, function(elem, index) {
if (elem.Id === req.params.id) {
indexToDel = index;
}
});
if (~indexToDel) {
json.splice(indexToDel, 1);
}
res.json(json);
});
This code will loop through the collection of movies and find a matching item by comparing the values of Id. If a match is found, the array index for the matched item is held until the loop is finished. Using the array.splice function, we can remove an array item at a specific index. Once the data has been updated by removing the requested item, the JSON data is returned. Notice in the following screenshot that the updated JSON that’s returned is in fact no longer displaying the original second item we deleted.
Note that ~ in there! That’s a little bit of JavaScript black magic! The tilde (~) in JavaScript will bit flip a value. In other words, take a value and return the negative of that value incremented by one, that is ~n === -(n+1). Typically, the tilde is used with functions that return -1 as a false response. By using ~ on -1, you are converting it to a 0. If you were to perform a Boolean check on -1 in JavaScript, it would return true. You will see ~ is used primarily with the indexOf function and jQuery’s $.inArray()—both return -1 as a false response.
All of the endpoints defined in this article are extremely rudimentary, and most of these should never ever see the light of day in a production environment! Whenever you have an API that accepts anything other than GET requests, you need to be sure to enforce extremely strict validation and authentication rules. After all, you are basically giving your users direct access to your data.
Consuming external APIs from Node.js
There will undoubtedly be a time when you want to consume an API directly from within your Node.js code. Perhaps, your own API endpoint needs to first fetch data from some other unrelated third-party API before sending a response. Whatever the reason, the act of sending a request to an external API endpoint and receiving a response can be done fairly easily using a popular and well-known npm module called Request. Request was written by Mikeal Rogers and is currently the third most popular and (most relied upon) npm module after async and underscore.
Request is basically a super simple HTTP client, so everything you’ve been doing with Postman REST Client so far is basically what Request can do, only the resulting data is available to you in your node code as well as the response status codes and/or errors, if any.
Consuming an API endpoint using Request
Let’s do a neat trick and actually consume our own endpoint as if it was some third-party external API. First, we need to ensure we have Request installed and can include it in our app:
$ npm install --save request
Next, edit server.js and make sure you include Request as a required module at the start of the file:
var express = require('express'),
bodyParser = require('body-parser'),
_ = require('underscore'),
json = require('./movies.json'),
app = express(),
request = require('request');
Now let’s add a new endpoint after our existing routes, which will be an endpoint accessible in our server via a GET request to /external-api. This endpoint, however, will actually consume another endpoint on another server, but for the purposes of this example, that other server is actually the same server we’re currently running!
The Request module accepts an options object with a number of different parameters and settings, but for this particular example, we only care about a few. We’re going to pass an object that has a setting for the method (GET, POST, PUT, and so on) and the URL of the endpoint we want to consume. After the request is made and a response is received, we want an inline callback function to execute. Place the following block of code after your existing list of routes in server.js:
router.get('/external-api', function(req, res) {
request({
method: 'GET',
uri: 'http://localhost:' + (process.env.PORT || 3500),
}, function(error, response, body) {
if (error) { throw error; }
var movies = [];
_.each(JSON.parse(body), function(elem, index) {
movies.push({
Title: elem.Title,
Rating: elem.Rating
});
});
res.json(_.sortBy(movies, 'Rating').reverse());
});
});
The callback function accepts three parameters: error, response, and body. The response object is like any other response that Express handles and has all of the various parameters as such. The third parameter, body, is what we’re really interested in. That will contain the actual result of the request to the endpoint that we called. In this case, it is the JSON data from our main GET route we defined earlier that returns our own list of movies. It’s important to note that the data returned from the request is returned as a string. We need to use JSON.parse to convert that string to actual usable JSON data.
Using the data that came back from the request, we transform it a little bit. That is, we take that data and manipulate it a bit to suit our needs. In this example, we took the master list of movies and just returned a new collection that consists of only the title and rating of each movie and then sorts the results by the top scores. Load this new endpoint by pointing your browser to http://localhost:3500/external-api, and you can see the new transformed JSON output to the screen.
Let’s take a look at another example that’s a little more real world. Let’s say that we want to display a list of similar movies for each one in our collection, but we want to look up that data somewhere such as www.imdb.com. Here is the sample code that will send a GET request to IMDB’s JSON API, specifically for the word aliens, and returns a list of related movies by the title and year. Go ahead and place this block of code after the previous route for external-api:
router.get('/imdb', function(req, res) {
request({
method: 'GET',
uri: 'http://sg.media-imdb.com/suggests/a/aliens.json',
}, function(err, response, body) {
var data = body.substring(body.indexOf('(')+1);
data = JSON.parse(data.substring(0,data.length-1));
var related = [];
_.each(data.d, function(movie, index) {
related.push({
Title: movie.l,
Year: movie.y,
Poster: movie.i ? movie.i[0] : ''
});
});
res.json(related);
});
});
If we take a look at this new endpoint in a browser, we can see the JSON data that’s returned from our /imdb endpoint is actually itself retrieving and returning data from some other API endpoint:
Note that the JSON endpoint I’m using for IMDB isn’t actually from their API, but rather what they use on their homepage when you type in the main search box. This would not really be the most appropriate way to use their data, but it’s more of a hack to show this example. In reality, to use their API (like most other APIs), you would need to register and get an API key that you would use so that they can properly track how much data you are requesting on a daily or an hourly basis. Most APIs will to require you to use a private key with them for this same reason.
Summary
In this article, we took a brief look at how APIs work in general, the RESTful API approach to semantic URL paths and arguments, and created a bare bones API. We used Postman REST Client to interact with the API by consuming endpoints and testing the different types of request methods (GET, POST, PUT, and so on). You also learned how to consume an external API endpoint by using the third-party node module Request.
Resources for Article:
Further resources on this subject:
- RESTful Services JAX-RS 2.0 [Article]
- REST – Where It Begins [Article]
- RESTful Web Services – Server-Sent Events (SSE) [Article]
