Categories: CMS & E-CommerceWeb DevelopmentTutorials

Communicating with Servers

24 min read

(For more resources related to this topic, see here.)

Creating an HTTP GET request to fetch JSON

One of the basic means of retrieving information from the server is using HTTP GET. This type of method in a RESTful manner should be only used for reading data. So, GET calls should never change server state. Now, this may not be true for every possible case, for example, if we have a view counter on a certain resource, is that a real change? Well, if we follow the definition literally then yes, this is a change, but it’s far from significant to be taken into account.

Opening a web page in a browser does a GET request, but often we want to have a scripted way of retrieving data. This is usually to achieve Asynchronous JavaScript and XML (AJAX ), allowing reloading of data without doing a complete page reload. Despite the name, the use of XML is not required, and these days, JSON is the format of choice.

A combination of JavaScript and the XMLHttpRequest object provides a method for exchanging data asynchronously, and in this recipe, we are going to see how to read JSON for the server using plain JavaScript and jQuery. Why use plain JavaScript rather than using jQuery directly? We strongly believe that jQuery simplifies the DOM API, but it is not always available to us, and additionally, we need have to know the underlying code behind asynchronous data transfer in order to fully grasp how applications work.

Getting ready

The server will be implemented using Node.js. In this example, for simplicity, we will use restify (http://mcavage.github.io/node-restify/), a Node.js module for creation of correct REST web services.

How to do it…

Let’s perform the following steps.

In order to include restify to our project in the root directory of our server side scripts, use the following command:
```
npm install restify
```
After adding the dependency, we can proceed to creating the server code. We create a server.js file that will be run by Node.js, and at the beginning of it we add restify:
```
var restify = require('restify');
```

With this restify object, we can now create a server object and add handlers for get methods:

var server = restify.createServer();
server.get('hi', respond);
server.get('hi/:index', respond);

The get handlers do a callback to a function called respond, so we can now define this function that will return the JSON data. We will create a sample JavaScript object called hello, and in case the function was called having a parameter index part of the request it was called from the “hi/:index” handler:

 function respond(req, res, next) {
  console.log("Got HTTP " + req.method + " on " + req.url + " responding");
  var hello = [{
    'id':'0',
    'hello': 'world'
  },{
    'id':'1',
    'say':'what'
  }];
  if(req.params.index){
    var found = hello[req.params.index];
    if(found){
      res.send(found);
    } else {
      res.status(404);
      res.send();
    }
  };
  res.send(hello);
  addHeaders(req,res);
  return next();
}

The following addHeaders function that we call at the beginning is adding headers to enable access to the resources served from a different domain or a different server port:
```
function addHeaders(req, res) {
  res.header("Access-Control-Allow-Origin", "*");
  res.header("Access-Control-Allow-Headers", "X-Requested-With");
 };
```
The definition of headers and what they mean will be discussed later on in the Article. For now, let’s just say they enable accesses to the resources from a browser using AJAX. At the end, we add a block of code that will set the server to listen on port 8080:
```
server.listen(8080, function() {
  console.log('%s listening at %s', server.name, server.url);
});
```
To start the sever using command line, we type the following command:
```
node server.js
```
If everything went as it should, we will get a message in the log:
```
restify listening at http://0.0.0.0:8080
```
We can then test it by accessing directly from the browser on the URL we defined http://localhost:8080/hi

Now we can proceed with the client-side HTML and JavaScript. We will implement two ways for reading data from the server, one using standard XMLHttpRequest and the other using jQuery.get(). Note that not all features are fully compatible with all browsers.

We create a simple page where we have two div elements, one with the ID data and another with the ID say. These elements will be used as placeholders to load data form the server into them:

Hello <div id="data">loading</div>
    <hr/>
    Say <div id="say">No</div>s
    <script src = "http://ajax.googleapis.com/ajax/libs/jquery
/1.8.2/jquery.min.js"></script>
    <script src = "example.js"></script>
    <script src = "exampleJQuery.js"></script>

In the example.js file, we define a function called getData that will create a AJAX call to a given url and do a callback if the request went successfully:

  function getData(url, onSuccess) {
    var request = new XMLHttpRequest();
    request.open("GET", url);
    request.onload = function() {
      if (request.status === 200) {
        console.log(request);
        onSuccess(request.response);
      }
    };
    request.send(null);
  }

After that, we can call the function directly, but in order to demonstrate that the call happens after the page is loaded, we will call it after a timeout of three seconds:

 setTimeout(
    function() {
      getData(
        'http://localhost:8080/hi',
        function(response){
          console.log('finished getting data');
          var div = document.getElementById('data');
          var data = JSON.parse(response);
          div.innerHTML = data[0].hello;
        })
    },
    3000);

The jQuery version is a lot cleaner, as the complexity that comes with the standard DOM API and the event handling is reduced substantially:
```
    (function(){
    $.getJSON('http://localhost:8080/hi/1', function(data) {
      $('#say').text(data.say);
 });
}())
```

How it works…

At the beginning, we installed the dependency using npm install restify; this is sufficient to have it working, but in order to define dependencies in a more expressive way, npm has a way of specifying it. We can add a file called package.json, a packaging format that is mainly used for for publishing details for Node.js applications. In our case, we can define package.json with the flowing code:

{ "name" : "ch8-tip1-http-get-example", "description" : "example on http get", "dependencies" : ["restify"], "author" : "Mite Mitreski", "main" : "html5dasc", "version" : "0.0.1" }

If we have a file like this, npm will automatically handle the installation of dependencies after calling npm install from the command line in the directory where the package.json file is placed.

Restify has a simple routing where functions are mapped to appropriate methods for a given URL. The HTTP GET request for ‘/hi’ is mapped with server.get(‘hi’, theCallback), where theCallback is executed, and a response should be returned.

When we have a parameterized resource, for example in ‘hi/:index’, the value associated with :index will be available under req.params. For example, in a request to ‘/hi/john’ to access the john value, we simple have req.params.index. Additionally, the value for index will automatically get URL-decoded before it is passed to our handler. One other notable part of the request handlers in restify is the next() function that we called at the end. In our case, it mostly does not makes much sense, but in general, we are responsible for calling it if we want the next handler function in the chain to be called. For exceptional circumstances, there is also an option to call next() with an error object triggering custom responses.

When it comes to the client-side code, XMLHttpRequest is the mechanism behind the async calls, and on calling request.open(“GET”, url, true) with the last parameter value as true, we get a truly asynchronous execution. Now you might be wondering why is this parameter here, isn’t the call already done after loading the page? That is true, the call is done after loading the page, but if, for example, the parameter was set to false, the execution of the request will be a blocking method, or to put it in layman’s terms, the script will pause until we get a response. This might look like a small detail, but it can have a huge impact on performance.

The jQuery part is pretty straightforward; there is function that accepts a URL value of the resource, the data handler function, and a success function that gets called after successfully getting a response:

jQuery.getJSON( url [, data ] [, success(data, textStatus, jqXHR) ] )

When we open index.htm, the server should log something like the following:

Got HTTP GET on /hi/1 responding Got HTTP GET on /hi responding

Here one is from the jQuery request and the other from the plain JavaScript.

There’s more…

XMLHttpRequest Level 2 is one of the new improvements being added to the browsers, although not part of HTML5 it is still a significant change. There are several features with the Level 2 changes, mostly to enable working with files and data streams, but there is one simplification we already used. Earlier we would have to use onreadystatechange and go through all of the states, and if the readyState was 4, which is equal to DONE, we could read the data:

var xhr = new XMLHttpRequest();
xhr.open('GET', 'someurl', true);
xhr.onreadystatechange = function(e) {
  if (this.readyState == 4 && this.status == 200) {
   // response is loaded
  }
}

In a Level 2 request however, we can use request.onload = function() {} directly without checking states. Possible states can be seen in the table:

table

One other thing to note is that XMLHttpRequest Level 2 is supported in all major browsers and IE 10; the older XMLHttpRequest has a different way of instantiation on older versions of IE (older than IE 7), where we can access it through an ActiveX object via new ActiveXObject(“Msxml2.XMLHTTP.6.0”);.

Creating a request with custom headers

The HTTP headers are a part of the request object being sent to the server. Many of them give information about the client’s user agent setup and configuration, as that is sometimes the basis of making description for the resources being fetched from the server. Several of them such as Etag, Expires, and If-Modified-Since are closely related to caching, while others such as DNT that stands for “Do Not Track” (http://www.w3.org/2011/tracking-protection/drafts/tracking-dnt.html) can be quite controversial. In this recipe, we will take a look at a way for using the custom X-Myapp header in our server and client-side code.

Getting ready

The server will be implemented using Node.js. In this example, again for simplicity, we will use restify (http://mcavage.github.io/node-restify/). Also, monitoring the console in your browser and server is crucial in order to understand what happens in the background.

How to do it…

We can start by defining the dependencies for the server side in package.json file:

{
  "name" : "ch8-tip2-custom-headers",
  "dependencies" : ["restify"],
  "main" : "html5dasc",
  "version" : "0.0.1"
}

After that, we can call npm install from the command line that will automatically retrieve restify and place it in a node_modules folder created in the root directory of the project. After this part, we can proceed to creating the server-side code in a server.js file where we set the server to listen on port 8080 and add a route handler for ‘hi’ and for every other path when the request method is HTTP OPTIONS:
```
var restify = require('restify');
var server = restify.createServer();
server.get('hi', addHeaders, respond);
server.opts(/.*/, addHeaders, function (req, res, next) {
  console.log("Got HTTP " + req.method + " on " + req.url + " with headersn");
 res.send(200);
  return next();
});
server.listen(8080, function() {
  console.log('%s listening at %s', server.name, server.url);
});
```
In most cases, the documentation should be enough when we write the application’s build onto Restify, but sometimes, it is a good idea to take a look a the source code as well. It can be found on https://github.com/mcavage/node-restify/.
One thing to notice is that we can have multiple chained handlers; in this case, we have addHeaders before the others. In order for every handler to be propagated, next() should be called:
```
function addHeaders(req, res, next) {
  res.setHeader("Access-Control-Allow-Origin", "*");
  res.setHeader('Access-Control-Allow-Headers', 'X-Requested-With, X-Myapp');
  res.setHeader('Access-Control-Allow-Methods', 'GET, OPTIONS');
  res.setHeader('Access-Control-Expose-Headers', 'X-Myapp, X-Requested-With');
  return next();
};
```
The addHeaders adds access control options in order to enable cross-origin resource sharing. Cross-origin resource sharing (CORS ) defines a way in which the browser and server can interact to determine if the request should be allowed. It is more secure than allowing all cross-origin requests, but is more powerful than simply allowing all of them.

After this, we can create the handler function that will return a JSON response with the headers the server received and a hello world kind of object:

   function respond(req, res, next) {
  console.log("Got HTTP " + req.method + " on " + req.url + " with headersn");
  console.log("Request: ", req.headers);
  var hello = [{
    'id':'0',
    'hello': 'world',
    'headers': req.headers
  }];
  res.send(hello);
  console.log('Response:n ', res.headers());
  return next();
}

We additionally log the request and response headers to the sever console log in order to see what happens in the background.

For the client-side code, we need a plain “vanilla” JavaScript approach and jQuery method, so in order to do that, include example.js and exampleJquery.js as well as a few div elements that we will use for displaying data retrieved from the server:

     Hi <div id="data">loading</div>
    <hr/>
    Headers list from the request: <div id="headers"></div>
    <hr/>
    Data from jQuery: <div id="dataRecieved">loading</div>
    <script src = "http://ajax.googleapis.com/ajax/libs/
jquery/1.8.2/jquery.min.js"></script>
    <script src = "example.js"></script>
    <script src = "exampleJQuery.js"></script>

A simple way to add the headers is to call setRequestHeader on a XMLHttpRequest object after the call of open():

  function getData(url, onSucess) {
    var request = new XMLHttpRequest();
    request.open("GET", url, true);
    request.setRequestHeader("X-Myapp","super");
    request.setRequestHeader("X-Myapp","awesome");
    request.onload = function() {
      if (request.status === 200) {
        onSuccess(request.response);
      }
    };
    request.send(null);
  }

The XMLHttpRequest automatically sets headers, such as “Content-Length”,“Referer”, and “User-Agent”, and does not allow you to change them using JavaScript.
A more complete list of headers and the reasoning behind this can be found in the W3C documentation at http://www.w3.org/TR/XMLHttpRequest/#the-setrequestheader%28%29-method.

To print out the results, we add a function that will add each of the header keys and values to an unordered list:

  getData(
    'http://localhost:8080/hi',
    function(response){
      console.log('finished getting data');
      var data = JSON.parse(response);
      document.getElementById('data').innerHTML = data[0].hello;
      var headers = data[0].headers,
          headersList = "<ul>";
      for(var key in headers){
        headersList += '<li><b>' + key + '</b>: ' + headers[key] +'</li>';
      };
      headersList += "</ul>";
      document.getElementById('headers').innerHTML = headersList;
    });

When this gets executed. a list of all the request headers should be displayed on a page, and our custom x-myapp should be shown:

host: localhost:8080 connection: keep-alive origin: http://localhost:8000 x-myapp: super, awesome user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.27
(KHTML, like Gecko) Chrome/26.0.1386.0 Safari/537.27

The jQuery approach is far simpler, we can use the beforeSend hook to call a function that will set the ‘x-myapp’ header. When we receive the response, write it down to the element with the ID dataRecived:
```
$.ajax({
    beforeSend: function (xhr) {
      xhr.setRequestHeader('x-myapp', 'this was easy');
    },
    success: function (data) {
     $('#dataRecieved').text(data[0].headers['x-myapp']);
    }
```
Output from the jQuery example will be the data contained in x-myapp header:
```
Data from jQuery: this was easy
```

How it works…

You may have noticed that on the server side, we added a route that has a handler for HTTP OPTIONS method, but we never explicitly did a call there. If we take a look at the server log, there should be something like the following output:

Got HTTP OPTIONS on /hi with headers
Got HTTP GET on /hi with headers

This happens because the browser first issues a preflight request , which in a way is the browser’s question whether or not there is a permission to make the “real” request. Once the permission has been received, the original GET request happens. If the OPTIONS response is cached, the browser will not issue any extra preflight calls for subsequent requests.

The setRequestHeader function of XMLHttpRequest actually appends each value as a comma-separated list of values. As we called the function two times, the value for the header is as follows:

'x-myapp': 'super, awesome'

There’s more…

For most use cases, we do not need custom headers to be part of our logic, but there are plenty of API’s that make good use of them. For example, many server-side technologies add the X-Powered-By header that contains some meta information, such as JBoss 6 or PHP/5.3.0. Another example is Google Cloud Storage, where among other headers there are x-goog-meta-prefixed headers such as x-goog-meta-project-name and x-goog-meta-project-manager.

Versioning your API

We do not always have the best solution while doing the first implementation. The API can be extended up to a certain point, but afterwards needs to undergo some structural changes. But we might already have users that depend on the current version, so we need a way to have different representation versions of the same resource. Once a module has users, the API cannot be changed at our own will.

One way to resolve this issue is to use a so-called URL versioning, where we simply add a prefix. For example, if the old URL was http://example.com/rest/employees, the new one could be http://example.com/rest/v1/employees, or under subdomain it could be http://v1.example.com/rest/employee. This approach only works if you have direct control over all the servers and clients. Otherwise, you need to have a way of handling fallback to older versions.

In this recipe, we are going implement a so-called “Semantic versioning”, http://semver.org/, using HTTP headers to specify accepted versions.

Getting ready

The server will be implemented using Node.js. In this example, we will use restify (http://mcavage.github.io/node-restify/) for the server-side logic to monitor the requests to understand what is sent.

How to do it…

Let’s perform the following steps.

We need to define the dependencies first, and after installing restify, we can proceed to the creation of the server code. The main difference with the previous examples is the definition of the “Accept-version” header. restify has built-in handling for this header using versioned routes . After creating the server object, we can set which methods will get called for what version:
```
server.get({ path: "hi", version: '2.1.1'}, addHeaders, helloV2, logReqRes);
server.get({ path: "hi", version: '1.1.1'}, addHeaders, helloV1, logReqRes);
```
We also need the handler for the HTTP OPTIONS, as we are using cross-origin resource sharing and the browser needs to do the additional request in order to get permissions:
```
server.opts(/.*/, addHeaders, logReqRes, function (req, res, next) {
  res.send(200);
  return next();
});
```
The handlers for Version 1 and Version 2 will return different objects in order for us to easily notice the difference between the API calls. In the general case, the resource should be the same, but can have different structural changes. For Version 1, we can have the following:
```
function helloV1(req, res, next) {
  var hello = [{
    'id':'0',
    'hello': 'grumpy old data',
    'headers': req.headers
  }];
  res.send(hello);
  return next()
}
```

As for Version 2, we have the following:

function helloV2(req, res, next) {
  var hello = [{
    'id':'0',
    'awesome-new-feature':{
      'hello': 'awesomeness'
    },
    'headers': req.headers
  }];
  res.send(hello);
  return next();
}

One other thing we must do is add the CORS headers in order to enable the accept-version header, so in the route we included the addHeaders that should be something like the following:

function addHeaders(req, res, next) {
  res.setHeader("Access-Control-Allow-Origin", "*");
  res.setHeader('Access-Control-Allow-Headers', 
'X-Requested-With, accept-version');
  res.setHeader('Access-Control-Allow-Methods', 'GET, OPTIONS');
  res.setHeader('Access-Control-Expose-Headers', 
'X-Requested-With, accept-version');
  return next();
};

Note that you should not forget to the call to next() in order to call the next function in the route chain.

For simplicity, we will only implement the client side in jQuery, so we create a simple HTML document, where we include the necessary JavaScript dependencies:

Old api: <div id="data">loading</div>
    <hr/>
     New one: <div id="dataNew"> </div>
    <hr/>
    <script src = "http://ajax.googleapis.com/ajax/libs/jquery/
1.8.2/jquery.min.js"></script>
    <script src = "exampleJQuery.js"></script>

In the example.js file, we do two AJAX calls to our REST API, one is set to use the Version 1 and other to use Version 2:

  $.ajax({
      url: 'http://localhost:8080/hi',
      type: 'GET',
      dataType: 'json',
      success: function (data) {
      $('#data').text(data[0].hello);
    },
    beforeSend: function (xhr) {
      xhr.setRequestHeader('accept-version', '~1');
    }
  });
  $.ajax({
      url: 'http://localhost:8080/hi',
      type: 'GET',
      dataType: 'json',
      success: function (data) {
      $('#dataNew').text(data[0]['awesome-new-feature'].hello);
    },
    beforeSend: function (xhr) {
      xhr.setRequestHeader('accept-version', '~2');
    }
  });

Notice that the accept-version header contains values ~1 and ~2. These designate that all the semantic versions such as 1.1.0 and 1.1.1 1.2.1 will get matched by ~1 and similarly for ~2. At the end, we should get an output like the following text:

Old api:grumpy old data New one:awesomeness

How it works…

Versioned routes are a built-in feature of restify that work through the use of accept-version. In our example, we used Versions ~1 and ~2, but what happens if we don’t specify a version? restify will do the choice for us, as the the request will be treated in the same manner as if the client has sent a * version. The first defined matching route in our code will be used. There is also an option to set up the routes to match multiple versions by adding a list of versions for a certain handler:

server.get({path: 'hi', version: ['1.1.0',  '1.1.1',  '1.2.1']}, sendOld);

The reason why this type of versioning is very suitable for use in constantly growing applications is because as the API changes, the client can stick with their version of the API without any additional effort or changes needed in the client-side development. Meaning that we don’t have to do updates on the application. On the other hand, if the client is sure that their application will work on newer API versions, they can simply change the request headers.

There’s more…

Versioning can be implemented by using custom content types prefixed with vnd for example, application/vnd.mycompany.user-v1. An example of this is Google Earth’s content type KML where it is defined as application/vnd.google-earth.kml+xml. Notice that the content type can be in two parts; we could have application/vnd.mycompany-v1+json where the second part will be the format of the response.

Fetching JSON data with JSONP

JSONP or JSON with padding is a mechanism of making cross-domain requests by taking advantage of the <script> tag. AJAX transport is done by simply setting the src attribute on a script element or adding the element itself if not present. The browser will do an HTTP request to download the URL specified, and that is not subject to the same origin policy, meaning that we can use it to get data from servers that are not under our control. In this recipe, we will create a simple JSONP request, and a simple server to back that up.

Getting ready

We will make a simplified implementation of the server we used in previous examples, so we need Node.js and restify (http://mcavage.github.io/node-restify/) installed either via definition of package.json or a simple install. For working with Node.js.

How to do it…

First, we will create a simple route handler that will return a JSON object:

function respond(req, res, next) {
  console.log("Got HTTP " + req.method + " on " + req.url + " responding");
  var hello = [{
    'id':'0',
    'what': 'hi there stranger'
  }];
  res.send(hello);
  return next();
}

We could roll our own version that will wrap the response into a JavaScript function with the given name, but in order to enable JSONP when using restify, we can simply enable the bundled plugin. This is done by specifying what plugin to be used:
```
var server = restify.createServer();
server.use(restify.jsonp());
server.get('hi', respond);
```

After this, we just set the server to listen on port 8080:

server.listen(8080, function() {
  console.log('%s listening at %s', server.name, server.url);
});

The built-in plugin checks the request string for parameters called callback or jsonp, and if those are found, the result will be JSONP with the function name of the one passed as value to one of these parameters. For example, in our case, if we open the browser on http://localhost:8080/hi, we get the following:
```
[{"id":"0","what":"hi there stranger"}]
```
If we access the same URL with the callback parameter or a JSONP set, such as http://localhost:8080/hi?callback=great, we should receive the same data wrapped with that function name:
```
great([{"id":"0","what":"hi there stranger"}]);
```
This is where the P in JSONP, which stands for padded, comes into the picture.

So, what we need to do next is create an HTML file where we would show the data from the server and include two scripts, one for the pure JavaScript approach and another for the jQuery way:

    <b>Hello far away server: </b>
    <div id="data">loading</div>
    <hr/>
    <div id="oneMoreTime">...</div>
    <script src = "http://ajax.googleapis.com/ajax/libs/jquery
/1.8.2/jquery.min.js"></script>
    <script src = "example.js"></script>
    <script src = "exampleJQuery.js"></script>

We can proceed with the creation of example.js, where we create two functions; one will create a script element and set the value of src to http://localhost:8080/?callback=cool.run, and the other will serve as a callback upon receiving the data:

var cool = (function(){
  var module = {};

  module.run = function(data){
    document.getElementById('data').innerHTML = data[0].what;
  }

  module.addElement = function (){
    var script = document.createElement('script');
    script.src = 'http://localhost:8080/hi?callback=cool.run'
    document.getElementById('data').appendChild(script);
    return true;
  }
  return module;
}());

Afterwards we only need the function that adds the element:
```
cool.addElement();
```
This should read the data from the server and show a result similar to the following:
```
Hello far away server:
hi there stranger
```
From the cool object, we can run the addElement function directly as we defined it as self-executable.

The jQuery example is a lot simpler; We can set the datatype to JSONP and everything else is the same as any other AJAX call, at least from the API point of view:

$.ajax({
    type : "GET",
    dataType : "jsonp",
    url : 'http://localhost:8080/hi',
    success: function(obj){
      $('#oneMoreTime').text(obj[0].what);
    }
});

We can now use the standard success callback to handle the data received from the server, and we don’t have to specify the parameter in the request. jQuery will automatically append a callback parameter to the URL and delegate the call to the success callback.

How it works…

The first large leap we are doing here is trusting the source of the data. Results from the server is evaluated after the data is downloaded from the server. There has been some efforts to define a safer JSONP on http://json-p.org/, but it is far from being widespread.

The download itself is a HTTP GET method adding another major limitation to usability. Hypermedia as the Engine of Application State (HATEOAS ), among other things, defines the use of HTTP methods for the create, update, and delete operations, making JSONP very unstable for those use cases.

Another interesting point is how jQuery delegates the call to the success callback. In order to achieve this, a unique function name is created and is sent to the callback parameter, for example:

/hi?callback=jQuery182031846177391707897_1359599143721&_=1359599143727

This function later does a callback to the appropriate handler of jQuey.ajax.

There’s more…

With jQuery, we can also use a custom function if the server parameter that should handle jsonp is not called callback. This is done using the flowing config:

jsonp: false, jsonpCallback: "my callback"

As with JSONP, we don’t do XMLHttpRequest and expect any of the functions that are used with AJAX call to be executed or have their parameters filled as such call. It is a very common mistake to expect just that. More on this can be found in the jQuery documentation at http://api.jquery.com/category/ajax/.

Packt