9 min read

(For more resources related to this topic, see here.)

Examining a JSON file with the aeson package

JavaScript Object Notation (JSON) is a way to represent key-value pairs in plain text. The format is described extensively in RFC 4627 (http://www.ietf.org/rfc/rfc4627).

In this recipe, we will parse a JSON description about a person. We often encounter JSON in APIs from web applications.

Getting ready

Install the aeson library from hackage using Cabal.

Prepare an input.json file representing data about a mathematician, such as the one in the following code snippet:

$ cat input.json

{"name":"Gauss", "nationality":"German", "born":1777, "died":1855}

We will be parsing this JSON and representing it as a usable data type in Haskell.

How to do it…

  1. Use the OverloadedStrings language extension to represent strings as ByteString, as shown in the following line of code:

    {-# LANGUAGE OverloadedStrings #-}

  2. Import aeson as well as some helper functions as follows:

    import Data.Aeson
import Control.Applicative
import qualified Data.ByteString.Lazy as B

  3. Create the data type corresponding to the JSON structure, as shown in the following code:

    data Mathematician = Mathematician 
                     { name :: String
                     , nationality :: String
                     , born :: Int
                     , died :: Maybe Int
                     }

  4. Provide an instance for the parseJSON function, as shown in the following code snippet:

    instance FromJSON Mathematician where
  parseJSON (Object v) = Mathematician
                         <$> (v .: "name")
                         <*> (v .: "nationality")
                         <*> (v .: "born")
                         <*> (v .:? "died")

  5. Define and implement main as follows:

    main :: IO ()
main = do

  6. Read the input and decode the JSON, as shown in the following code snippet:

      input <- B.readFile "input.json"

  let mm = decode input :: Maybe Mathematician

  case mm of
    Nothing -> print "error parsing JSON"
    Just m -> (putStrLn.greet) m

  7. Now we will do something interesting with the data as follows:

    greet m = (show.name) m ++ 
          " was born in the year " ++ 
          (show.born) m

  8. We can run the code to see the following output:

    $ runhaskell Main.hs

"Gauss" was born in the year 1777

How it works…

Aeson takes care of the complications in representing JSON. It creates native usable data out of a structured text. In this recipe, we use the .: and .:? functions provided by the Data.Aeson module.

As the Aeson package uses ByteStrings instead of Strings, it is very helpful to tell the compiler that characters between quotation marks should be treated as the proper data type. This is done in the first line of the code which invokes the OverloadedStrings language extension.

We use the decode function provided by Aeson to transform a string into a data type. It has the type FromJSON a => B.ByteString -> Maybe a. Our Mathematician data type must implement an instance of the FromJSON typeclass to properly use this function. Fortunately, the only required function for implementing FromJSON is parseJSON. The syntax used in this recipe for implementing parseJSON is a little strange, but this is because we’re leveraging applicative functions and lenses, which are more advanced Haskell topics.

The .: function has two arguments, Object and Text, and returns a Parser a data type. As per the documentation, it retrieves the value associated with the given key of an object. This function is used if the key and the value exist in the JSON document. The 😕 function also retrieves the associated value from the given key of an object, but the existence of the key and value are not mandatory. So, we use .:? for optional key value pairs in a JSON document.

There’s more…

If the implementation of the FromJSON typeclass is too involved, we can easily let GHC automatically fill it out using the DeriveGeneric language extension. The following is a simpler rewrite of the code:

{-# LANGUAGE OverloadedStrings #-}
{-# LANGUAGE DeriveGeneric #-}
import Data.Aeson
import qualified Data.ByteString.Lazy as B
import GHC.Generics

data Mathematician = Mathematician { name :: String
                                   , nationality :: String
                                   , born :: Int
                                   , died :: Maybe Int
                                   } deriving Generic

instance FromJSON Mathematician

main = do
  input <- B.readFile "input.json"
  let mm = decode input :: Maybe Mathematician
  case mm of
    Nothing -> print "error parsing JSON"
    Just m -> (putStrLn.greet) m
    
greet m = (show.name) m ++" was born in the year "++ (show.born) m

Although Aeson is powerful and generalizable, it may be an overkill for some simple JSON interactions. Alternatively, if we wish to use a very minimal JSON parser and printer, we can use Yocto, which can be downloaded from http://hackage.haskell.org/package/yocto.

Reading an XML file using the HXT package

Extensible Markup Language (XML) is an encoding of plain text to provide machine-readable annotations on a document. The standard is specified by W3C (http://www.w3.org/TR/2008/REC-xml-20081126/).

In this recipe, we will parse an XML document representing an e-mail conversation and extract all the dates.

Getting ready

We will first set up an XML file called input.xml with the following values, representing an e-mail thread between Databender and Princess on December 18, 2014 as follows:

$ cat input.xml

<thread>
    <email>
        <to>Databender</to>
        <from>Princess</from>
        <date>Thu Dec 18 15:03:23 EST 2014</date>
        <subject>Joke</subject>
        <body>Why did you divide sin by tan?</body>
    </email>
    <email>
        <to>Princess</to>
        <from>Databender</from>
        <date>Fri Dec 19 3:12:00 EST 2014</date>
        <subject>RE: Joke</subject>
        <body>Just cos.</body>
    </email>
</thread>

Using Cabal, install the HXT library which we use for manipulating XML documents:

$ cabal install hxt

How to do it…

  1. We only need one import, which will be for parsing XML, using the following line of code:

    import Text.XML.HXT.Core

  2. Define and implement main and specify the XML location. For this recipe, the file is retrieved from input.xml. Refer to the following code:

    main :: IO ()
main = do
    input <- readFile "input.xml"

  3. Apply the readString function to the input and extract all the date documents. We filter items with a specific name using the hasName :: String -> a XmlTree XmlTree function. Also, we extract the text using the getText :: a XmlTree String function, as shown in the following code snippet:

        dates <- runX $ readString [withValidate no] input 
        //> hasName "date" 
        //> getText

  4. We can now use the list of extracted dates as follows:

        print dates

  5. By running the code, we print the following output:

     $ runhaskell Main.hs

["Thu Dec 18 15:03:23 EST 2014", "Fri Dec 19 3:12:00 EST 2014"]

How it works…

The library function, runX, takes in an Arrow. Think of an Arrow as a more powerful version of a Monad. Arrows allow for stateful global XML processing. Specifically, the runX function in this recipe takes in IOSArrow XmlTree String and returns an IO action of the String type. We generate this IOSArrow object using the readString function, which performs a series of operations to the XML data.

For a deep insight into the XML document, //> should be used whereas /> only looks at the current level. We use the //> function to look up the date attributes and display all the associated text.

As defined in the documentation, the hasName function tests whether a node has a specific name, and the getText function selects the text of a text node. Some other functions include the following:

  • isText: This is used to test for text nodes

  • isAttr: This is used to test for an attribute tree

  • hasAttr: This is used to test whether an element node has an attribute node with a specific name

  • getElemName: This is used to select the name of an element node

All the Arrow functions can be found on the Text.XML.HXT.Arrow.XmlArrow documentation at http://hackage.haskell.org/package/hxt/docs/Text-XML-HXT-Arrow-XmlArrow.html.

Capturing table rows from an HTML page

Mining Hypertext Markup Language (HTML) is often a feat of identifying and parsing only its structured segments. Not all text in an HTML file may be useful, so we find ourselves only focusing on a specific subset. For instance, HTML tables and lists provide a strong and commonly used structure to extract data whereas a paragraph in an article may be too unstructured and complicated to process.

In this recipe, we will find a table on a web page and gather all rows to be used in the program.

Getting ready

We will be extracting the values from an HTML table, so start by creating an input.html file containing a table as shown in the following figure:

The HTML behind this table is as follows:

$ cat input.html

<!DOCTYPE html>
<html>
    <body>
        <h1>Course Listing</h1>
        <table>
            <tr>
                <th>Course</th>
                <th>Time</th>
                <th>Capacity</th>
            </tr>
            <tr>
                <td>CS 1501</td>
                <td>17:00</td>
                <td>60</td>
            </tr>
            <tr>
                <td>MATH 7600</td>
                <td>14:00</td>
                <td>25</td>
            </tr>
            <tr>
                <td>PHIL 1000</td>
                <td>9:30</td>
                <td>120</td>
            </tr>
        </table>
    </body>
</html>

If not already installed, use Cabal to set up the HXT library and the split library, as shown in the following command lines:

$ cabal install hxt
$ cabal install split

How to do it…

  1. We will need the htx package for XML manipulations and the chunksOf function from the split package, as presented in the following code snippet:

    import Text.XML.HXT.Core
import Data.List.Split (chunksOf)

  2. Define and implement main to read the input.html file.

    main :: IO ()
main = do
  input <- readFile "input.html"

  3. Feed the HTML data into readString, thereby setting withParseHTML to yes and optionally turning off warnings. Extract all the td tags and obtain the remaining text, as shown in the following code:

      texts <- runX $ readString 
           [withParseHTML yes, withWarnings no] input 
    //> hasName "td"
    //> getText

  4. The data is now usable as a list of strings. It can be converted into a list of lists similar to how CSV was presented in the previous CSV recipe, as shown in the following code:

      let rows = chunksOf 3 texts
  print $ findBiggest rows

  5. By folding through the data, identify the course with the largest capacity using the following code snippet:

    findBiggest :: [[String]] -> [String]
findBiggest [] = []
findBiggest items = foldl1 
                    (a x -> if capacity x > capacity a 
                             then x else a) items

capacity [a,b,c] = toInt c
capacity _ = -1

toInt :: String -> Int
toInt = read

  6. Running the code will display the class with the largest capacity as follows:

    $ runhaskell Main.hs

{"PHIL 1000", "9:30", "120"}

How it works…

This is very similar to XML parsing, except we adjust the options of readString to [withParseHTML yes, withWarnings no].

Packt

LEAVE A REPLY

Please enter your comment!
Please enter your name here