Getting Twitter data

In this article by Paulo A Pereira, the author of Elixir Cookbook, we will build an application that will query the Twitter timeline for a given word and will display any new tweet with that keyword in real time. We will be using an Elixir twitter client extwitter as well as an Erlang application to deal with OAuth. We will wrap all in a phoenix web application.

(For more resources related to this topic, see here.)

Getting ready

Before getting started, we need to register a new application with Twitter to get the API keys that will allow the authentication and use of Twitter’s API. To do this, we will go to https://apps.twitter.com and click on the Create New App button. After following the steps, we will have access to four items that we need: consumer_key, consumer_secret, access_token, and access_token_secret.

These values can be used directly in the application or setup as environment variables in an initialization file for bash or zsh (if using Unix).

After getting the keys, we are ready to start building the application.

How to do it…

To begin with building the application, we need to follow these steps:

1. Create a new Phoenix application:

   > mix phoenix.new phoenix_twitter_stream code/phoenix_twitter_stream

2. Add the dependencies in the mix.exs file:

   defp deps do
     [
       {:phoenix, "~> 0.8.0"},
       {:cowboy, "~> 1.0"},
       {:oauth, github: "tim/erlang-oauth"},
       {:extwitter, "~> 0.1"}
     ]
   end

3. Get the dependencies and compile them:

   > mix deps.get && mix deps.compile

4. Configure the application to use the Twitter API keys by adding the configuration block with the keys we got from Twitter in the Getting ready section of this article. Edit lib/phoenix_twitter_stream.ex so that it looks like this:

   defmodule PhoenixTweeterStream do
     use Application
     def start(_type, _args) do
       import Supervisor.Spec, warn: false
       ExTwitter.configure(
         consumer_key: System.get_env("SMM_TWITTER_CONSUMER_KEY"),
         consumer_secret: System.get_env("SMM_TWITTER_CONSUMER_SECRET"),
         access_token: System.get_env("SMM_TWITTER_ACCESS_TOKEN"),
         access_token_secret: System.get_env("SMM_TWITTER_ACCESS_TOKEN_SECRET")
       )
       children = [
         # Start the endpoint when the application starts
         worker(PhoenixTweeterStream.Endpoint, []),
         # Here you could define other workers and supervisors as children
         # worker(PhoenixTweeterStream.Worker, [arg1, arg2, arg3]),
       ]
       opts = [strategy: :one_for_one, name: PhoenixTweeterStream.Supervisor]
       Supervisor.start_link(children, opts)
     end
     def config_change(changed, _new, removed) do
       PhoenixTweeterStream.Endpoint.config_change(changed, removed)
       :ok
     end
   end

   In this case, the keys are stored as environment variables, so we use the System.get_env function:

   System.get_env("SMM_TWITTER_CONSUMER_KEY") (…)

   If you don’t want to set the keys as environment variables, the keys can be directly declared as strings this way:

   consumer_key: "this-is-an-example-key" (…)

5. Define a module that will handle the query for new tweets in the lib/phoenix_twitter_stream/tweet_streamer.ex file, and add the following code:

   defmodule PhoenixTwitterStream.TweetStreamer do
     def start(socket, query) do
       stream = ExTwitter.stream_filter(track: query)
       for tweet <- stream do
         Phoenix.Channel.reply(socket, "tweet:stream", tweet)
       end
     end
   end

6. Create the channel that will handle the tweets in the web/channels/tweets.ex file:

   defmodule PhoenixTwitterStream.Channels.Tweets do
     use Phoenix.Channel
     alias PhoenixTwitterStream.TweetStreamer
     def join("tweets", %{"track" => query}, socket) do
       spawn(fn() -> TweetStreamer.start(socket, query) end)
       {:ok, socket}
     end 
   end

7. Edit the application router (/web/router.ex) to register the websocket handler and the tweets channel. The file will look like this:

   defmodule PhoenixTwitterStream.Router do
     use Phoenix.Router
     pipeline :browser do
       plug :accepts, ~w(html)
       plug :fetch_session
       plug :fetch_flash
       plug :protect_from_forgery
     end
     pipeline :api do
       plug :accepts, ~w(json)
     end
     socket "/ws" do
       channel "tweets", PhoenixTwitterStream.Channels.Tweets
     end
     scope "/", PhoenixTwitterStream do
       pipe_through :browser # Use the default browser stack
       get "/", PageController, :index
     end
   end

8. Replace the index template (web/templates/page/index.html.eex) content with this:

   <div class="row">
     <div class="col-lg-12">
       <ul id="tweets"></ul>
     </div>
     <script src="/js/phoenix.js" type="text/javascript"></script>
     <script src="https://code.jquery.com/jquery-2.1.1.js" type="text/javascript"></script>
     <script type="text/javascript">
       var my_track = "programming";
       var socket = new Phoenix.Socket("ws://" + location.host + "/ws");
       socket.join("tweets", {track: my_track}, function(chan){
         chan.on("tweet:stream", function(message){
           console.log(message);
           $('#tweets').prepend($('<li>').text(message.text));
           });
       });
     </script>
   </div>

9. Start the application:

   > mix phoenix.server

10. Go to http://localhost:4000/ and after a few seconds, tweets should start arriving and the page will be updated to display every new tweet at the top.
    

How it works…

We start by creating a Phoenix application. We could have created a simple application to output the tweets in the console. However, Phoenix is a great choice for our purposes, displaying a web page with tweets getting updated in real time via websockets!

In step 2, we add the dependencies needed to work with the Twitter API. We use parroty’s extwitter Elixir application (https://hex.pm/packages/extwitter) and Tim’s erlang-oauth application (https://github.com/tim/erlang-oauth/). After getting the dependencies and compiling them, we add the Twitter API keys to our application (step 4). These keys will be used to authenticate against Twitter where we previously registered our application.

In step 5, we define a function that, when started, will query Twitter for any tweets containing a specific query.

The stream = ExTwitter.stream_filter(track: query) line defines a stream that is returned by the ExTwitter application and is the result of filtering Twitter’s timeline, extracting only the entries (tracks) that contain the defined query.

The next line, which is for tweet <- stream do Phoenix.Channel.reply(socket, “tweet:stream”, tweet), is a stream comprehension. For every new entry in the stream defined previously, send the entry through a Phoenix channel.

Step 6 is where we define the channel. This channel is like a websocket handler. Actually, we define a join function:

 def join(socket, "stream", %{"track" => query}) do
   reply socket, "join", %{status: "connected"}
   spawn(fn() -> TweetStreamer.start(socket, query) end)
   {:ok, socket}
 end

It is here, when the websocket connection is performed, that we initialize the module defined in step 5 in the spawn call. This function receives a query string defined in the frontend code as track and passes that string to ExTwitter, which will use it as the filter.

In step 7, we register and mount the websocket handler in the router using use Phoenix.Router.Socket, mount: “/ws”, and we define the channel and its handler module using channel “tweets”, PhoenixTwitterStream.Channels.Tweets.

The channel definition must occur outside any scope definition!

If we tried to define it, say, right before get “/”, PageController, :index, the compiler would issue an error message and the application wouldn’t even start.

The last code we need to add is related to the frontend. In step 8, we mix HTML and JavaScript on the same file that will be responsible for displaying the root page and establishing the websocket connection with the server. We use a phoenix.js library helper (<script src=”/js/phoenix.js” type=”text/javascript”></script>), providing some functions to deal with Phoenix websockets and channels.

We will take a closer look at some of the code in the frontend:

// initializes the query … in this case filter the timeline for
// all tweets containing "programming" 
var my_track = "programming";
// initialize the websocket connection. The endpoint is /ws.  //(we already have registered with the phoenix router on step 7)
var socket = new Phoenix.Socket("ws://" + location.host + "/ws");
// in here we join the channel 'tweets'
// this code triggers the join function we saw on step 6
// when a new tweet arrives from the server via websocket
// connection it is prepended to the existing tweets in the page
socket.join("tweets", "stream", {track: my_track}, function(chan){
      chan.on("tweet:stream", function(message){
        $('#tweets').prepend($('<li>').text(message.text));
        });
    });

There’s more…

If you wish to see the page getting updated really fast, select a more popular word for the query.

Summary

In this article, we looked at how we can use extwitter to query Twitter for relevant tweets.

Resources for Article:

Further resources on this subject:

• NMAP Fundamentals [article]
• Api With Mongodb And Node.JS [article]
• Creating a Restful Api [article]