In this article by Matthijs Kooijman, the author of the book Building Wireless Sensor Networks Using Arduino, we will explore some of the ways to persistently store the collected sensor data and visualize the data using convenient graphs.
First, you will see how to connect your coordinator to the Internet and send its data to the Beebotte cloud platform. You will learn how to create a custom dashboard in this platform that can show you the collected data in a convenient graph format.
Second, you will see how you can collect and visualize the data on your own computer instead of sending it to the Internet directly.
(For more resources related to this topic, see here.)
For the first part, you will need some shield to connect your coordinator Arduino to the Internet in addition to the hardware that has been recommended for the coordinator in the previous chapter. This article provides examples for the following two shields:
- The Arduino Ethernet shield (https://www.arduino.cc/en/Main/ArduinoEthernetShield)
- The Adafruit CC3000 Wi-Fi shield (https://www.adafruit.com/products/1491)
If possible, the Ethernet shield is recommended as its library is small, and it is easier to keep a reliable connection using a wired connection. Additionally, the CC3000 shield conflicts with the SparkFun XBee shield and this would require some modification on the latter’s part to make them work together.
For the second part, no additional hardware is needed.
There are other Ethernet or Wi-Fi shields that you can use.
Storing your data in the cloud
When it comes to storing your data online somewhere, there are literally dozens of online platforms that offer some kind of data storage service aimed at collecting sensor data. Each of these has different features, complexity, and cost, and you are encouraged to have a look at what is available.
Even though a lot of platforms are available, almost none of them are really suited for a hobby sensor network like the one presented in this article. Most platforms support the basic collection of data and offer some web API to access the data but there were two requirements that ruled out most of the platforms:
- It has to be affordable for a home user with just a bit of data. Ideally, there is a free version to get started.
- It has to support creating a dashboard that can show data and graphs and also show input elements that can be used to talk back to the network. (This will be used in the next chapter to create an online thermostat).
When this article was written, only two platforms seemed completely suitable: Beebotte (https://beebotte.com/) and Adafruit IO (https://io.adafruit.com/). The examples in this article use Beebotte because at the time of writing, Adafruit IO was not publicly available yet and Beebotte currently has some additional features. However, you are encouraged to check out Adafruit IO as an alternative. As both the platforms use the MQTT protocol (explained in the next section), you should be able to reuse the example code with just minimal changes for Adafruit IO.
Introducing Beebotte
Beebotte, like most of these services, can be seen as a big online database that stores any data you send to it and allows retrieving any data you are interested in. Additionally, you can easily create dashboards that allow you to look at your data and even interact with it through various configurable widgets. By the end of this chapter, you might have a dashboard that looks like this:
Before showing you how to talk to Beebotte from your Arduino, some important concepts in the Beebotte system will be introduced: channels, resources, security tokens, and access protocols.
The examples in this article serve to get started with Beebotte, but will certainly not cover all features and details. Be sure to check out the extensive documentation on the Beebotte site at https://beebotte.com/overview.
Channels and resources
All data collected by Beebotte is organized into resources, each representing a single series of data. All data stored in a resource signifies the same thing, such as the temperature in your living room or on/off status of the air conditioner but at different points in time. This kind of data is also often referred to as time-series data.
To keep your data organized, Beebotte supports the concept of channels. Essentially, a channel is just a group of resources that somehow belong together. Typically, a channel represents a single device or data source but you are free to group your resources in any way that you see fit.
In this example, every sensor module in the network will get its own channel, each containing a resource to store the temperature data and a resource to store the humidity data.
Security
To be able to access the data stored in your Beebotte account or publish new data, every connection needs to be authenticated. This happens using a secret token or key (similar to a password) that you configure in your Arduino code and proves to the Beebotte server that you are allowed to access the data.
There are two kinds of secrets currently supported by Beebotte:
- Your account secret key: This is a single key for your account, which allows access to all resources and all channels in your account. It additionally allows the creation and deletion of the channels and resources.
- Channel tokens: Each channel has an associated channel token, which allows reading and writing data from that channel only. Additionally, the channel token can be regenerated if the token is ever compromised.
This example uses the account secret key to authenticate the connection. It would be better to use a more limited channel token (to limit the consequences if the token is leaked) but in this example, as the coordinator forwards data for multiple sensor nodes (each of which have their own channel), a channel token does not provide enough access.
As an alternative, you could consider using a single channel containing all the resources (named, for example, “Livingroom_Temperature” to still allow grouping) so that you can use the slightly more secure channel tokens. In the future, Beebotte might also support more flexible limited tokens that support writing to more than one channel.
The examples in this article use an unencrypted connection, so make sure at least your Wi-Fi connection is encrypted (using WPA or WPA2). If you are working with particularly sensitive information, be sure to consider using SSL/TLS for the connection.
Due to limited microcontroller speeds, running SSL/TLS directly on the microcontroller does not seem feasible, so this would need external cryptographic hardware or support on the Wi-Fi / Ethernet shield that is being used. At the time of writing, there does not seem to be any shield that directly supports this but it seems that at least the ESP2866-based shields and the Arduino Yún could be made to support this and the upcoming Arduino Wi-Fi shield 101 might support it as well (but this is out of the scope for this article).
Access protocols
To store new data and access the existing data over the Internet, a few different access methods are supported by Beebotte:
- HTTP / REST: The hypertext transfer protocol (HTTP) is the protocol that powers the web. Originally, it was used to let a web browser request a web page from a server, but now HTTP is also commonly used to let all kinds of devices send and request arbitrary data (instead of webpages) to and from servers as well. In this case, the server is commonly said to export an HTTP or REST (Relational state transfer) API.
HTTP APIs are convenient, since HTTP is a very widespread protocol and HTTP libraries are available for most programming languages and environments. - WebSockets: A downside of HTTP is that it is not very convenient for sending events from the server to the client. A server can only send data to the client after the client sends a request, which means the client must poll for new events continuously.
To overcome this, the WebSockets standard was created. WebSockets is a protocol on top of HTTP that keeps a connection (socket) open indefinitely, ready for the server to send new data whenever it wants to, instead of having to wait for the client to request new data. - MQTT: The message queuing telemetry transport protocol (MQTT) is a so-called publish/subscribe (pubsub) protocol. The idea is that multiple devices can connect to a central server and each can publish a message to a given topic and also subscribe to any number of topics. Whenever a message is published to a topic, it is automatically forwarded to all the devices that have subscribed to the same topic.
MQTT, like WebSockets, keeps a connection open continuously so that both the client and server can send data at any time, making this protocol especially suitable for real-time data and events. MQTT can not be used to access historical data, though.
A lot of alternative platforms only support the HTTP access protocol, which works fine to push and access the data and would be suitable for the examples in this chapter but is less suitable to control your network from the Internet, as used in the next chapter. To prepare for this, the examples in this chapter will already use the MQTT protocol, which supports both the use cases efficiently.
Sending your data to Beebotte
Now that you have learned about some important Beebotte concepts, you are ready to send your collected sensor data to Beebotte. First, you will prepare Beebotte and your Arduino to connect to each other. Then, you will write a sketch for your coordinator to send the data. Finally, you will see how to access and visualize the stored data.
Preparing Beebotte
Before you can start sending data to Beebotte, you will have to prepare the proper channels and resources to store the data. This example uses two channels: “Livingroom” and “Study”, referring to the rooms where the sensors have been placed. You should, of course, use names that reflect your setup and adapt things if you have more or fewer sensors.
The first step is to register an account on beebotte.com. Once you have done this, you can access your Control Panel, which will initially show you an empty list of channels:
You can create a new channel by clicking the Create New channel button. In the resulting window, you can fill in a name and description for the channel and define the resources. This should look something like this:
After creating a channel for every sensor node that you have built, you have prepared Beebotte to receive all the sensor data. The next step is to modify the coordinator sketch to actually send the data.
Connecting your Arduino to the Internet
In order to let your coordinator send its data to Beebotte, it must be connected to the Internet somehow. There are plenty of shields out there that add wired Ethernet or wireless Wi-Fi connectivity to your Arduino. Wi-Fi shields are typically a lot more expensive than the Ethernet ones but the recently introduced ESP2866 cheap Wi-Fi chipset will likely change that. (However, at the time of writing, no ready-to-use Arduino shield was available yet.)
As the code to connect your Arduino to the Internet will be significantly different for each shield, every part of the code will not be discussed in this article. Instead, we will focus on the code that connects to the MQTT server and publishes the data, assuming that the Internet connection is already set up. In the code bundle for this article, two complete examples are available for use with the Arduino Ethernet shield and Adafruit CC3000 shield. These examples should be usable as a starting point to use the other hardware as well.
Some things to keep in mind when selecting your hardware have been listed, as follows:
- Check carefully for conflicting pins. For example, the Adafruit CC3000 Wi-Fi shield uses pins 2 and 3[t1] to communicate with the shield but you might be using these pins to talk to the XBee module as well (particularly, when using the SparkFun XBee shield).
- The libraries for the various Wi-Fi shields take up a lot of code space on the Arduino. For example, using the SparkFun or Adafruit CC3000 library along with the Adafruit MQTT library fills up most of the available code space on an Arduino Uno. The Ethernet library is a bit (but not much) smaller.
- It is not always easy to keep a reliable Wi-Fi connection. In theory, this is a matter of simply reconnecting when the Wi-Fi connection is failing but in practice it can be tricky to implement this completely reliably. Again, this is easier with wired Ethernet as it does not have the same disconnection issues as Wi-Fi.
- If you use a different hardware than recommended (including the recently announced Arduino Ethernet shield 2), you will likely need a different Arduino library and need to make changes to the example code provided.
Writing the sketch
To send the data to Beebotte, the Coordinator.ino sketch from the previous chapter needs to be modified. As noted before, only the MQTT code will be shown here. However, the code to establish the Internet connection is included in the full examples in the code bundle (in the Coordinator_Ethernet.ino and Coordinator_CC3000.ino sketches).
This example uses the “Adafruit MQTT library” for the MQTT protocol, which can be installed through the Arduino library manager or can be found here: https://github.com/adafruit/Adafruit_MQTT_Library. Depending on the Internet shield, you might need more libraries as well (see the comments in the example sketches). Do not forget to add the appropriate includes for the libraries that you are using. For the MQTT library, this is:
#include <Adafruit_MQTT.h>
#include <Adafruit_MQTT_Client.h>To set up the MQTT library, you first need to define some settings:
const char MQTT_SERVER[] PROGMEM = "mqtt.beebotte.com";
const char MQTT_CLIENTID[] PROGMEM = "";
const char MQTT_USERNAME[] PROGMEM = "your_key_here";
const char MQTT_PASSWORD[] PROGMEM = "";
const int MQTT_PORT = 1883;This defines the connection settings to use for the MQTT connection. These are appropriate for Beebotte; if you are using some other platform, check its documentation for the appropriate settings.
Note that here the username must be set to your account’s secret key, for example:
const char MQTT_USERNAME[] PROGMEM =
"840f626930a07c87aa315e27b22448468844edcad03fe34f551ac747533f544f";If you are using a channel token, it must be set to the token prefixed with “token:”, such as:
const char MQTT_USERNAME[] PROGMEM = "token:1438006626319_UNJoxdmKBoMFIPt7";The password is unused and can be empty, just like the client identifier. The port is just the default MQTT port number, 1883.
Note that all the string variables are marked with the PROGMEM keyword. This tells the compiler to store the string variables in program memory, just like the F() macro you have seen before does (which also uses the PROGMEM keyword underwater). However, the F() macro can only be used in the functions, which is why these variables use the PROGMEM keyword directly. This also means that the extra checking offered by the F() macro is not available, so be careful not to mix up the normal and PROGMEM strings here as this will not result in any compiler error and instead things will be broken when you run the code.
Using the configuration constants defined earlier, you can now define the main mqtt object:
Adafruit_MQTT_Client mqtt(&client, MQTT_SERVER, MQTT_PORT, MQTT_CLIENTID,
MQTT_USERNAME, MQTT_PASSWORD);There are a few different flavors of this object. For example, there are flavors optimized for the specific hardware and corresponding libraries, and there is also a generic flavor that works with any hardware whose library exposes a generic Client object (like most libraries currently do). The latter flavor, Adafruit_MQTT_Client, is used in this example and should be fine.
The &client [t2] part of this line refers to a previously created Client object (not shown here as it depends on the Internet shield used), which is used by the MQTT library to set up the MQTT connection.
To actually connect to the MQTT server, a function called connect() will be defined. This function is called to connect once at startup and to reconnect every time when the publishing of some data fails.
In the CC3300 version, this function associates to the WiFi access point and then sets up the connection to the MQTT server. On the Ethernet version, where the network is always available after initial initialization, the connect() function only sets up the MQTT connection. The latter version is shown as follows:
void connect() {
client.stop(); // Ensure any old connection is closed
uint8_t ret = mqtt.connect();
if (ret == 0)
DebugSerial.println(F("MQTT connected"));
else
DebugSerial.println(mqtt.connectErrorString(ret));
}This calls mqtt.connect() to connect to the MQTT server and writes a debug message to report the success or failure. Note that mqtt.connect() returns a number as an error code (with 0 meaning OK), which is translated to a human readable error message using the mqtt.connectErrorString() function.
Now, to actually publish a single value, there is the publish() function:
void publish(const __FlashStringHelper *resource, float value) {
// Use JSON to wrap the data, so Beebotte will remember the data
// (instead of just publishing it to whoever is currently listening).
String data;
data += "{"data": ";
data += value;
data += ", "write": true}";
DebugSerial.print(F("Publishing "));
DebugSerial.print(data);
DebugSerial.print(F( " to "));
DebugSerial.println(resource);
// Publish data and try to reconnect when publishing data fails
if (!mqtt.publish(resource, data.c_str())) {
DebugSerial.println(F("Failed to publish, trying reconnect..."));
connect();
if (!mqtt.publish(resource, data.c_str()))
DebugSerial.println(F("Still failed to publish data"));
}
}This function takes two parameters: the name of the resource to publish to and the value to publish. Note the type of the resource parameter: __FlashStringHelper* is similar to the more common char* string type but indicates that the string is stored in the program memory instead of RAM. This is also the type returned by the F() macro that you have seen before. Just like the MQTT server configuration values that used the PROGMEM keyword before, the MQTT library also expects the MQTT topic names to be stored in the program memory.
The actual value is sent using the JSON (JavaScript object notation) format. For example, for a temperature of 20 degrees, it constructs {“data”: 20.00, “write”: true}. This format allows, in addition to transmitting the value, indicating that Beebotte should store the value so that it can be retrieved later. If the write value is false, or not present, Beebotte will only forward the value to any other devices currently subscribed to the appropriate topic without saving it for later. This example uses some quick and dirty string concatenation to generate the JSON. If you want something more robust and elegant, have a look at the ArduinoJson library at https://github.com/bblanchon/ArduinoJson.
If publishing the data fails, it is likely that the WiFi or MQTT connection has failed and so it attempts to reconnect and publish the data once more.
As before, there is a processRxPacket() function that gets called when a radio packet is received through the XBee module:
void processRxPacket(ZBRxResponse& rx, uintptr_t) {
Buffer b(rx.getData(), rx.getDataLength());
uint8_t type = b.remove<uint8_t>();
XBeeAddress64 addr = rx.getRemoteAddress64();
if (addr == 0x0013A20040DADEE0 && type == 1 && b.len() == 8) {
publish(F("Livingroom/Temperature"), b.remove<float>());
publish(F("Livingroom/Humidity"), b.remove<float>());
return;
}
if (addr == 0x0013A20040E2C832 && type == 1 && b.len() == 8) {
publish(F("Study/Temperature"), b.remove<float>());
publish(F("Study/Humidity"), b.remove<float>());
return;
}
DebugSerial.println(F("Unknown or invalid packet"));
printResponse(rx, DebugSerial);
}Instead of simply printing the packet contents as before, it figures out who the sender of the packet is and which Beebotte resource corresponds to it and calls the publish() function that was defined earlier.
As you can see, the Beebotte resources are identified using “Channel/Resource”, resulting in a unique identifier for each resource (which is later used in the MQTT message as the topic identifier). Note that the F() macro is used for the resource names to store them in program memory as this is what publish() and the MQTT library expect.
If you run the resulting sketch and everything connects correctly, the coordinator will forward any sensor values that it receives to the Beebotte server. If you wait for (at most) five minutes to pass (or reset the sensor Arduino to have it send a reading right away) and then go to the appropriate channel in your Beebotte control panel, it should look something like this:
Visualizing your data
To easily allow the visualizing of your data, Beebotte supports dashboards. A dashboard is essentially a web page where you can add graphs, gauges, tables, buttons, and so on (collectively called widgets). These widgets can then display or control the data in one or more previously defined resources.
To create such a dashboard, head over to the My Dashboards section of your control panel and click Create Dashboard to start building one.
Once you set a name for the dashboard, you can start adding widgets to it. To display the temperature and humidity for all the sensors that you are using, you could use the Multi-line Chart widget. As the temperature and humidity values will be fairly far apart, it makes sense to put them each in a separate chart. Adding the temperature chart could look like this:
If you also add a chart for the humidity, it should look like this:
Here, only the living room sensor has been powered on, so no data is shown for the study yet. Also, to make the graphs a bit more interesting, some warm and humid air was breathed onto the sensor, causing the big spike in both the charts.
There are plenty of other widget types that will prove useful to you. The Beebotte documentation provides information on the supported types, but you are encouraged to just play with the widgets a bit to see what they can add to your project. In the next chapter, you will see how to use the control widgets, which allow sending events and data back to the coordinator to control it.
Accessing your data
You have been accessing your data through a Beebotte dashboard so far. However, when these dashboards are not powerful enough for your needs or you want to access your data from an existing application, you can also access the recent and historical data through the Beebotte’s HTTP or WebSocket API’s. This gives you full control over the processing and display of the data without being limited in any way by what Beebotte offers in its dashboards.
As creating a custom (web) application is out of the scope of this article, the HTTP and WebSocket API will not be discussed in detail. Instead, you should be able to find extensive documentation on this API on the Beebotte site at https://beebotte.com/overview.
Summary
In this article we learnt some of the ways to store the collected sensor data and visualize the data using convenient graphs. We also learnt as to how to save our data on the Cloud.
Resources for Article:
Further resources on this subject:
- TV Set Constant Volume Controller[article]
- Internet Connected Smart Water Meter[article]
- Getting Started with Arduino [article]