In this tutorial, we will build a sample internet of things application using Google Cloud IoT.
We will start off by implementing the end-to-end solution, where we take the data from the DHT11 sensor and post it to the Google IoT Core state topic.
This article is an excerpt from the book, Enterprise Internet of Things Handbook, written by Arvind Ravulavaru.
To get started with Google IoT Core, we need to have a Google account. If you do not have a Google account, you can create one by navigating to this URL: https://accounts.google.com/SignUp?hl=en.
Once you have created your account, you can login and navigate to Google Cloud Console: https://console.cloud.google.com.
The first thing we are going to do is create a project. If you have already worked with Google Cloud Platform and have at least one project, you will be taken to the first project in the list or you will be taken to the Getting started page.
As of the time of writing this book, Google Cloud Platform has a free trial for 12 months with $300 if the offer is still available when you are reading this chapter, I would highly recommend signing up:
Following are the steps to be followed for enabling APIs:
The following steps should be used for enabling device registry and devices:
Field | Value |
Registry ID | Pi3-DHT11-Nodes |
Cloud region | us-central1 |
Protocol | MQTT HTTP |
Default telemetry topic | device-events |
Default state topic | dht11 |
We will add the required certificates later on.
openssl req -x509 -newkey rsa:2048 -keyout rsa_private.pem -nodes -out rsa_cert.pem -subj "/CN=unused"
Now that the public key has been successfully added, we can connect to the cloud using the private key.
Now that we have our device set up in Google IoT Core, we are going to complete the remaining operation on Raspberry Pi 3 to send data.
The requirements for setting up Raspberry Pi 3 on a DHT11 node are:
The following steps are to be used for the setup process:
Refer to the following steps to install Node.js:
$ sudo apt update $ sudo apt full-upgrade
$ curl -sL https://deb.nodesource.com/setup_7.x | sudo -E bash - $ sudo apt install nodejs
$ node -v $ npm -v
Now, we will set up the app and write the required code:
$ npm init -y
$ npm install jsonwebtoken mqtt--save
$ npm install rpi-dht-sensor --save
{ "name": "Google-IoT-Device", "version": "1.0.0", "description": "", "main": "index.js", "scripts": { "test": "echo "Error: no test specified" && exit 1" }, "keywords": [], "author": "", "license": "ISC", "dependencies": { "jsonwebtoken": "^8.1.1", "mqtt": "^2.15.3", "rpi-dht-sensor": "^0.1.1" } }
var fs = require('fs'); var jwt = require('jsonwebtoken'); var mqtt = require('mqtt'); var rpiDhtSensor = require('rpi-dht-sensor'); var dht = new rpiDhtSensor.DHT11(2); // `2` => GPIO2 var projectId = 'pi-iot-project'; var cloudRegion = 'us-central1'; var registryId = 'Pi3-DHT11-Nodes'; var deviceId = 'Pi3-DHT11-Node'; var mqttHost = 'mqtt.googleapis.com'; var mqttPort = 8883; var privateKeyFile = '../certs/rsa_private.pem'; var algorithm = 'RS256'; var messageType = 'state'; // or event var mqttClientId = 'projects/' + projectId + '/locations/' + cloudRegion + '/registries/' + registryId + '/devices/' + deviceId; var mqttTopic = '/devices/' + deviceId + '/' + messageType; var connectionArgs = { host: mqttHost, port: mqttPort, clientId: mqttClientId, username: 'unused', password: createJwt(projectId, privateKeyFile, algorithm), protocol: 'mqtts', secureProtocol: 'TLSv1_2_method' }; console.log('connecting...'); var client = mqtt.connect(connectionArgs); // Subscribe to the /devices/{device-id}/config topic to receive config updates. client.subscribe('/devices/' + deviceId + '/config'); client.on('connect', function(success) { if (success) { console.log('Client connected...'); sendData(); } else { console.log('Client not connected...'); } }); client.on('close', function() { console.log('close'); }); client.on('error', function(err) { console.log('error', err); }); client.on('message', function(topic, message, packet) { console.log(topic, 'message received: ', Buffer.from(message, 'base64').toString('ascii')); }); function createJwt(projectId, privateKeyFile, algorithm) { var token = { 'iat': parseInt(Date.now() / 1000), 'exp': parseInt(Date.now() / 1000) + 86400 * 60, // 1 day 'aud': projectId }; var privateKey = fs.readFileSync(privateKeyFile); return jwt.sign(token, privateKey, { algorithm: algorithm }); } function fetchData() { var readout = dht.read(); var temp = readout.temperature.toFixed(2); var humd = readout.humidity.toFixed(2); return { 'temp': temp, 'humd': humd, 'time': new Date().toISOString().slice(0, 19).replace('T', ' ') // https://stackoverflow.com/a/11150727/1015046 }; } function sendData() { var payload = fetchData(); payload = JSON.stringify(payload); console.log(mqttTopic, ': Publishing message:', payload); client.publish(mqttTopic, payload, { qos: 1 }); console.log('Transmitting in 30 seconds'); setTimeout(sendData, 30000); }
In the previous code, we first define the projectId, cloudRegion, registryId, and deviceId based on what we have created. Next, we build the connectionArgs object, using which we are going to connect to Google IoT Core using MQTT-SN. Do note that the password property is a JSON Web Token (JWT), based on the projectId and privateKeyFile algorithm.
The username value is the Common Name (CN) of the certificate we have created, which is unused.
Using mqtt.connect(), we are going to connect to the Google IoT Core. And we are subscribing to the device config topic, which can be used to send device configurations when connected.
Once the connection is successful, we callsendData() every 30 seconds to send data to the state topic.
Save the previous file and run the following command:
$ sudo node index.js
And we should see something like this:
As you can see from the previous Terminal logs, the device first gets connected then starts transmitting the temperature and humidity along with time. We are sending time as well, so we can save it in the BigQuery table and then build a time series chart quite easily.
Now, if we head back to the Device page of Google IoT Core and navigate to the Configuration & state history tab, we should see the data that we are sending to the state topic here:
Now that the device is sending data, let’s actually read the data from another client.
For this, you can either use the same Raspberry Pi 3 or another computer. I am going to use MacBook as a client that is interested in the data sent by the Thing.
Before we start reading data from Google IoT Core, we have to set up our computer (for example, MacBook) as a trusted device, so our computer can request data. Let’s perform the following steps to set the credentials:
Do not share this file; this file is as good as giving someone owner-level permissions to all assets of this project.
The data from the device is being sent to Google IoT Core using the state topic. If you recall, we have named that topic dht11. Now, we are going to create a subscription for this topic:
Now that we have provided the required credentials as well as subscribed to a Pub/Sub topic, we will set up the Pub/Sub client. Follow these steps:
$ npm init -y
$ npm install @google-cloud/pubsub --save
var PubSub = require('@google-cloud/pubsub'); var projectId = 'pi-iot-project'; var stateSubscriber = 'dht11-data' // Instantiates a client var pubsub = new PubSub({ projectId: projectId, }); var subscription = pubsub.subscription('projects/' + projectId + '/subscriptions/' + stateSubscriber); var messageHandler = function(message) { console.log('Message Begin >>>>>>>>'); console.log('message.connectionId', message.connectionId); console.log('message.attributes', message.attributes); console.log('message.data', Buffer.from(message.data, 'base64').toString('ascii')); console.log('Message End >>>>>>>>>>'); // "Ack" (acknowledge receipt of) the message message.ack(); }; // Listen for new messages subscription.on('message', messageHandler);
$ node index.js
This way, any client that is interested in the data of this device can use this approach to get the latest data.
With this, we conclude the section on posting data to Google IoT Core and fetching the data. In the next section, we are going to work on building a dashboard.
Now that we have seen how a client can read the data from our device on demand, we will move on to building a dashboard, where we display data in real time.
For this, we are going to use Google Cloud Functions, Google BigQuery, and Google Data Studio.
Cloud Functions are solution for serverless services. Cloud Functions is a lightweight solution for creating standalone and single-purpose functions that respond to cloud events.
You can read more about Google Cloud Functions at https://cloud.google.com/functions/.
Google BigQuery is an enterprise data warehouse that solves this problem by enabling super-fast SQL queries using the processing power of Google’s infrastructure.
You can read more about Google BigQuery at https://cloud.google.com/bigquery/.
Google Data Studio helps to build dashboards and reports using various data connectors, such as BigQuery or Google Analytics.
You can read more about Google Data Studio at https://cloud.google.com/data-studio/.
As we have already seen in the Architecture section, once the data is published on the state topic, we are going to create a cloud function that will get triggered by the data event on the Pub/Sub client. And inside our cloud function, we are going to get a copy of the published data and then insert it into the BigQuery dataset.
Once the data is inserted, we are going to use Google Data Studio to create a new report by linking the BigQuery dataset to the input.
So, let’s get started.
The first thing we are going to do is set up BigQuery:
Now that we have our table ready, we will write a cloud function to insert the incoming data from Pub/Sub into this table.
Now, we are going to set up a cloud function that will be triggered by the incoming data:
var BigQuery = require('@google-cloud/bigquery'); var projectId = 'pi-iot-project'; var bigquery = new BigQuery({ projectId: projectId, }); var datasetName = 'pi3_dht11_dataset'; var tableName = 'dht11_data'; exports.pubsubToBQ = function(event, callback) { var msg = event.data; var data = JSON.parse(Buffer.from(msg.data, 'base64').toString()); // console.log(data); bigquery .dataset(datasetName) .table(tableName) .insert(data) .then(function() { console.log('Inserted rows'); callback(); // task done }) .catch(function(err) { if (err && err.name === 'PartialFailureError') { if (err.errors && err.errors.length > 0) { console.log('Insert errors:'); err.errors.forEach(function(err) { console.error(err); }); } } else { console.error('ERROR:', err); } callback(); // task done }); };
{ "name": "cloud_function", "version": "0.0.1", "dependencies": { "@google-cloud/bigquery": "^1.0.0" } }
Finally, for the Function to execute field, enter pubsubToBQ. pubsubToBQ is the name of the function that has our logic and this function will be called when the data event occurs.
Click on the Create button and our function should be deployed in a minute.
Now that the entire setup is done, we will start pumping data into BigQuery:
Or we can go one step ahead and use the Google Data Studio to do the same.
Now that the data is ready in BigQuery, we are going to set up Google Data Studio and then connect both of them, so we can access the data from BigQuery in Google Data Studio:
You can play with the styles as per your preference and we should see something similar to the following screenshot:
This report can then be shared with any user. With this, we have seen the basic features and implementation process needed to work with Google Cloud IoT Core as well other features of the platform.
If you found this post useful, do check out the book, Enterprise Internet of Things Handbook, to build state of the art IoT applications best-fit for Enterprises.
Cognitive IoT: How Artificial Intelligence is remoulding Industrial and Consumer IoT
I remember deciding to pursue my first IT certification, the CompTIA A+. I had signed…
Key takeaways The transformer architecture has proved to be revolutionary in outperforming the classical RNN…
Once we learn how to deploy an Ubuntu server, how to manage users, and how…
Key-takeaways: Clean code isn’t just a nice thing to have or a luxury in software projects; it's a necessity. If we…
While developing a web application, or setting dynamic pages and meta tags we need to deal with…
Software architecture is one of the most discussed topics in the software industry today, and…
View Comments
Hello, I have followed all the steps and when executing it I get the error:
connecting ...
error {Error: Connection refused: Bad username or password
Could you help me?
same problem for me also please reply asap
same problem here. i think its because my private key has a passphrase but i don't know how to add this to the jwt.
No, it wasn't the passphrase. I have been trying for days to get this working but the same problem every time:
Error: error { Error: Connection refused: Not authorized
at MqttClient._handleConnack (/home/pi/Desktop/Google-IoT-
Device/node_modules/mqtt/lib/client.js:920:15)
error { Error: Connection refused: Bad username or password
at MqttClient._handleConnack (/home/pi/Desktop/Google-IoT-
Device/node_modules/mqtt/lib/client.js:920:15)
Please help if you know what the issue is.
Great article, really love GCP and Node.js. One question though. In the text you say that you're using MQTT-SN to connect to the MQTT Bridge, that is not the case right? From what I understand neither Google or AWS support MQTT-SN (over UDP) today? With QoS=-1 - that is fire and forget? Which I have to use with my project.
I have an error while installing : $ npm install rpi-dht-sensor --save
C:\Users\Jesus\Documents\Arduino\my ESP32\Esp32-lwmqtt_FINAL\node_modules\rpi-dht-sensor>if not defined npm_config_node_gyp (node "C:\Program Files\nodejs\node_modules\npm\bin\node-gyp-bin\\..\..\node_modules\node-gyp\bin\node-gyp.js" rebuild ) else (node "" rebuild )
Los proyectos de esta solución se van a compilar de uno en uno. Para habilitar la compilación en paralelo, agregue el modificador "/m".
Hello everyone issue is resolved of bad username and password
just reduce expiry time in jwt token
Anybody ever solve the above errors?
I'm getting the following:
error { Error: Connection refused: Not authorized
at MqttClient._handleConnack (/home/pi/Google-IoT-Device/node_modules/mqtt/lib/client.js:1076:15)
at MqttClient._handlePacket (/home/pi/Google-IoT-Device/node_modules/mqtt/lib/client.js:365:12)
at work (/home/pi/Google-IoT-Device/node_modules/mqtt/lib/client.js:283:12)
at Writable.writable._write (/home/pi/Google-IoT-Device/node_modules/mqtt/lib/client.js:294:5)
at doWrite (/home/pi/Google-IoT-Device/node_modules/mqtt/node_modules/readable-stream/lib/_stream_writable.js:428:64)
at writeOrBuffer (/home/pi/Google-IoT-Device/node_modules/mqtt/node_modules/readable-stream/lib/_stream_writable.js:417:5)
at Writable.write (/home/pi/Google-IoT-Device/node_modules/mqtt/node_modules/readable-stream/lib/_stream_writable.js:334:11)
at TLSSocket.ondata (_stream_readable.js:639:20)
at emitOne (events.js:116:13)
at TLSSocket.emit (events.js:211:7) code: 5 }
close
I have the same problem as Paul Ruddlesdin:
Did anyone ever solve this issue. I have a feeling it has to do with incorrect region?
New to this tutorial - ran into same issue with connection error. Love to hear if a solution exists.
I tried this tutorial and got some errors.
I noted my fix method.
Section: Setting up Node.js
[Error]: Not authorized, Bad username or password
-> Use ProjectID, not ProjectName
-> Verify Region, RegistryID, DeviceID
-> Re-generate certificate
-> In function createJwt(), reduce expire time (I set 20 minutes as in Google example code)
[Error]: File not found
var privateKeyFile = '../certs/rsa_private.pem';
-> var privateKeyFile = './certs/rsa_private.pem';
Section: Setting up the client
[Error]: PubSub is not a constructor
var PubSub = require('@google-cloud/pubsub');
-> var {PubSub} = require('@google-cloud/pubsub');
Section: Setting up Google Cloud Function
[Error]: TypeError: First argument must be a string, Buffer, ArrayBuffer, Array, or array-like object.
-> When create cloud function, choose Runtime: Node.js 6