In this article by Özen Özkaya and Giray Yıllıkçı, author of the book Arduino Computer Vision Programming, you will learn how to approach computer vision applications, how to divide an application development process into basic steps, how to realize these design steps and how to combine a vision system with the Arduino. Now it is time to connect all the pieces into one!
In this article you will learn about building a vision-assisted robot which can go to any point you want within the boundaries of the camera’s sight. In this scenario there will be a camera attached to the ceiling and, once you get the video stream from the robot and click on any place in the view, the robot will go there. This application will give you an all-in-one development application.
Before getting started, let’s try to draw the application scheme and define the potential steps. We want to build a vision-enabled robot which can be controlled via a camera attached to the ceiling and, when we click on any point in the camera view, we want our robot to go to this specific point.
This operation requires a mobile robot that can communicate with the vision system. The vision system should be able to detect or recognize the robot and calculate the position and orientation of the robot. The vision system should also give us the opportunity to click on any point in the view and it should calculate the path and the robot movements to get to the destination. This scheme requires a communication line between the robot and the vision controller. In the following illustration, you can see the physical scheme of the application setup on the left hand side and the user application window on the right hand side:
After interpreting the application scheme, the next step is to divide the application into small steps by using the computer vision approach.
In the data acquisition phase, we’ll only use the scene’s video stream. There won’t be an external sensor on the robot because, for this application, we don’t need one. Camera selection is important and the camera distance (the height from the robot plane) should be enough to see the whole area. We’ll use the blue and red circles above the robot to detect the robot and calculate its orientation. We don’t need smaller details. A resolution of about 640×480 pixels is sufficient for a camera distance of 120 cm. We need an RGB camera stream because we’ll use the color properties of the circles. We will use the Logitech C110, which is an affordable webcam. Any other OpenCV compatible webcam will work because this application is not very demanding in terms of vision input. If you need more cable length you can use a USB extension cable.
In the preprocessing phase, the first step is to remove the small details from the surface. Blurring is a simple and effective operation for this purpose. If you need to, you can resize your input image to reduce the image size and processing time. Do not forget that, if you resize to too small a resolution, you won’t be able to extract useful information. The following picture is of the Logitech C110 webcam:
The next step is processing. There are two main steps in this phase. The first step is to detect the circles in the image. The second step is to calculate the robot orientation and the path to the destination point. The robot can then follow the path and reach its destination. In color processing with which we can apply color filters to the image to get the image masks of the red circle and the blue circle, as shown in the following picture. Then we can use contour detection or blob analysis to detect the circles and extract useful features. It is important to keep it simple and logical:
Blob analysis detects the bounding boxes of two circles on the robot and, if we draw a line between the centers of the circles, once we calculate the line angle, we will get the orientation of the robot itself. The mid-point of this line will be the center of the robot. If we draw a line from the center of the robot to the destination point we obtain the straightest route. The circles on the robot can also be detected by using the Hough transform for circles but, because it is a relatively slow algorithm and it is hard to extract image statistics from the results, the blob analysis-based approach is better.
Another approach is by using the SURF, SIFT or ORB features. But these methods probably won’t provide fast real-time behavior, so blob analysis will probably work better.
After detecting blobs, we can apply post-filtering to remove the unwanted blobs. We can use the diameter of the circles, the area of the bounding box, and the color information, to filter the unwanted blobs.
By using the properties of the blobs (extracted features), it is possible to detect or recognize the circles, and then the robot. To be able to check if the robot has reached the destination or not, a distance calculation from the center of the robot to the destination point would be useful. In this scenario, the robot will be detected by our vision controller. Detecting the center of the robot is sufficient to track the robot.
Once we calculate the robot’s position and orientation, we can combine this information with the distance and orientation to the destination point and we can send the robot the commands to move it! Efficient planning algorithms can be applied in this phase but, we’ll implement a simple path planning approach. Firstly, the robot will orientate itself towards the destination point by turning right or left and then it will go forward to reach the destination. This scenario will work for scenarios without obstacles. If you want to extend the application for a complex environment with obstacles, you should implement an obstacle detection mechanism and an efficient path planning algorithm.
We can send the commands such as Left!, Right!, Go!, or Stop! to the robot over a wireless line. RF communication is an efficient solution for this problem. In this scenario, we need two NRF24L01 modules—the first module is connected to the robot controller and the other is connected to the vision controller.
The Arduino is the perfect means to control the robot and communicate with the vision controller. The vision controller can be built on any hardware platform such as a PC, tablet, or a smartphone. The vision controller application can be implemented on lots of operating systems as OpenCV is platform-independent. We preferred Windows and a laptop to run our vision controller application.
As you can see, we have divided our application into small and easy-to-implement parts. Now it is time to build them all!
Building a robot
It is time to explain how to build our Click-to-Go robot. Before going any further we would like to boldly say that robotic projects can teach us the fundamental fields of science such as mechanics, electronics, and programming.
As we go through the building process of our Click-to-Go robot, you will see that we have kept it as simple as possible. Moreover, instead of buying ready-to-use robot kits, we have built our own simple and robust robot. Of course, if you are planning to buy a robot kit or already have a kit available, you can simply adapt your existing robot into this project.
Our robot design is relatively simple in terms of mechanics. We will use only a box-shaped container platform, two gear motors with two individual wheels, a battery to drive the motors, one nRF24L01 Radio Frequency (RF) transceiver module, a bunch of jumper wires, an L293D IC and, of course, one Arduino Uno board module. We will use one more nRF24L01 and one more Arduino Uno for the vision controller communication circuit.
Our Click-to-Go robot will be operated by a simplified version of a differential drive. A differential drive can be summarized as a relative speed change on the wheels, which assigns a direction to the robot. In other words, if both wheels spin at the same rate, the robot goes forward. To drive in reverse, the wheels spin in the opposite direction. To turn left, the left wheel turns backwards and the right wheel stays still or turns forwards. Similarly, to turn right, the right wheel turns backwards and the left stays still or turns forwards.
You can get curved paths by varying the rotation speeds of the wheels. Yet, to cover every aspect of this comprehensive project, we will drive the wheels of both the motors forward to go forwards. To turn left, the left wheel stays still and the right wheel turns forward. Symmetrically, to turn right, the right motor stays still and the left motor runs forward. We will not use running motors in a reverse direction to go backwards. Instead, we will change the direction of the robot by turning right or left.
Building mechanics
As we stated earlier, the mechanics of the robot are fairly simple. First of all we need a small box-shaped container to use as both a rigid surface and the storage for the battery and electronics. For this purpose, we will use a simple plywood box. We will attach gear motors in front of the plywood box and any kind of support surface to the bottom of the box. As can be seen in the following picture, we used a small wooden rod to support the back of the robot to level the box:
If you think that the wooden rod support is dragging, we recommend adding a small ball support similar to Pololu’s ball caster, shown at https://www.pololu.com/product/950. It is not a very expensive component and it significantly improves the mobility of the robot.
You may want to drill two holes next to the motor wirings to keep the platform tidy. The easiest way to attach the motors and the support rod is by using two-sided tape. Just make sure that the tape is not too thin. It is much better to use two-sided foamy tape.
The topside of the robot can be covered with a black shell to enhance the contrast between the red and blue circles. We will use these circles to ascertain the orientation of the robot during the operation, as mentioned earlier. For now, don’t worry too much about this detail. Just be aware that we need to cover the top of the robot with a flat surface. We will explain in detail on how these red and blue circles are used. It is worth mentioning that we used large water bottle lids. It is better to use matt surfaces instead of shiny surfaces to avoid glare in the image.
The finished Click-to-Go robot should be similar to the robot shown in the following picture. The robot’s head is on the side with the red circle:
As we have now covered building the mechanics of our robot we can move on to building the electronics.
Building the electronics
We will use two separate Arduino Unos for this vision-enabled robot project, one each for the robot and the transmitter system. The electronic setup needs a little bit more attention than the mechanics. The electronic components of the robot and the transmitter units are similar. However, the robot needs more work.
We have selected nRF24L01 modules for the wireless communication module,. These modules are reliable and easy to find from both the Internet and local hobby stores. It is possible to use any pair of wireless connectivity modules but, for this project, we will stick with nRF24L01 modules, as shown in this picture:
For the driving motors we will need to use a quadruple half-H driver, L293D. Again, every electronic shop should have these ICs. As a reminder, you may need to buy a couple of spare L293D ICs in case you burn the IC by mistake. Following is the picture of the L293D IC:
We will need a bunch of jumper wires to connect the components together. It is nice to have a small breadboard for the robot/receiver, to wire the L293D. The transmitter part is very simple so a breadboard is not essential.
Robot/receiver and transmitter drawings
The drawings of both the receiver and the transmitter have two common modules: Arduino Uno and nRF24L01 connectivity modules. The connections of the nRF24L01 modules on both sides are the same. In addition to these connectivity modules, for the receiver, we need to put some effort into connecting the L293D IC and the battery to power up the motors.
In the following picture, we can see a drawing of the transmitter. As it will always be connected to the OpenCV platform via the USB cable, there is no need to feed the system with an external battery:
As shown in the following picture of the receiver and the robot, it is a good idea to separate the motor battery from the battery that feeds the Arduino Uno board because the motors may draw high loads or create high loads, which can easily damage the Arduino board’s pin outs. Another reason is to keep the Arduino working even if the battery motor has drained. Separating the feeder batteries is a very good practice to follow if you are planning to use more than one 12V battery. To keep everything safe, we fed the Arduino Uno with a 6V battery pack and the motors with a 9V battery:
Drawings of receiver systems can be little bit confusing and lead to errors. It is a good idea to open the drawings and investigate how the connections are made by using Fritzing. You can download the Fritzing drawings of this project from https://github.com/ozenozkaya/click_to_go_robot_drawings.
To download the Fritzing application, visit the Fritzing download page:
Building the robot controller and communications
We are now ready to go through the software implementation of the robot and the transmitter. Basically what we are doing here is building the required connectivity to send data to the remote robot continuously from OpenCV via a transmitter. OpenCV will send commands to the transmitter through a USB cable to the first Arduino board, which will then send the data to the unit on the robot. And it will send this data to the remote robot over the RF module. Follow these steps:
- Before explaining the code, we need to import the RF24 library. To download RF24 library drawings please go to the GitHub link at https://github.com/maniacbug/RF24.
- After downloading the library, go to Sketch | Include Library | Add .ZIP Library… to include the library in the Arduino IDE environment.
- After clicking Add .ZIP Library…, a window will appear. Go into the downloads directory and select the RF24-master folder that you just downloaded. Now you are ready to use the RF24 library. As a reminder, it is pretty much the same to include a library in Arduino IDE as on other platforms. It is time to move on to the explanation of the code!
It is important to mention that we use the same code for both the robot and the transmitter, with a small trick! The same code works differently for the robot and the transmitter. Now, let’s make everything simpler during the code explanation. The receiver mode needs to ground an analog 4 pin. The idea behind the operation is simple; we are setting the role_pin to high through its internal pull-up resistor. So, it will read high even if you don’t connect it, but you can still safely connect it to ground and it will read low. Basically, the analog 4 pin reads 0 if the there is a connection with a ground pin. On the other hand, if there is no connection to the ground, the analog 4 pin value is kept as 1. By doing this at the beginning, we determine the role of the board and can use the same code on both sides. Here is the code:
#include <SPI.h>
#include "nRF24L01.h"
#include "RF24.h"
#define MOTOR_PIN_1 3
#define MOTOR_PIN_2 5
#define MOTOR_PIN_3 6
#define MOTOR_PIN_4 7
#define ENABLE_PIN 4
#define SPI_ENABLE_PIN 9
#define SPI_SELECT_PIN 10
const int role_pin = A4;
typedef enum {transmitter = 1, receiver} e_role;
unsigned long motor_value[2];
String input_string = "";
boolean string_complete = false;
RF24 radio(SPI_ENABLE_PIN, SPI_SELECT_PIN);
const uint64_t pipes[2] = { 0xF0F0F0F0E1LL, 0xF0F0F0F0D2LL };
e_role role = receiver;
void setup() {
pinMode(role_pin, INPUT);
digitalWrite(role_pin, HIGH);
delay(20);
radio.begin();
radio.setRetries(15, 15);
Serial.begin(9600);
Serial.println(" Setup Finished");
if (digitalRead(role_pin)) {
Serial.println(digitalRead(role_pin));
role = transmitter;
}
else {
Serial.println(digitalRead(role_pin));
role = receiver;
}
if (role == transmitter) {
radio.openWritingPipe(pipes[0]);
radio.openReadingPipe(1, pipes[1]);
}
else {
pinMode(MOTOR_PIN_1, OUTPUT);
pinMode(MOTOR_PIN_2, OUTPUT);
pinMode(MOTOR_PIN_3, OUTPUT);
pinMode(MOTOR_PIN_4, OUTPUT);
pinMode(ENABLE_PIN, OUTPUT);
digitalWrite(ENABLE_PIN, HIGH);
radio.openWritingPipe(pipes[1]);
radio.openReadingPipe(1, pipes[0]);
}
radio.startListening();
}
void loop() {
// TRANSMITTER CODE BLOCK //
if (role == transmitter) {
Serial.println("Transmitter");
if (string_complete)
{
if (input_string == "Right!")
{
motor_value[0] = 0;
motor_value[1] = 120;
}
else if (input_string == "Left!")
{
motor_value[0] = 120;
motor_value[1] = 0;
}
else if (input_string == "Go!")
{
motor_value[0] = 120;
motor_value[1] = 110;
}
else
{
motor_value[0] = 0;
motor_value[1] = 0;
}
input_string = "";
string_complete = false;
}
radio.stopListening();
radio.write(motor_value, 2 * sizeof(unsigned long));
radio.startListening();
delay(20);
}
// RECEIVER CODE BLOCK //
if (role == receiver) {
Serial.println("Receiver");
if (radio.available()) {
bool done = false;
while (!done) {
done = radio.read(motor_value, 2 * sizeof(unsigned long));
delay(20);
}
Serial.println(motor_value[0]);
Serial.println(motor_value[1]);
analogWrite(MOTOR_PIN_1, motor_value[1]);
digitalWrite(MOTOR_PIN_2, LOW);
analogWrite(MOTOR_PIN_3, motor_value[0]);
digitalWrite(MOTOR_PIN_4 , LOW);
radio.stopListening();
radio.startListening();
}
}
}
void serialEvent() {
while (Serial.available()) {
// get the new byte:
char inChar = (char)Serial.read();
// add it to the inputString:
input_string += inChar;
// if the incoming character is a newline, set a flag
// so the main loop can do something about it:
if (inChar == '!' || inChar == '?') {
string_complete = true;
Serial.print("data_received");
}
}
}This example code is taken from one of the examples in the RF24 library. We have changed it in order to serve our needs in this project. The original example can be found in the RF24-master/Examples/pingpair directory.
Summary
We have combined everything we have learned up to now and built an all-in-one application. By designing and building the Click-to-Go robot from scratch you have embraced the concepts. You can see that the vision approach very well, even for complex applications. You now know how to divide a computer vision application into small pieces, how to design and implement each design step, and how to efficiently use the tools you have.
Resources for Article:
Further resources on this subject:
- Getting Started with Arduino[article]
- Arduino Development[article]
- Programmable DC Motor Controller with an LCD [article]