This article by Brandon Sanders, the author of the book Mastering Leap Motion, focuses on what he specializes in—hardware. While normal applications are all fine and good, he finds it much more gratifying if a program he writes has an impact in the physical world.
(For more resources related to this topic, see here.)
One of the most popular hobbyist hardware solutions, as I’m sure you know, is the Arduino. This cute little blue board from Italy brought the power of micro controllers to the masses.
Throughout this article, we’re going to work on integrating a Leap Motion Controller with an Arduino board via a simplistic program; the end goal is to make the built-in LED on an Arduino board blink either slower or faster depending on how far a user’s hand is away from the Leap. While this is a relatively simple task, it’s a great way to demonstrate how you can connect something like the Leap to an external piece of hardware. From there, it’s only a hop, skip, and jump to control robots and other cool things with the Leap!
This project will follow the client-server model of programming: we’ll be writing a simple Java server which will be run from a computer, and a C++ client which will run on an Arduino board connected to the computer. The server will be responsible for retrieving Leap Motion input and sending it to the client, while the client will be responsible for making an LED blink based on data received from the server.
Before we begin, I’d like to note that you can download the completed (and working) project from GitHub at https://github.com/Mizumi/Mastering-Leap-Motion-Chapter-9-Project-Leapduino.
A few things you’ll need
Before you begin working on this tutorial, there are a few things you’re going to need:
- A computer (for obvious reasons).
- A Leap Motion Controller.
- An Arduino of some kind. This tutorial is based around the Uno model, but other similar models like the Mega should work just as well.
- A USB cable to connect your Arduino to your computer.
- Optionally, the Eclipse IDE (this tutorial will assume you’re using Eclipse for the sake of readability and instruction).
Setting up the environment
First off, you’re going to need a copy of the Leap Motion SDK so that you can add the requisite library jar files and DLLs to the project. If you don’t already have it, you can get a copy of the SDK from https://www.developer.leapmotion.com/downloads/.
Next, you’re going to need the Java Simple Serial Connector (JSSC) library and the Arduino IDE.
You can download the library JAR file for JSSC from GitHub at https://github.com/scream3r/java-simple-serial-connector/releases. Once the download completes, extract the JAR file from the downloaded ZIP folder and store it somewhere safe; you’ll need it later on in this tutorial.
You can then proceed to download the Arduino IDE from their official website at http://arduino.cc/en/Main/Software. If you’re on Windows, you will be able to download a Windows installer file which will automagically install the entire IDE on to your computer. On the other hand, Mac and Linux users will need to instead download .zip or .tgz files and then extract them manually, running the executable binary from the extracted folder contents.
Setting up the project
To set up our project, perform the following steps:
- The first thing we’re going to do is create a new Java project. This can be easily achieved by opening up Eclipse (to reiterate for the third time, this tutorial will assume you’re using Eclipse) and heading over to File -> New -> Java Project.
- You will then be greeted by a project creation wizard, where you’ll be prompted to choose a name for the project (I used Leapduino).
- Click on the Finish button when you’re done.
My current development environment is based around the Eclipse IDE for Java Developers, which can be found at http://www.eclipse.org/downloads. The instructions that follow will use Eclipse nomenclature and jargon, but they will still be usable if you’re using something else (like NetBeans).
Once the project is created, navigate to it in the Package Explorer window. You’ll want to go ahead and perform the following actions:
- Create a new package for the project by right-clicking on the src folder for your project in the Package Explorer and then navigating to New | Package in the resulting tooltip. You can name it whatever you like; I personally called mine com.mechakana.tutorials.
- You’ll now want to add three files to our newly-created package: Leapduino.java, LeapduinoListener.java, and RS232Protocol.java. To create a new file, simply right-click on the package and then navigate to New | Class.
- Create a new folder in your project by right-clicking on the project name in the Package Explorer and then navigating to New | Folder in the resulting tooltip. For the purposes of this tutorial, please name it Leapduino.
- Now add one file to your newly created folder: Leapduino.ino. This file will contain all of the code that we’re going to upload to the Arduino.
With all of our files created, we need to add the libraries to the project. Go ahead and create a new folder at the root directory of your project, called lib. Within the lib folder, you’ll want to place the jssc.jar file that you downloaded earlier, along with the LeapJava.jar file from the Leap Motion SDK. Then, you will want to add the appropriate Leap.dll and LeapJava.dll files for your platform to the root of your project.
Finally, you’ll need to modify your Java build path to link the LeapJava.jar and jssc.jar files to your project. This can be achieved by right-clicking on your project in the Package Explorer (within Eclipse) and navigating to Build Path… | Configure Build Path…. From there, go to the Libraries tab and click on Add JARs…, selecting the two aforementioned JAR files (LeapJava.jar and jssc.jar).
When you’re done, your project should look similar to the following screenshot:
And you’re done; now to write some code!
Writing the Java side of things
With everything set up and ready to go, we can start writing some code. First off, we’re going to write the RS232Protocol class, which will allow our application to communicate with any Arduino board connected to the computer via a serial (RS-232) connection.
This is where the JSSC library will come into play, allowing us to quickly and easily write code that would otherwise be quite lengthy (and not fun).
Fun fact
RS-232 is a standard for serial communications and transmission of data. There was a time when it was a common feature on a personal computer, used for modems, printers, mice, hard drives, and so on. With time, though, the Universal Serial Bus (USB) technology replaced RS-232 for many of those roles.
Despite this, today’s industrial machines, scientific equipment and (of course) robots still make heavy usage of this protocol due to its light weight and ease of use; the Arduino is no exception!
Go ahead and open up the RS232Protocol.java file which we created earlier, and enter the following:
package com.mechakana.tutorials;
import jssc.SerialPort;
import jssc.SerialPortEvent;
import jssc.SerialPortEventListener;
import jssc.SerialPortException;
public class RS232Protocol
{
//Serial port we're manipulating.
private SerialPort port;
//Class: RS232Listener
public class RS232Listener implements SerialPortEventListener
{
public void serialEvent(SerialPortEvent event)
{
//Check if data is available.
if (event.isRXCHAR() && event.getEventValue() > 0)
{
try
{
int bytesCount = event.getEventValue();
System.out.print(port.readString(bytesCount));
}
catch (SerialPortException e) { e.printStackTrace(); }
}
}
}
//Member Function: connect
public void connect(String newAddress)
{
try
{
//Set up a connection.
port = new SerialPort(newAddress);
//Open the new port and set its parameters.
port.openPort();
port.setParams(38400, 8, 1, 0);
//Attach our event listener.
port.addEventListener(new RS232Listener());
}
catch (SerialPortException e) { e.printStackTrace(); }
}
//Member Function: disconnect
public void disconnect()
{
try { port.closePort(); }
catch (SerialPortException e) { e.printStackTrace(); }
}
//Member Function: write
public void write(String text)
{
try { port.writeBytes(text.getBytes()); }
catch (SerialPortException e) { e.printStackTrace(); }
}
}
All in all, RS232Protocol is a simple class—there really isn’t a whole lot to talk about here! However, I’d love to point your attention to one interesting part of the class:
public class RS232Listener implements SerialPortEventListener
{
public void serialEvent(SerialPortEvent event) { /*code*/ }
}
You might have found it rather odd that we didn’t create a function for reading from the serial port—we only created a function for writing to it. This is because we’ve opted to utilize an event listener, the nested RS232Listener class. Under normal operating conditions, this class’s serialEvent function will be called and executed every single time new information is received from the port. When this happens, the function will print all of the incoming data out to the user’s screen. Isn’t that nifty?
Moving on, our next class is a familiar one—LeapduinoListener, a simple Listener implementation. This class represents the meat of our program, receiving Leap Motion tracking data and then sending it over our serial port to the connected Arduino.
Go ahead and open up LeapduinoListener.java and enter the following code:
package com.mechakana.tutorials;
import com.leapmotion.leap.*;
public class LeapduinoListener extends Listener
{
//Serial port that we'll be using to communicate with the Arduino.
private RS232Protocol serial;
//Constructor
public LeapduinoListener(RS232Protocol serial)
{
this.serial = serial;
}
//Member Function: onInit
public void onInit(Controller controller)
{
System.out.println("Initialized");
}
//Member Function: onConnect
public void onConnect(Controller controller)
{
System.out.println("Connected");
}
//Member Function: onDisconnect
public void onDisconnect(Controller controller)
{
System.out.println("Disconnected");
}
//Member Function: onExit
public void onExit(Controller controller)
{
System.out.println("Exited");
}
//Member Function: onFrame
public void onFrame(Controller controller)
{
//Get the most recent frame.
Frame frame = controller.frame();
//Verify a hand is in view.
if (frame.hands().count() > 0)
{
//Get some hand tracking data.
int hand = (int) (frame.hands().frontmost().palmPosition().getY());
//Send the hand pitch to the Arduino.
serial.write(String.valueOf(hand));
//Give the Arduino some time to process our data.
try { Thread.sleep(30); }
catch (InterruptedException e) { e.printStackTrace(); }
}
}
}
In this class, we’ve got the basic Leap Motion API onInit, onConnect, onDisconnect, onExit, and onFrame functions.
Our onFrame function is fairly straightforward: we get the most recent frame, verify a hand is within view, retrieve its y axis coordinates (height from the Leap Motion Controller) and then send it off to the Arduino via our instance of the RS232Protocol class (which gets assigned during initialization). The remaining functions simply print text out to the console telling us when the Leap has initialized, connected, disconnected, and exited (respectively).
And now, for our final class on the Java side of things: Leapduino! This class is a super basic main class that simply initializes the RS232Protocol class and the LeapduinoListener—that’s it!
Without further ado, go on ahead and open up Leapduino.java and enter the following code:
package com.mechakana.tutorials;
import com.leapmotion.leap.Controller;
public class Leapduino
{
//Main
public static final void main(String args[])
{
//Initialize serial communications.
RS232Protocol serial = new RS232Protocol();
serial.connect("COM4");
//Initialize the Leapduino listener.
LeapduinoListener leap = new LeapduinoListener(serial);
Controller controller = new Controller();
controller.addListener(leap);
}
}
Like all of the classes so far, there isn’t a whole lot to say here. That said, there is one line that you must absolutely be aware of, since it can change depending on how you’re Arduino is connected:
serial.connect("COM4");
Depending on which port Windows chose for your Arduino when it connected to your computer (more on that next), you will need to modify the COM4 value in the above line to match the port your Arduino is on. Examples of values you’ll probable use are COM3, COM4, and COM5.
And with that, the Java side of things is complete. If you run this project right now, most likely all you’ll see will be two lines of output: Initialized and Connected. If you want to see anything else happen, you’ll need to move on to the next section and get the Arduino side of things working.
Writing the Arduino side of things
With our Java coding done, it’s time to write some good-old C++ for the Arduino. If you were able to use the Windows installer for Arduino, simply navigate to the Leapduino.ino file in your Eclipse project explorer and double click on it.
If you had to extract the entire Arduino IDE and store it somewhere instead of running a simple Windows installer, navigate to it and launch the Arduino.exe file. From there, select File | Open, navigate to the Leapduino.ino file on your computer and double click on it.
You will now be presented with a screen similar to the one here:
This is the wonderful Arduino IDE—a minimalistic and straightforward text editor and compiler for the Arduino microcontrollers.
On the top left of the IDE, you’ll find two circular buttons: the check mark verifies (compiles) your code to make sure it works, and the arrow deploys your code to the Arduino board connected to your computer. On the bottom of the IDE, you’ll find the compiler output console (the black box), and on the very bottom right you’ll see a line of text telling you which Arduino model is connected to your computer, and on what port (I have an Arduino Uno on COM4 in the preceding screenshot). As is typical for many IDEs and text editors, the big white area in the middle is where your code will go.
So without further ado, let’s get started with writing some code! Input all of the text shown here into the Arduino IDE:
//Most Arduino boards have an LED pre-wired to pin 13.
int led = 13;
//Current LED state. LOW is off and HIGH is on.
int ledState = LOW;
//Blink rate in milliseconds.
long blinkRate = 500;
//Last time the LED was updated.
long previousTime = 0;
//Function: setup
void setup()
{
//Initialize the built-in LED (assuming the Arduino board has one)
pinMode(led, OUTPUT);
//Start a serial connection at a baud rate of 38,400.
Serial.begin(38400);
}
//Function: loop
void loop()
{
//Get the current system time in milliseconds.
unsigned long currentTime = millis();
//Check if it's time to toggle the LED on or off.
if (currentTime - previousTime >= blinkRate)
{
previousTime = currentTime;
if (ledState == LOW) ledState = HIGH;
else ledState = LOW;
digitalWrite(led, ledState);
}
//Check if there is serial data available.
if (Serial.available())
{
//Wait for all data to arrive.
delay(20);
//Our data.
String data = "";
//Iterate over all of the available data and compound it into
a string.
while (Serial.available())
data += (char) (Serial.read());
//Set the blink rate based on our newly-read data.
blinkRate = abs(data.toInt() * 2);
//A blink rate lower than 30 milliseconds won't really be
perceptable by a human.
if (blinkRate < 30) blinkRate = 30;
//Echo the data.
Serial.println("Leapduino Client Received:");
Serial.println("Raw Leap Data: " + data + " | Blink Rate (MS):
" + blinkRate);
}
}
Now, let’s go over the contents.
The first few lines are basic global variables, which we’ll be using throughout the program (the comments do a good job of describing them, so we won’t go into much detail here).
The first function, setup, is an Arduino’s equivalent of a constructor; it’s called only once, when the Arduino is first turned on. Within the setup function, we initialize the built-in LED (most Arduino boards have an LED pre-wired to pin 13) on the board. We then initialize serial communications at a baud rate of 38,400 bits per second—this will allow our board to communicate with the computer later on.
Fun fact
The baud rate (abbreviated as Bd in some diagrams) is the unit for symbol rate or modulation rate in symbols or pulses per second. Simply put, on serial ports, the baud rate controls how many bits a serial port can send per second—the higher the number, the faster a serial port can communicate.
The question is, why don’t we set a ridiculously high rate? Well, the higher you go with the baud rate, the more likely it is for there to be data loss—and we all know data loss just isn’t good. For many applications, though, a baud rate of 9,600 to 38,400 bits per second is sufficient.
Moving on to the second function, loop is the main function in any Arduino program, which is repeatedly called while the Arduino is turned on. Due to this functionality, many programs will treat any code within this function as if it were inside a while (true) loop.
In loop, we start off by getting the current system time (in milliseconds) and then comparing it to our ideal blink rate for the LED. If the time elapsed since our last blink exceeds the ideal blink rate, we’ll go ahead and toggle the LED on or off accordingly.
We then proceed to check if any data has been received over the serial port. If it has, we’ll proceed to wait for a brief period of time, 20 milliseconds, to make sure all data has been received. At that point, our code will proceed to read in all of the data, parse it for an integer (which will be our new blink rate), and then echo the data back out to the serial port for diagnostics purposes.
As you can see, an Arduino program (or sketch, as they are formally known) is quite simple. Why don’t we test it out?
Deploying and testing the application
With all of the code written, it’s time to deploy the Arduino side of things to the, well, Arduino.
The first step is to simply open up your Leapduino.ino file in the Arduino IDE. Once that’s done, navigate to Tools | Board and select the appropriate option for your Arduino board. In my case, it’s an Arduino Uno. At this point, you’ll want to verify that you have an Arduino connected to your computer via a USB cable—after all, we can’t deploy to thin air!
At this point, once everything is ready, simply hit the Deploy button in the top-left of the IDE, as seen here:
If all goes well, you’ll see the following output in the console after 15 or so seconds:
And with that, your Arduino is ready to go! How about we test it out? Keeping your Arduino plugged into your computer, go on over to Eclipse and run the project we just made. Once it’s running, try moving your hand up and down over your Leap Motion controller; if all goes well, you’ll see the following output from within the console in Eclipse:
All of that data is coming directly from the Arduino, not your Java program; isn’t that cool? Now, take a look at your Arduino while you’re doing this; you should notice that the built-in LED (circled in the following image, labelled L on the board itself) will begin to blink slower or faster depending on how close your hand gets to the Leap.
Circled in red: the built-in L LED on an Arduino Uno, wired to pin 13 by default.
With this, you’ve created a simple Leap Motion application for use with an Arduino. From here, you could go on to make an Arduino-controlled robotic arm driven by coordinates from the Leap, or maybe an interactive light show. The possibilities are endless, and this is just the (albeit extremely, extremely simple) tip of the iceberg.
Summary
In this article, you had a lengthy look at some things you can do with the Leap Motion Controller and hardware such as Arduino.
If you have any questions, I encourage you to contact me directly at [email protected]. You can also visit my website, http://www.mechakana.com, for more technological goodies and tutorials.
Resources for Article:
Further resources on this subject:
- Major SDK components [Article]
- 2D Twin-stick Shooter [Article]
- What’s Your Input? [Article]
