(For more resources related to this topic, see here.)
Getting ready
Create a project in Visual Studio and prepare it for working with OpenNI and NiTE.
How to do it…
- Copy ReadLastCharOfLine() and HandleStatus() functions to the top of your source code (just below the #include lines).
- Then add following lines of code:
class MouseController : public nite::HandTracker::NewFrameListener { private: float startPosX, startPosY; int curX, curY; nite::HandId handId; RECT desktopRect; public: MouseController(){ startPosX = startPosY = -1; POINT curPos; if (GetCursorPos(&curPos)) { curX = curPos.x; curY = curPos.y; }else{ curX = curY = 0; } handId = -1; const HWND hDesktop = GetDesktopWindow(); GetWindowRect(hDesktop, &desktopRect); } void onNewFrame(nite::HandTracker& hTracker){ nite::Status status = nite::STATUS_OK; nite::HandTrackerFrameRef newFrame; status = hTracker.readFrame(&newFrame); if (!HandleStatus(status) || !newFrame.isValid()) return; const nite::Array<nite::GestureData>& gestures = newFrame.getGestures(); for (int i = 0; i < gestures.getSize(); ++i){ if (gestures[i].isComplete()){ if (gestures[i].getType() == nite::GESTURE_CLICK){ INPUT Input = {0}; Input.type = INPUT_MOUSE; Input.mi.dwFlags = MOUSEEVENTF_LEFTDOWN | MOUSEEVENTF_LEFTUP; SendInput(1, &Input, sizeof(INPUT)); }else{ nite::HandId handId; status = hTracker.startHandTracking( gestures[i].getCurrentPosition(), &handId); } } } const nite::Array<nite::HandData>& hands = newFrame.getHands(); for (int i = hands.getSize() -1 ; i >= 0 ; --i){ if (hands[i].isTracking()){ if (hands[i].isNew() || handId != hands[i].getId()){ status = hTracker.convertHandCoordinatesToDepth( hands[i].getPosition().x, hands[i].getPosition().y, hands[i].getPosition().z, &startPosX, &startPosY); handId = hands[i].getId(); if (status != nite::STATUS_OK){ startPosX = startPosY = -1; } }else if (startPosX >= 0 && startPosY >= 0){ float posX, posY; status = hTracker.convertHandCoordinatesToDepth( hands[i].getPosition().x, hands[i].getPosition().y, hands[i].getPosition().z, &posX, &posY); if (status == nite::STATUS_OK){ if (abs(int(posX - startPosX)) > 10) curX += ((posX - startPosX) - 10) / 3; if (abs(int(posY - startPosY)) > 10) curY += ((posY - startPosY) - 10) / 3; curX = min(curX, desktopRect.right); curX = max(curX, desktopRect.left); curY = min(curY, desktopRect.bottom); curY = max(curY, desktopRect.top); SetCursorPos(curX, curY); } } break; } } } };
- Then locate the following line:
int _tmain(int argc, _TCHAR* argv[]) {
- Add the following inside this function:
nite::Status status = nite::STATUS_OK; status = nite::NiTE::initialize(); if (!HandleStatus(status)) return 1; printf("Creating hand tracker ...rn"); nite::HandTracker hTracker; status = hTracker.create(); if (!HandleStatus(status)) return 1; MouseController* listener = new MouseController(); hTracker.addNewFrameListener(listener); hTracker.startGestureDetection(nite::GESTURE_HAND_RAISE); hTracker.startGestureDetection(nite::GESTURE_CLICK); printf("Reading data from hand tracker ...rn"); ReadLastCharOfLine(); nite::NiTE::shutdown(); openni::OpenNI::shutdown(); return 0;
How it works…
Both the ReadLastCharOfLine() and HandleStatus() functions are present here too. These functions are well known to you and don’t need any explanation.
Then in the second part, we declared a class/struct that we are going to use for capturing the new data available event from the nite::HandTracker object.
But the definition of this class is a little different here. Other than the onNewFrame() method, we defined a number of variables and a constructor method for this class too. We also changed its name to MouseController to be able to make more sense of it.
class MouseController : public nite::HandTracker::NewFrameListener { private: float startPosX, startPosY; int curX, curY; nite::HandId handId; RECT desktopRect;
As you can see, our class is still a child class of nite::HandTracker::NewFrameListener because we are going use it to listen to the nite::HandTracker events. Also, we defined six variables in our class. startPosX and startPosY are going to hold the initial position of the active hand whereas curY and curX are going to hold the position of the mouse when in motion. The handId variable is responsible for holding the ID of the active hand and desktopRecthold for holding the size of the desktop so that we can move our mouse only in this area. These variables are all private variables; this means they will not be accessible from the outside of the class. Then we have the class’s constructor method that initializes some of the preceding variables. Refer to the following code:
public: MouseController(){ startPosX = startPosY = -1; POINT curPos; if (GetCursorPos(&curPos)) { curX = curPos.x; curY = curPos.y; }else{ curX = curY = 0; } handId = -1; const HWND hDesktop = GetDesktopWindow(); GetWindowRect(hDesktop, &desktopRect); }
In the constructor, we set both startPosX and startPosY to -1 and then store the current position of the mouse in the curX and curY variables. Then we set the handId variable to -1 to know-mark that there is no active hand currently, and retrieve the value of desktopRect using two Windows API methods, GetDesktopWindow() and GetWindowRect().
The most important tasks are happening in the onNewFrame() method. This method is the one that will be called when new data becomes available in nite::HandTracker; after that, this method will be responsible for processing this data.
As the running of this method means that new data is available, the first thing to do in its body is to read this data. So we used the nite::HandTracker::readFrame() method to read the data from this object:
void onNewFrame(nite::HandTracker& hTracker){ nite::Status status = nite::STATUS_OK; nite::HandTrackerFrameRef newFrame; status = hTracker.readFrame(&newFrame);
When working with nite::HandTracker, the first thing to do after reading the data is to handle gestures if you expect any. We expect to have Hand Raise to detect new hands and click gesture to perform the mouse click:
const nite::Array<nite::GestureData>& gestures = newFrame.getGestures(); for (int i = 0; i < gestures.getSize(); ++i){ if (gestures[i].isComplete()){ if (gestures[i].getType() == nite::GESTURE_CLICK){ INPUT Input = {0}; Input.type = INPUT_MOUSE; Input.mi.dwFlags = MOUSEEVENTF_LEFTDOWN | MOUSEEVENTF_LEFTUP; SendInput(1, &Input, sizeof(INPUT)); }else{ nite::HandId handId; status = hTracker.startHandTracking( gestures[i].getCurrentPosition(), &handId); } } }
As you can see, we retrieved the list of all the gestures using nite::HandTrackerFrameRef::getGestures() and then looped through them, searching for the ones that are in the completed state. Then if they are the nite::GESTURE_CLICK gesture, we need to perform a mouse click. We used the SendInput() function from the Windows API to do it here. But if the recognized gesture wasn’t of the type nite::GESTURE_CLICK, it must be a nite::GESTURE_HAND_RAISE gesture; so, we need to request for the tracking of this newly recognized hand using the nite::HandTracker::startHandTracking() method.
The next thing is to take care of the hands being tracked. To do so, we first need to retrieve a list of them using the nite::HandTrackerFrameRef::getHands() method and then loop through them. This can be done using a simple for loop as we used for the gestures. But as we want to read this list in a reverse order, we need to use a reverse for loop. The reason we need to read this list in the reverse order is that we always want the last recognized hand to control the mouse:
const nite::Array<nite::HandData>& hands = newFrame.getHands(); for (int i = hands.getSize() - 1 ; i >= 0 ; --i){
Then we need to make sure that the current hand is under-tracking because we don’t want an invisible hand to control the mouse. The first hand being tracked is the one we want, so we will break the looping there, of course, after the processing part, which we will remove from the following code to make it clearer.
if (hands[i].isTracking()){ . . . break; }
Speaking of processing, in the preceding three lines of code (with periods) we have another condition. This condition is responsible for finding out if this hand is the same one that had control of the mouse in the last frame. If it is a new hand (either it is a newly recognized hand or it is a newly active hand), we need to save its current position in the startPosX and startPosY variables.
if (hands[i].isNew() || handId != hands[i].getId()){ status = hTracker.convertHandCoordinatesToDepth( hands[i].getPosition().x, hands[i].getPosition().y, hands[i].getPosition().z, &startPosX, &startPosY); handId = hands[i].getId(); if (status != nite::STATUS_OK){ startPosX = startPosY = -1; }
If it was the same hand, we have another condition.
Do we have the startPosX and startPosY variables already or do we not have them yet? If we have them, we can calculate the mouse’s movement. But first we need to calculate the position of the hand relative to the depth frame.
}else if (startPosX >= 0 && startPosY >= 0){ float posX, posY; status = hTracker.convertHandCoordinatesToDepth( hands[i].getPosition().x, hands[i].getPosition().y, hands[i].getPosition().z, &posX, &posY);
Once the process of conversation ends, we need to calculate the new position of the mouse depending on how the hand’s position changes. But we want to define a safe area for it to be static when small changes happen. So we calculate the new position of the mouse only if it has moved by more than 10 pixels in our depth frame:
if (status == nite::STATUS_OK){ if (abs(int(posX - startPosX)) > 10) curX += ((posX - startPosX) - 10) / 3; if (abs(int(posY - startPosY)) > 10) curY += ((posY - startPosY) - 10) / 3;
As you can see in the preceding code, we also divided the changes by 3 because we didn’t want it to move too fast.
But before setting the position of the mouse, we need to first make sure that the new positions are in the screen view port using the desktopRect variable:
curX = min(curX, desktopRect.right); curX = max(curX, desktopRect.left); curY = min(curY, desktopRect.bottom); curY = max(curY, desktopRect.top);
After calculating everything, we can set the new position of the mouse using SetCursorPos() from the Windows API:
SetCursorPos(curX, curY);
Step three and four are not markedly differently. In this step, we have the initialization process; this includes the initialization of NiTE and the creation of the nite::HandTracker variable.
status = nite::NiTE::initialize(); . . . nite::HandTracker hTracker; status = hTracker.create();
Then we should add our newly defined structure/class as a listener to nite::HandTracker so that nite::HandTracker can call it later when a new frame becomes available:
MouseController* listener = new MouseController(); hTracker.addNewFrameListener(listener);
Also, we need to have an active search for a hand gesture because we need to locate the position of the hands and we also have to search for another gesture for the mouse click. So we need to call the nite::HandTracker::startGestureDetection() method twice for both the Click (also known as push) and Hand Raise gestures here:
hTracker.startGestureDetection(nite::GESTURE_HAND_RAISE); hTracker.startGestureDetection(nite::GESTURE_CLICK);
At the end, we will wait until the user presses the Enter key to end the app. We do nothing more in our main thread except just waiting. Everything happens in another thread.
ReadLastCharOfLine(); nite::NiTE::shutdown(); openni::OpenNI::shutdown(); return 0;
Summary
In this article, we learnt how to write a working example for using nite::HandTracker, controlled the position of the mouse cursor using the NiTE hand tracker feature, and simulated a click event
Resources for Article:
Further resources on this subject:
- Getting started with Kinect for Windows SDK Programming [Article]
- Kinect in Motion – An Overview [Article]
- Active Directory migration [Article]
