(For more resources related to this topic, see here.)
What is multi-touch?
The genesis of multi-touch on Mac OS X was the ability to perform two finger scrolling on a trackpad. The technology was further refined on mobile touch screen devices such as the iPod Touch, iPhone, and iPad. And it has also matured on the Mac OS X platform to allow the use of multi-touch or magic trackpad combined with one or more fingers and a motion to interact with the computer. Gestures are intuitive and allow us to control what is on the screen with fluid motions. Some of the things that we can do using multi-touch are as follows:
-
Two finger scrolling: This is done by placing two fingers on the trackpad and dragging in a line
-
Tap or pinch to zoom : This is done by tapping once with a single finger, or by placing two fingers on the trackpad and dragging them closer to each other
-
Swipe to navigate: This is done by placing one or more fingers on the trackpad and quickly dragging in any direction followed by lifting all the fingers
-
Rotate : This is done by placing two fingers on the trackpad and turning them in a circular motion while keeping them on the trackpad
But these gestures just touch the surface of what is possible with multi-touch hardware. The magic trackpad can detect and track all 10 of our fingers with ease. There are plenty of new things that can be done with multi-touch — we are just waiting for someone to invent them.
Implementing a custom view
Multi-touch events are sent to the NSView objects. So before we can invent that great new multi-touch thing, we first need to understand how to implement a custom view.
Essentially, a custom view is a subclass of NSView that overrides some of the behavior of the NSView object. Primarily, it will override the drawRect: method and some of the event handling methods.
Time for action — creating a GUI with a custom view
By now we should be familiar with creating new Xcode projects so some of the steps here are very high level. Let’s get started!
-
Create a new Xcode project with Automatic Reference Counting enabled and these options enabled as follows:
Option
Value
Product Name
Multi-Finger Paint
Company Identifier
com.yourdomain
Class Prefix
Your initials
-
After Xcode creates the new project, design an icon and drag it in to the App Icon field on the TARGET Summary.
-
Remember to set the Organization in the Project Document section of the File inspector.
-
Click on the filename MainMenu.xib in the project navigator.
-
Select the Multi-Finger Paint window and in the Size inspector change its Width and Height to 700 and 600 respectively.
-
Enable both the Minimum Size and Maximum Size Constraints values.
-
From the Object Library , drag a custom view into the window.
-
In the Size inspector , change the Width and Height of the custom view to 400 and 300 respectively.
-
Center the window using the guides that appear.
-
From the File menu, select New>, then select the File…option.
-
Select the Mac OS X Cocoa Objective-C class template and click on the Next button.
-
Name the class BTSFingerView and select subclass of NSView.
It is very important that the subclass is NSView. If we make a mistake and select the wrong subclass, our App won’t work.
-
Click on the button titled Create to create the .h and .m files.
-
Click on the filename BTSFingerView.m and look at it carefully. It should look similar to the following code:
// // BTSFingerView.m // Multi-Finger Paint // // Created by rwiebe on 12-05-23. // Copyright (c) 2012 BurningThumb Software. All rights reserved. // #import "BTSFingerView.h" @implementation BTSFingerView - (id)initWithFrame:(NSRect)frame { self = [super initWithFrame:frame]; if (self) { // Initialization code here. } return self; } - (void)drawRect:(NSRect)dirtyRect { // Drawing code here. } @end
-
By default, custom views do not receive events (keyboard, mouse, trackpad, and so on) but we need our custom view to receive events. To ensure our custom view will receive events, add the following code to the BTSFingerView.m file to accept the first responder:
/* ** - (BOOL) acceptsFirstResponder ** ** Make sure the view will receive ** events. ** ** Input: none ** ** Output: YES to accept, NO to reject */ - (BOOL) acceptsFirstResponder { return YES; }
-
And, still in the BTSFingerView.m file, modify the initWithFrame method to allow the view to accept touch events from the trackpad as follows:
- (id)initWithFrame:(NSRect)frame { self = [super initWithFrame:frame]; if (self) { // Initialization code here. // Accept trackpad events [self setAcceptsTouchEvents: YES]; } return self; }
-
Once we are sure our custom view will receive events, we can start the process of drawing its content. This is done in the drawRect: method. Add the following code to the drawRect: method to clear it with a transparent color and draw a focus ring if the view is first responder:
/* ** - (void)drawRect:(NSRect)dirtyRect ** ** Draw the view content ** ** Input: dirtyRect - the rectangle to draw ** ** Output: none */ - (void)drawRect:(NSRect)dirtyRect { // Drawing code here. // Preserve the graphics content // so that other things we draw // don't get focus rings [NSGraphicsContext saveGraphicsState]; // color the background transparent [[NSColor clearColor] set]; // If this view has accepted first responder // it should draw the focus ring if ([[self window] firstResponder] == self) { NSSetFocusRingStyle(NSFocusRingAbove); } // Fill the view with fully transparent // color so that we can see through it // to whatever is below [[NSBezierPath bezierPathWithRect:[self bounds]] fill]; // Restore the graphics content // so that other things we draw // don't get focus rings [NSGraphicsContext restoreGraphicsState]; }
-
Next, we need to go back into the .xib file, and select our custom view, and then select the Identity Inspector where we will see that in the section titled Custom Class, the Class field contains NSView as the class.
-
Finally, to connect this object to our new custom view program code, we need to change the Class to BTSFingerView as shown in the following screenshot:
What just happened?
We created our Xcode project and implemented a custom NSView object that will receive events. When we run the project we notice that the focus ring is drawn so that we can be confident the view has accepted the firstResponder status.
How to receive multi-touch events
Because our custom view accepts first responder, the Mac OS will automatically send events to it. We can override the methods that process the events that we want to handle in our view. Specifically, we can override the following events and process them to handle multi-touch events in our custom view:
-
– (void)touchesBeganWithEvent:(NSEvent *)event
-
– (void)touchesMovedWithEvent:(NSEvent *)event
-
– (void)touchesEndedWithEvent:(NSEvent *)event
-
– (void)touchesCancelledWithEvent:(NSEvent *)event
Time for action — drawing our fingers
When the multi-touch or magic trackpad is touched, our custom view methods will be invoked and we will be able to draw the placement of our fingers on the trackpad in our custom view.
-
In Xcode, click on the filename BTSFingerView.h in the project navigator and add the following highlighted property:
// // BTSFingerView.h // Multi-Finger Paint // // Created by rwiebe on 12-05-23. // Copyright (c) 2012 BurningThumb Software. All rights reserved. // #import <Cocoa/Cocoa.h> @interface BTSFingerView : NSView // A reference to the object that will // store the currently active touches @property (strong) NSMutableDictionary *m_activeTouches; @end
-
In Xcode, click on the file BTSFingerView.m in the project navigator and add the following program code to synthesize the property:
// // BTSFingerView.m // Multi-Finger Paint // // Created by rwiebe on 12-05-23. // Copyright (c) 2012 BurningThumb Software. All rights reserved. // #import "BTSFingerView.h" @implementation BTSFingerView // Synthesize the object that will // store the currently active touches @synthesize m_activeTouches;
-
Add the following code to the initWithFrame: method in the BTSFingerView.m file to create the dictionary object that will be used to store the active touch objects:
- (id)initWithFrame:(NSRect)frame { self = [super initWithFrame:frame]; if (self) { // Initialization code here. // Create the mutable dictionary that // will hold the list of currently active // touch events m_activeTouches = [[NSMutableDictionary alloc] init]; } return self; }
-
Add the following code to the BTSFingerView.m file to add BeganWith touch events to the dictionary of active touches:
/** ** - (void)touchesBeganWithEvent:(NSEvent *)event ** ** Invoked when a finger touches the trackpad ** ** Input: event - the touch event ** ** Output: none */ - (void)touchesBeganWithEvent:(NSEvent *)event { // Get the set of began touches NSSet *l_touches = [event touchesMatchingPhase:NSTouchPhaseBegan inView:self]; // For each began touch, add the touch // to the active touches dictionary // using its identity as the key for (NSTouch *l_touch in l_touches) { [m_activeTouches setObject:l_touch forKey:l_touch. identity]; } // Redisplay the view [self setNeedsDisplay:YES]; }
-
Add the following code to the BTSFingerView.m file to add moved touch events to the dictionary of active touches:
/** ** - (void)touchesMovedWithEvent:(NSEvent *)event ** ** Invoked when a finger moves on the trackpad ** ** Input: event - the touch event ** ** Output: none */ - (void)touchesMovedWithEvent:(NSEvent *)event { // Get the set of move touches NSSet *l_touches = [event touchesMatchingPhase:NSTouchPhaseMoved inView:self]; // For each move touch, update the touch // in the active touches dictionary // using its identity as the key for (NSTouch *l_touch in l_touches) { // Update the touch only if it is found // in the active touches dictionary if ([m_activeTouches objectForKey:l_touch.identity]) { [m_activeTouches setObject:l_touch forKey:l_touch.identity]; } } // Redisplay the view [self setNeedsDisplay:YES]; }
-
Add the following code to the BTSFingerView.m file to remove the touch from the dictionary of active touches when the touch ends:
/** ** - (void)touchesEndedWithEvent:(NSEvent *)event ** ** Invoked when a finger lifts off the trackpad ** ** Input: event - the touch event ** ** Output: none */ - (void)touchesEndedWithEvent:(NSEvent *)event { // Get the set of ended touches NSSet *l_touches = [event touchesMatchingPhase:NSTouchPhaseEnded inView:self]; // For each ended touch, remove the touch // from the active touches dictionary // using its identity as the key for (NSTouch *l_touch in l_touches) { [m_activeTouches removeObjectForKey:l_touch.identity]; } // Redisplay the view [self setNeedsDisplay:YES]; }
-
Add the following code to the BTSFingerView.m file to remove the touch from the dictionary of active touches when the touch is cancelled:
/** ** - (void)touchesCancelledWithEvent:(NSEvent *)event ** ** Invoked when a touch is cancelled ** ** Input: event - the touch event ** ** Output: none */ - (void)touchesCancelledWithEvent:(NSEvent *)event { // Get the set of cancelled touches NSSet *l_touches = [event touchesMatchingPhase:NSTouchPhaseCancelled inView:self]; // For each cancelled touch, remove the touch // from the active touches dictionary // using its identity as the key for (NSTouch *l_touch in l_touches) { [m_activeTouches removeObjectForKey:l_touch.identity]; } // Redisplay the view [self setNeedsDisplay:YES]; }
-
When we touch the trackpad we are going to draw a “finger cursor” in our custom view. We need to decide how big we want that cursor to be and the color that we want the cursor to be. Then we can add a series of #define to the file named BTSFingerView.h to define that value:
// Define the size of the cursor that // will be drawn in the view for each // finger on the trackpad #define D_FINGER_CURSOR_SIZE 20 // Define the color values that will // be used for the finger cursor #define D_FINGER_CURSOR_RED 1.0 #define D_FINGER_CURSOR_GREEN 0.0 #define D_FINGER_CURSOR_BLUE 0.0 #define D_FINGER_CURSOR_ALPHA 0.5
-
Now we can add the program code to our drawRect: implementation that will draw the finger cursors in the custom view.
// For each active touch for (NSTouch *l_touch in m_activeTouches.allValues) { // Create a rectangle reference to hold the // location of the cursor NSRect l_cursor; // Determine where the touch point NSPoint l_touchNP = [l_touch normalizedPosition]; // Calculate the pixel position of the touch point l_touchNP.x = l_touchNP.x * [self bounds].size.width; l_touchNP.y = l_touchNP.y * [self bounds].size.height; // Calculate the rectangle around the cursor l_cursor.origin.x = l_touchNP.x - (D_FINGER_CURSOR_SIZE / 2); l_cursor.origin.y = l_touchNP.y - (D_FINGER_CURSOR_SIZE / 2); l_cursor.size.width = D_FINGER_CURSOR_SIZE; l_cursor.size.height = D_FINGER_CURSOR_SIZE; // Set the color of the cursor [[NSColor colorWithDeviceRed: D_FINGER_CURSOR_RED green: D_FINGER_CURSOR_GREEN blue: D_FINGER_CURSOR_BLUE alpha: D_FINGER_CURSOR_ALPHA] set]; // Draw the cursor as a circle [[NSBezierPath bezierPathWithOvalInRect: l_cursor] fill]; }
What just happened?
We implemented the methods required to keep track of the touches and to draw the location of the touches in our custom view. If we run the App now, and move the mouse pointer over the view area, and then touch the trackpad, we will see red circles that track our fingers being drawn in the view as shown in the following screenshot:
What is an NSBezierPath?
A Bezier Path consists of straight and curved line segments that can be used to draw recognizable shapes. In our program code, we use Bezier Paths to draw a rectangle and a circle but a Bezier Path can be used to draw many other shapes.
How to manage the mouse cursor
One of the interesting things about the trackpad and the mouse is the association between a single finger touch and the movement of the mouse cursor. Essentially, Mac OS X treats a single finger movement as if it was a mouse movement. The problem with this is that when we move just a single finger on the trackpad, the mouse cursor will move away from our NSView causing it to lose focus so that when we lift our finger we need to move the mouse cursor back to our NSView to receive touch events.
Time for action — detaching the mouse cursor from the mouse hardware
The solution to this problem is to detach the mouse cursor from the mouse hardware (typically called capturing the mouse) whenever a touch event is active so that the cursor is not moved by touch events. In addition, since a “stuck” mouse cursor may be cause for concern to our App user, we can hide the mouse cursor when touches are active.
-
In Xcode, click on the file named BTSFingerView.h in the project navigator and add the following flag to the interface:
@interface BTSFingerView : NSView { // Define a flag so that touch methods can behave // differently depending on the visibility of // the mouse cursor BOOL m_cursorIsHidden; }
-
In Xcode, click on the file named BTSFingerView.m in the project navigator.
-
Add the following code to the beginning of the touchesBeganWithEvent: method to detach and hide the mouse cursor when a touch begins. We only want to do this one time so it is guarded by a BOOL flag and an if statement to make sure we don’t do it for every touch that begins.
- (void)touchesBeganWithEvent:(NSEvent *)event { // If the mouse cursor is not already hidden, if (NO == m_cursorIsHidden) { // Detach the mouse cursor from the mouse // hardware so that moving the mouse (or a // single finger) will not move the cursor CGAssociateMouseAndMouseCursorPosition(false); // Hide the mouse cursor [NSCursor hide]; // Remember that we detached and hid the // mouse cursor m_cursorIsHidden = YES; }
-
Add the following code to the end of the touchesEndedWithEvent: method to attach and unhide the mouse cursor when all touches end. We use a BOOL flag to remember the state of the cursor so that the touchesBeganWithEvent: method will re-hide it when the next touch begins.
// If there are no remaining active touches if (0 == [m_activeTouches count]) { // Attach the mouse cursor to the mouse // hardware so that moving the mouse (or a // single finger) will move the cursor CGAssociateMouseAndMouseCursorPosition(true); // Show the mouse cursor [NSCursor unhide]; // Remember that we attached and unhid the // mouse cursor so that the next touch that // begins will detach and hide it m_cursorIsHidden = NO; } // Redisplay the view [self setNeedsDisplay:YES]; }
-
Add the following code to the end of the touchesCancelledWithEvent: method to attach and unhide the mouse cursor when all touches end. We use a BOOL flag to remember the state of the cursor so that the touchesBeganWithEvent: method will re-hide it when the next touch begins.
// If there are no remaining active touches if (0 == [m_activeTouches count]) { // Attach the mouse cursor to the mouse // hardware so that moving the mouse (or a // single finger) will move the cursor CGAssociateMouseAndMouseCursorPosition(true); // Show the mouse cursor [NSCursor unhide]; // Remember that we attached and unhid the // mouse cursor so that the next touch that // begins will detach and hide it m_cursorIsHidden = NO; } // Redisplay the view [self setNeedsDisplay:YES]; }
-
While we are looking at the movement of the mouse, we also notice that the focus ring for our custom view is being drawn regardless of whether or not the mouse cursor is over our view. Since touch events will only be sent to our view if the mouse cursor is over it, we want to change the program code so that the focus ring only appears when the mouse cursor is over the custom view. This is something we can do with another BOOL flag. Add the following code to the file to define a BOOL flag that will allow us to determine if the mouse cursor is over our custom view:
// Define a flag so that view methods can behave // differently depending on the position of the // mouse cursor BOOL m_mouseIsInFingerView;
-
In the file named BTSFingerView.m, add the following code to create a tracking rectangle that matches the bounds of our custom view. Once the tracking rectangle is active, the methods mouseEntered: and mouseExited: will be automatically invoked as the mouse cursor enters and exits our custom view.
/** ** - (void)viewDidMoveToWindow ** ** Informs the receiver that it has been added to ** a new view hierarchy. ** ** We need to make sure the view window is valid ** and when it is, we can add the tracking rect ** ** Once the tracking rect is added the mouseEntered: ** and mouseExited: events will be sent to our view ** */ - (void)viewDidMoveToWindow { // Is the views window valid if ([self window] != nil) { // Add a tracking rect such that the // mouseEntered; and mouseExited: methods // will be automatically invoked [self addTrackingRect:[self bounds] owner:self userData:NULL assumeInside:NO]; } }
-
In the file named BTSFingerView.m, add the following code to implement the mouseEntered: and mouseExited: methods. In those methods, we set the BOOL flag so that the drawRect: method knows whether or not to draw the focus ring.
/** ** - (void)mouseEntered: ** ** Informs the receiver that the mouse cursor ** entered a tracking rectangle ** ** Since we only have a single tracking rect ** we know the mouse is over our custom view ** */ - (void)mouseEntered:(NSEvent *)theEvent { // Set the flag so that other methods know // the mouse cursor is over our view m_mouseIsInFingerView = YES; // Redraw the view so that the focus ring // will appear [self setNeedsDisplay:YES]; } /** ** - (void)mouseExited: ** ** Informs the receiver that the mouse cursor ** exited a tracking rectangle ** ** Since we only have a single tracking rect ** we know the mouse is not over our custom view ** */ - (void)mouseExited:(NSEvent *)theEvent { // Set the flag so that other methods know // the mouse cursor is not over our view m_mouseIsInFingerView = NO; // Redraw the view so that the focus ring // will not appear [self setNeedsDisplay:YES]; }
-
Finally, in the drawRect: method, change the program code that draws the focus ring to only do so if the mouse cursor is in the tracking rectangle:
// If this view has accepted first responder // it should draw the focus ring but only if // the mouse cursor is over this view if ( ([[self window] firstResponder] == self) && (YES == m_mouseIsInFingerView) ) { NSSetFocusRingStyle(NSFocusRingAbove); }
What just happened?
We implemented the program code that will prevent the mouse cursor from moving out of our custom view when touch events are active. In doing so we noticed that our focus ring behavior could be improved. Therefore we added additional program code to ensure the focus ring is visible only when the mouse pointer is over our view.
Performing 2D drawing in a custom view
Mac OS X provides a number of ways to perform drawing. The methods provided range from very simple methods to very complex methods. For our multi-finger painting program we are going to use the core graphics APIs designed to draw a path. We are going to collect each stroke as a series of points and construct a path from those points so that we can draw the stroke.
Each active touch event will have a corresponding active stroke object that needs to be drawn in our custom view. When a stroke is finished, and the App user lifts the finger, we are going to send the finished stroke to another custom view so that it is drawn only one time and not each time fingers move. The optimization of using the second view will ensure our finger tracking is not slowed down too much by drawing.
Before we can begin drawing, we need to create two new objects that will be used to store individual points and strokes. The program code for these two objects is not shown but the objects are included in the Multi-Finger Paint Xcode project. The two objects are as follows:
-
BTSPoint
-
BTSStroke
The BTSPoint object is a wrapper for an NSPoint structure. The NSPoint structure needs to be wrapped in an object so that it can be stored in an NSArray object. It has a single instance variable:
NSPoint m_point;
It implements the following methods which allows it to be initialized: return the point (x and y), return just the x value, or return just the y value. For more information on the object, we can read the source code file in the project:
- (id) initWithNSPoint:(NSPoint)a_point; - (NSPoint) point; - (CGFloat)x; - (CGFloat)y;
The BTSStroke object is a wrapper for an array of BTSPoint objects, a color, and a stroke width. It is used to store strokes that are drawn in our custom NSView. It has the following instance variables and properties:
float m_red; float m_green; float m_blue; float m_alpha; float m_width; @property (strong) NSMutableArray *m_points;
It implements the following methods which allows it to be initialized: a new point to be added, return the array of points, return any of the color components, and return the stroke width. For more information on the object, we can read the source code file in the project:
- (id) initWithWidth:(float)a_width red:(float)a_red green:(float)a_green blue:(float)a_blue alpha:(float)a_alpha; - (void) addPoint:(BTSPoint *)a_point; - (NSMutableArray *) points; - (float)red; - (float)green; - (float)blue; - (float)alpha; - (float)width;
