Leap Concepts

Coordinate System

The Ultraleap Hand Tracking employs a right-handed Cartesian coordinate system. The origin is centered at the top of the Ultraleap Hand Tracking Camera. The x- and z-axes lie in the plane of the camera sensors, with the x-axis running along the camera baseline. The y-axis is vertical, with positive values increasing upwards (in contrast to the downward orientation of most computer graphics coordinate systems). The z-axis has positive values increasing toward the user.

Hand Tracking Camera axes

The LeapC API measures physical quantities with the following units:

  • Distance: millimeters

  • Time: microseconds (unless otherwise noted)

  • Speed: millimeters/second

  • Angle: radians

Frame

A frame is the collection of tracking data derived from a single stereo-pair of images. The Ultraleap Tracking Service analyzes each pair of stereo images from the device sensors to detect the presense, posture, and motion of hands. This data is transmitted as a LEAP_TRACKING_EVENT() struct. The images used to create the frame of data can be requested separately.

The Ultraleap Hand Tracking Software analyzes the images and fits an internal model of the human hand to the observed scene. If parts of a hand are occluded, or out of view, the software estimates the data based on what is visible, past observations, and model constraints.

Get LEAP_TRACKING_EVENT() objects by calling LeapPollConnection() or LeapInterpolateFrame().

Frame Interpolation

Frame interpolation uses the nearest frame data to synthesize a new frame at a specified time. If this specified time is between two past frames, the data of these two frames is interpolated to synthesize an intermediate frame. If the specified time is more recent than the latest frame, then the new frame is synthesized by extrapolating the most recent data.

Extrapolation can reduce latency between the motion of the hand and the corresponding change in the graphical display – especially on systems that double or triple buffer rendering frames. However, extrapolating too far in the future can lead to jerky motion and other artifacts.

Request an interpolated frame with LeapInterpolateFrame(). Past tracking data is only kept for a few frames. If you attempt to interpolate too far in the past this function will return an error.

See Interpolated Frames for more information and Interpolated Frames Example for code samples.

Clock Rebasing

When requesting interpolated frames, you must specify the time for the desired frame. This time must be provided in terms of the internal clock used by the Ultraleap Hand Tracking Software. You can use LeapGetNow() to get the current Leap time. If you are synchronizing frames with an application clock, you can use clock rebasing to synchronize the Leap and application clocks. When the two clocks are synchronized, you can use the function LeapRebaseClock() to translate the application clock to the Ultraleap hand tracking system clock.

To use clock rebasing, create a LEAP_CLOCK_REBASER() object with LeapCreateClockRebaser(). Maintain synchronization between the clocks by calling LeapUpdateRebase() as often as possible, ideally every time you request a frame of data.

See Interpolated Frames Example for code samples.

Hand

Hands are the primary focus of the Ultraleap tracking data. The LEAP_HAND() struct is the parent of the physical tracking data measured for an individual hand. In addition to its own position and orientation, the hand provides access to the objects representing the digits and arm attached to that hand.

Get hand data from a LEAP_TRACKING_EVENT() object.

Digit/Finger

Fingers are represented by the LEAP_DIGIT() struct.

Fingers are made up of four bones in the following order from hand to tip:

  1. metacarpal

  2. proximal phalange

  3. intermediate phalange

  4. distal phalange

One thing to note – the thumb uses the same bones as the other fingers, even though an anatomical thumb has no intermediate phalange. To compensate, the Ultraleap Hand Tracking model of the thumb has a zero-length metacarpal bone.

Ultraleap Hand Bones Model

Bone

The segments of the fingers are represented by LEAP_BONE() structs. There are four bones per finger:

  • metacarpal

  • proximal phalange

  • intermediate phalange

  • distal phalange

A bone has two endpoints, a quaternion representing its orientation, and a width. Other physical quantities, such as length and direction, can be calculated from these values.

Arm

The arm is represented by the LEAP_BONE() struct. Given the centrality of the hand to the Ultraleap Hand Tracking system, the arm is a child of the hand, rather than vice versa.

Vector

The LEAP_VECTOR() struct is a three-element vector used to store positions, directions, velocities, and other coordinates. The LEAP_VECTOR() struct has no functions for performing vector math.

Quaternion

The LEAP_QUATERNION() struct is a four-element vector used to store rotations. The LEAP_QUATERNION() struct has no functions for performing quaternion operations.

Images

Ultraleap Hand Tracking Cameras use two monochrome IR cameras. You can request images by setting the Images Policy flag. When the images are received, the LEAP_IMAGE_EVENT() struct is provided by the LeapPollConnection() function. The image event also contains image properties, such as width, height, and the distortion map.

The image buffer is filled in with images from both cameras. The left image is first, followed by the right. Current devices use one byte per pixel, representing the recorded brightness value at that pixel. Image resolution can change dynamically and varies by device type, settings, and operating modes.

The format and number of bytes per pixel of an image are provided by the LEAP_IMAGE_PROPERTIES() struct.

See Using Images.

Distortion Map

The distortion map can be used to rectify an image so that the brightness value of a pixel represents the true value entering the camera after correcting for lens distortions. The distortion map is a 64 x 64 grid of 32bit floating point coordinates. Each point in the map has an x and y coordinate that represents the location in the image data to find the true brightness value. For points in between the grid centers, you can use bilinear interpolation. In fact, the distortion map is defined this way to take advantage of GPU texture map interpolation. Rectifying an image without a GPU is generally too slow for real-time systems. However, you can use the LeapPixelToRectilinear() function to rectify individual points.

See Image Distortion.

Policy

Policies are Ultraleap Hand Tracking system settings that can affect not just your own application, but any running Leap-enabled application. There are currently three defined policies (enumerated by the eLeapPolicyFlag() enum):

  • Background frames policy: whether the current application should access tracking data when it is not the focused, frontmost application on the computer. This policy is controlled by the Allow Background Apps setting in the Leap Motion Control Panel. Note that the LeapC library does not enforce this policy – tracking frames are provided whether or not this policy is set. You can use the current value of the policy to indicate whether the user wants apps to get frames in the background.

  • Images policy: whether the current application should receive events containing the current device’s stereo image pair.

  • Optimize for HMD policy: whether the Ultraleap Tracking Service expects to see hands from the HMD or the desktop perspective. Matching this policy to the current perspective greatly improves the quality of tracking. For example setting the policy to true when using a head-mounted display, and false when using the device placed on a desktop. This policy affects all applications that use Ultraleap Tracking data.

  • Allow pause and resume policy: whether applications can individually pause and resume tracking. This policy affects all applications that use Ultraleap Tracking data. It is generally unnecessary for an individual application to control service tracking. You can simply ignore incoming frames to effectively “pause” tracking within the scope of an application – and this won’t prevent other applications from receiving data. Pausing tracking may decrease the CPU usage of the Ultraleap Tracking Service, so consider pausing tracking on resource-constrained devices.

Setting and getting policies is asynchronous and may fail. When you call LeapSetPolicyFlags to read or write a policy, LeapPollConnection() will provide a LEAP_POLICY_EVENT when the policy change has been processed. Compare the desired policy flags with the active policy flags in the LEAP_POLICY_EVENT struct to determine whether the policy change succeeded.

See Controlling Policies for information about setting and checking policies.

Config Settings

The Ultraleap Hand Tracking Software has a number of configuration settings. Some settings reflect the user options in the control panel, other settings reflect default (but changeable) configuration values assigned in the config.json files read at system startup, still others are experimental software switches intended for internal use only. Only use documented settings in your applications. Ultraleap may remove or change non-documented settings without notice.

Setting and reading configuration values is asynchronous and may fail. When you call LeapRequestConfigValue() to read a setting, LeapPollConnection() will provide a LEAP_CONFIG_RESPONSE_EVENT() when the value is available. Similarly, when you write a config value with LeapSaveConfigValue(), LeapPollConnection() provides a LEAP_CONFIG_CHANGE_EVENT(), which reports whether the change was successful.

Device

The LEAP_DEVICE_INFO() struct provides information about the capabilities of the attached Ultraleap camera devices. Rather than hard-coding the capabilities based on your current device, you can use the device info struct to provide a measure of reliance against future changes. The key capabilities include:

  • The baseline distance between the two cameras.

  • The horizontal field of view of the sensors.

  • The vertical field of view of the sensors.

  • The estimated reliable tracking range.

Connection

Ultraleap Hand Tracking Software operates as a Windows service. The LeapC library connects to this service when you create and open a connection.

See Getting Tracking Frames and Other Event Messages for more information about creating and opening connections. See Examples for code samples.

Events/Messages

As events occur in the Ultraleap Tracking Service, they are added to an event queue. LeapPollConnection() reads this queue and populates the LEAP_CONNECTION_MESSAGE() struct type relevant to the event. For example, when a tracking frame is generated by the service, LeapPollConnection() provides a LEAP_TRACKING_EVENT() embedded in the LEAP_CONNECTION_MESSAGE() struct. If you don’t poll the connection fast enough, messages will be discarded.

See Getting Tracking Frames and Other Event Messages for more information about accessing tracking events. See Examples for code samples.

Back to top