Opengazer: open-source gaze tracker for ordinary webcams

Opengazer is an open source application that uses an ordinary webcam to estimate the direction of your gaze. This information can then be passed to other applications. For example, used in conjunction with Dasher, opengazer allows you to write with your eyes. Opengazer aims to be a low-cost software alternative to commercial hardware-based eye trackers.

The first version of Opengazer was developed by Piotr Zieliński, supported by Samsung and the Gatsby Charitable Foundation. More detail about this version can be found [here].

Research for Opengazer has been revived by Emli-Mari Nel, and is now supported by the European Commission in the context of the AEGIS project and the Gatsby Charitable Foundation.

Current work

Head tracking

The previous version of Opengazer is very sensitive to head-motion variations. To rectify this problem we are currently focussing on head tracking algorithms to correct for head pose variations before inferring the gaze positions. All the software is written in C++ and Python. An example video of one of our head tracking algorithm can be downloaded [here]. On Windows the video can be viewed with the VLC player. On Linux it is best displayed using Mplayer Movie Player.

The first version of our head tracking algorithm is an elementary one, based on the Viola-Jones face detector, that locates the largest face in the video stream (captured from a file/camera) as fast as possible, on a frame-by-frame basis. The xy-coordinates from tracking can already be used to type using Dasher. This can be done in 1D mode (e.g., from tracking just the y-coordinates), or in 2D mode. Although much better results can be expected after the release of our head-pose software, this software is already useful for fast face localisation. Our algorithm applies a simple autoregressive lowpass filter on the xy-coordinates and scale of the detection results from the Viola-Jones face detector, and also restricts the region of interest from frame to frame. The detection parameters have been determined according to our specific application (i.e., a single user working on his/her Desktop PC/laptop). The algorithm works best on 320x240 images, at a frame rate of 30 fps, and reasonable lighting conditions.

Downloads

[Head tracker version 0.0] This is the first version and includes installation instructions, and documentation on upcoming releases and current research.

Notes to developers/collaborators

We welcome collaboration with other opensource developers. If eager developers flood us with patches we will consider hosting a repository. To send patches follow these instructions in addition to the installation instructions provided in the download.

Gesture switch

A subproject of Opengazer involves the automatic detection of facial gestures to drive a switch-based program. This program has a short learning phase (under 30 seconds) for each gesture, after which the gesture is automatically detected. Many patients (e.g., patients with cerebral palsy) have involuntary head motions that can introduce false positives during detection. We therefore also train a background model to deal with involuntary motions. All the software is written in C++ and Python and will be available for download soon. An example video of our gesture switch algorithm can be downloaded [here]. On Windows the video can be viewed with the VLC player. On Linux it is best displayed using Mplayer Movie Player. Note that this video has sound. Three gestures have been trained to generate three possible switch events: a left smile, right smile, and upwards eyebrow movement all correspond to switch events. The background model, in this case, detects blinks, sudden changes in lighting, and large head motions. The first official release will be at the end of June 2012.

Previous work

Overview

The first version of Opengazer has the following workflow:

1. Feature point selection. During startup the user is expected to select feature points on the face using the mouse. These points are tracked in subsequent steps in the algorithm. The first two points correspond to the corners of the eyes, which are also used in subsequent steps to extract the eye images. The user has to keep his/her head very still during the whole procedure. At this point the user should preferably save all the selected feature points.
2. Calibrating the system. During this step a few red dots are displayed at various positions on the screen. Images of the eyes are extracted during the display of each dot. The eye images and their corresponding dot positions are used to train a Gaussian Process that represents the mapping between the image of an eye and the position on the screen. Note that changes in head pose often requires the system to be recalibrated.
3. Tracking. After all the calibration dots are processed the Gaussian Process yields a predictive distribution, so that the expected point of eye focus on the display monitor can be estimated given a new image of the eye. If the user loads his/her selected feature points, the Viola Jones face detector is used once to determine the search region where the feature points are expected. Subsequently, optical flow is used to track each feature point. The eye images are extracted (using the first two selected corner points as reference), and the gaze is predicted using the extracted eye images and the trained Gaussian Process.

1. S. Xu, H. Jiang, F. C. Lau, User-oriented document summarization through vision-based eye-tracking, in IU09: Proceedings of the 13th international conference on Intelligent user interfaces, ACM, New York, NY, USA, 2008, pp. 7-16.
2. S. Xu, H. Jiang, F. C. Lau, Personalized online document, image and video recommendation via commodity eye-tracking, in: RecSys08: Proceedings of the 2008 ACM conference on Recommender systems, ACM, New York, NY, USA, 2008, pp. 83-90.

Video

This short video shows opengazer in action, using a £50 Logitech QuickCam Pro 4000, with the resolution 640x480. The distance from the camera/screen to the user was about 50cm.

The user first selects "load points", which loads and matches a previously selected set of point trackers on the face, which allows opengazer to extract the image of the eye and compute head orientation. Then, the "calibrate" routine displays a series of red points on the screen, at which the user is asked to look. As the calibration progresses, the current gaze estimate, represented by a small blue circle is getting better and better. Finally, the user selects "test", which displays a series of green points to test and show the accuracy of the gaze tracking.

A higher resolution version of the video

Download

The first prerelease of opengazer is available to download. Note that this version is DEPRECATED, and the next version will not be released before December 2012.

Download opengazer 0.1.2 for Linux (source)

You will need to compile opengazer yourself. Consult the detailed instructions for external dependencies and usage.

Opengazer is now stored in the subversion repository accessible from the sourceforge opengazer page. To check out the latest revision consult the opengazer instructions as well as the sourceforge subversion manual.

Opengazer is licensed under GPL version 2.

System requirements

• the processor speed of at least 2GHz
• a webcam (see below)
• all required libraries installed

Recommended webcams

More info

Macs

Opengazer 0.1 successfully follows my gaze in this poorly lit room, via an iSight camera plugged into an iBook. Works just fine using everything from current darwinports/macports, together with the latest release of vxl, which unfortunately required "make -k install" to work on the mac. One thing that took a while to figure out was a few linking problems with the osx port of vxl, this forced me to add -lnetlib to the LINKER line. Finally, I had to remove the gtk-config call from the Makefile as this doesn't seem to be present on my system, but even without this it linked just fine.

I've attached the Makefile in case the above bits are unclear. [see the note below]

Took a good 20 minutes of fiddling with the distance of the cam, and learning to keep my head still, until eventually I realized that it would crash on save unless I'd placed enough good points. However, while it is noisy (poor lighting here), it gets the position of my eye on the screen with an x/y resolution of, i'd say, 16x16, which seems easily enough for dasher.

Note: Alexander’s Makefile was for opengazer 0.1; I’ve changed it a bit to be compatible with opengazer 0.1.1, but haven’t tested it.

Connecting opengazer and Dasher

History

Opengazer grew out of the Machine Intelligence Laboratory in Cambridge University Engineering Department. VIM (`visual inference machine') was developed by Ollie Williams and Roberto Cipolla.