University of Minnesota Driven to Discover
U of MNUniversity of Minnesota
Center for Transportation Studies

What are they looking at?

Understanding exactly what drivers are doing as they operate their vehicles is an important goal for intelligent transportation systems researchers. Because driving involves many sub-tasks and takes place in a complicated and constantly changing environment, finding out what drivers are doing is a challenge. But work at the University of Minnesota’s Department of Computer Science and Engineering could help researchers get a clearer picture.

Graduate student Eric Wahlstrom, research associate Osama Masoud, and professor Nikolaos Papanikolopoulos developed computer software that analyzes a series of images from a camera in the vehicle to determine the direction of the driver’s gaze or whether the driver is using a particular vehicle control.

This research relied on the Framework for Processing Video (FPV), original software developed by Masoud at the University of Minnesota. Masoud and Papanikolopoulos have made numerous contributions to intelligent transportation systems research at the University, and have been selected as CTS Faculty & Research Scholars.

To determine what a driver is looking at, the researchers realized, their system did not need to create a complete representation of a face. Instead, it first locates the lips, by matching a range of appropriate colors. Working from the mouth, the system then finds the locations of the eyes. Subsequent steps determine the position of the pupils and the relationship of each pupil to the corners of the eye. These measurements allow the system to determine not only the direction of the gaze, but roughly how far away the eyes are focused.

Other recent experiments using the system focused on detecting when a driver was using the control panel of the car stereo. A second camera (mounted outside a stationary vehicle) was used to monitor the control area; software detection algorithms determined whether the driver’s hand was active in the control area.

Future enhancements envisioned by the researchers include expanding the system to allow it to categorize different driver behaviors and track them over time, and improving the system’s error correction capabilities. The addition of a robust eye-tracking mechanism is also a possibility.

Monitoring Driver Activities, a final research report on the project, is available online.