Visual interpretation of natural pointing gestures in 3D space for human-robot interaction

Visual interpretation of natural pointing gestures is essential in a human robot interaction scenario. Both hands and head are involved in pointing behaviors. Given the color images acquired by a web camera and the depth data by a TOF range camera, we perform visual tracking of the head and hands in 3D space. We investigate both the Head-Finger Line (HFL) and the forearm orientation as the estimation of the pointing direction. HFL is determined by 3D positions of the face and finger tip. Forearm direction is calculated using the PCA method in the RANSAC framework. Their performances are evaluated and compared in the experiments. Face direction and eye gaze orientation provide important cues regarding where the person's attention is during a pointing operation, which is proven helpful for eliminating some false estimations in our experiment.