Loading [a11y]/accessibility-menu.js
Recognizing Micro-Actions and Reactions from Paired Egocentric Videos | IEEE Conference Publication | IEEE Xplore

Recognizing Micro-Actions and Reactions from Paired Egocentric Videos


Abstract:

We aim to understand the dynamics of social interactions between two people by recognizing their actions and reactions using a head-mounted camera. Our work will impact s...Show More

Abstract:

We aim to understand the dynamics of social interactions between two people by recognizing their actions and reactions using a head-mounted camera. Our work will impact several first-person vision tasks that need the detailed understanding of social interactions, such as automatic video summarization of group events and assistive systems. To recognize micro-level actions and reactions, such as slight shifts in attention, subtle nodding, or small hand actions, where only subtle body motion is apparent, we propose to use paired egocentric videos recorded by two interacting people. We show that the first-person and second-person points-of-view features of two people, enabled by paired egocentric videos, are complementary and essential for reliably recognizing micro-actions and reactions. We also build a new dataset of dyadic (two-persons) interactions that comprises more than 1000 pairs of egocentric videos to enable systematic evaluations on the task of micro-action and reaction recognition.
Date of Conference: 27-30 June 2016
Date Added to IEEE Xplore: 12 December 2016
ISBN Information:
Electronic ISSN: 1063-6919
Conference Location: Las Vegas, NV, USA

1. Introduction

The dynamics of social interactions between two people can be decomposed into a sequence of action and reaction pairs (such as pointing and sharing a point of attention, gesturing and nodding in agreement, or laughing and gesturing disagreement) to convey to each other a sense of their internal states. Our everyday interactions even include micro-actions and micro-reactions in which only subtle body motion is apparent, such as slight changes in focus of attention (small movement of the head in response to pointing), subtle nodding, or small hand actions. The ability to understand interaction dynamics with such micro-behaviors is important for human-to-human communications, as this mode of non-verbal communication is perhaps our primary means of understanding and expressing our internal state. Towards understanding the deeper complexities of social interaction dynamics, this work attempts to take the first step by developing a method to recognize micro-actions and reactions.

Challenges of recognizing micro-actions. Slight head motion of person induces only slight local motion in the person 's points-of-view in (1) and (2). Hand motion by person is difficult to observe from the 's points-of-view in (2) and (3).

References

References is not available for this document.