Attentiveness Map Estimation for Haptic Teleoperation of Mobile Robot Obstacle Avoidance and Approach

Haptic feedback can improve safety of teleoperated robots when situational awareness is limited or operators are inattentive. Standard potential field approaches increase haptic resistance as an obstacle is approached, which is desirable when the operator is unaware of the obstacle but undesirable when the movement is intentional, such as when the operator wishes to inspect or manipulate an object. This letter presents a novel haptic teleoperation framework that estimates the operator's attentiveness to obstacles and dampens haptic feedback for intentional movement. A biologically-inspired attention model is developed based on computational working memory theories to integrate visual saliency estimation with spatial mapping. The attentiveness map is generated in real-time, and our system renders lower haptic forces for obstacles that the operator is estimated to be aware of. Experimental results in simulation show that the proposed framework outperforms haptic teleoperation without attentiveness estimation in terms of task performance, robot safety, and user experience.

I. Introduction

Tteleoperation enables human operators to control mobile robots in complex environments in which human judgment and adaptability are necessary. However, operators may not be aware of obstacles, and proximity-based haptic feedback is a popular approach to inform an operator of obstacles that pose a risk of collision [1], [2]. Experiments have shown that haptic feedback is well-suited for alerting users to conditions that they are not visually aware of and require immediate response [3]. A disadvantage of such feedback is that it can distract or annoy an operator who is alert and attentive to an obstacle. Moreover, for mobile manipulators that must approach or make contact with an obstacle, such as reaching for items on tables and shelves or pushing furniture, we do not wish for the robot to avoid contact, but rather unintentional contact. For intentional movements toward an obstacle, repulsive haptic forces would “fight” against the operator, leading to control contention and frustration. Moreover, haptic proximity alerts such as vibrations would annoy and distract the operator. Experiments have shown that contentious haptic feedback both increases cognitive load and degrades task performance [4], [5]. Recent studies have addressed this issue using intent prediction, such as predicting intent from a predefined task set [6] or as a goal location [7]. However, it is difficult to infer intentions in open-world scenarios where the set of possible tasks is hard to define and intent is ambiguous.