MaskWarp: Visuo-Haptic Illusions in Mixed Reality using Real-time Video Inpainting

We present MaskWarp, a novel approach to enable haptic retargeting and other visuo-haptic illusions in video see-through (VST) mixed reality (MR). Virtual reality provides a unique ability to shift where users perceive their virtual hands. However, in VST MR, users can see their real hands through a video feed, breaking the illusion. MaskWarp uses real-time video inpainting to remove the physical hand from the video, which is then replaced with a virtual hand. The virtual hand position can then be warped for visuo-haptic illusions. We use haptic retargeting to demonstrate this technique in an engaging cups and ball game.