Accurate Human-Limb Segmentation in RGB-D Images for Intelligent Mobility Assistance Robots

Mobility impairment is one of the biggest challenges faced by elderly people in today's society. The inability to move about freely poses severe restrictions on their independence and general quality of life. This work is dedicated to developing intelligent robotic platforms that assist users to move without requiring a human attendant. This work was done in the context of an EU project involved in developing an intelligent robot for elderly user assistance. The robot is equipped with a Kinect sensor, and the vision component of the project has the responsibility of locating the user, estimating the user's pose, and recognizing gestures by the user. All these goals can take advantage of a method that accurately segments human-limbs in the colour (RGB) and depth (D) images captured by the Kinect sensor. We exploit recent advances in deep-learning to develop a system that performs accurate semantic segmentation of human limbs using colour and depth images. Our novel technical contributions are the following: 1) we describe a scheme for manual annotation of videos, that eliminates the need to annotate segmentation masks in every single frame, 2) we extend a state of the art deep learning system for semantic segmentation, to exploit diverse RGB and depth data, in a single framework for training and testing, 3) we evaluate different variants of our system and demonstrate promising performance, as well the contribution of diverse data, on our in-house Human-Limb dataset. Our method is very efficient, running at 8 frames per second on a GPU.