Real-Time End-to-End Portrait and In-Hand Object Segmentation with Background Fusion

Portrait segmentation has become increasingly important for video conferencing software. However, existing methods fail to produce consistent and accurate segmentation of in-hand objects due to the weak connection in pixel distribution between humans and the in-hand objects. In this paper, we propose a fast and lightweight end-to-end model that performs portrait and in-hand object segmentation simultaneously. Our design takes spatial and temporal information into account by using a model containing a Convolution-Transformer hybrid encoder and a recurrent decoder, followed by a background fusion refinement module to produce accurate and consistent segmentation results. Our experimental results show improvements both visually and quantitatively compared to existing real-time video portrait segmentation with respect to accuracy, consistency, and speed.