Depth Aware Portrait Segmentation Using Dual Focus Images

The rapid development of camera in hand-held devices and the emergence of social media has led to an uprise in capturing self-portrait images. Augmenting these images for beautification or applying special effects to mimic DSLR camera has become a popular practice. Most of these effects require separation of foreground from background where the effect can be applied solely on background. To employ such effects on portrait (upper half of human body) images, a pixel-accurate segmentation is imperative. In this paper, we propose an effective method of fast depth aware CNN based portrait segmentation from monocular images. The proposed method is capable of being deployed on mobile phones, within the constraints of time and memory. On the segmented images, we demonstrate the application of bokeh effect, which blurs out-of-focus regions. We experiment with different combinations of state of the art encoder and decoder networks for segmentation and infer that our proposed method can improve the inference speed by 76 ms on mobile device while maintaining an accuracy of 97.0 %.

1. Introduction

Recent research conducted by Info Trends [1] demonstrated, that the number of images captured has increased exponentially in the past few years and is expected to reach about 1.2 trillion by 2017. It was also noted that 85% of such images were captured using mobile phone camera and 70% of these were from the mobile front camera [1]. In addition to capturing images with a mobile phone, augmenting different effects e.g. focus refinement, skin color matching to yield similar impact as a DSLR camera, has become widespread. However, limited by the hardware capability of a phone, image analysis techniques are employed to generate these effects.