Resilient Remote Heart Rate Measurement from Partially Masked Faces

The remote heart rate measurement aims to use the remote photoplethysmography (rPPG) technology to extract the heart rate information from face videos in a non-contact manner. Most of existing methods treat each part of the face image equally and usually have significant performance drop in some real-world application scenarios, where the face area is partially masked or occluded. To address this challenge, we propose a part-based resilient remote heart rate measurement algorithm, namely PBHeart, which is based on the fine-grained estimation of the effectiveness of different parts in a face. In particular, we apply the 3D convolutional neural network to process face frames and evenly divide the feature map into adjacent parts, each of which is used to predict the rPPG signal and the heart rate jointly. Meanwhile, we measure the effectiveness of each part of face based on the precision of prediction and the cluster based reliability of model. Finally, the prediction results from each part of face is weighted by the effectiveness, and integrated to achieve the final result. Comprehensive experiments based on three real-world datasets are conducted to prove the state-of-the-art performance of PBHeart, especially for partially masked faces. More importantly, we have collected and will release a new masked face dataset namely FMD, which includes more than 1 hour videos from 22 volunteers, to support further research in this field.