Mask Scoring R-CNN

Letting a deep network be aware of the quality of its own predictions is an interesting yet important problem. In the task of instance segmentation, the confidence of instance classification is used as mask quality score in most instance segmentation frameworks. However, the mask quality, quantified as the IoU between the instance mask and its ground truth, is usually not well correlated with classification score. In this paper, we study this problem and propose Mask Scoring R-CNN which contains a network block to learn the quality of the predicted instance masks. The proposed network block takes the instance feature and the corresponding predicted mask together to regress the mask IoU. The mask scoring strategy calibrates the misalignment between mask quality and mask score, and improves instance segmentation performance by prioritizing more accurate mask predictions during COCO AP evaluation. By extensive evaluations on the COCO dataset, Mask Scoring R-CNN brings consistent and noticeable gain with different models and outperforms the state-of-the-art Mask R-CNN. We hope our simple and effective approach will provide a new direction for improving instance segmentation. The source code of our method is available at \url{https://github.com/zjhuang22/maskscoring_rcnn}.

1. Introduction

Deep networks are dramatically driving the development of computer vision, leading to a series of state-of-the-art in tasks including classification [22], [16], [35], object detection [12], [17], [32], [27], [33], [34], semantic segmentation [28], [4], [37], [18] etc. From the development of deep learning in computer vision, we can observe that the ability of deep networks is gradually growing from making image-level prediction [22] to region/box-level prediction [12], pixel-level prediction [28] and instance/mask-level prediction [15]. The ability of making fine-grained predictions requires not only more detailed labels but also more delicate network designing.