MUVA: A New Large-Scale Benchmark for Multi-view Amodal Instance Segmentation in the Shopping Scenario

Amodal Instance Segmentation (AIS) endeavors to accurately deduce complete object shapes that are partially or fully occluded. However, the inherent ill-posed nature of single-view datasets poses challenges in determining occluded shapes. A multi-view framework may help alleviate this problem, as humans often adjust their perspective when encountering occluded objects. At present, this approach has not yet been explored by existing methods and datasets. To bridge this gap, we propose a new task called Multi-view Amodal Instance Segmentation (MAIS) and introduce the MUVA dataset, the first MUlti-View AIS dataset that takes the shopping scenario as instantiation. MUVA provides comprehensive annotations, including multi-view amodal/visible segmentation masks, 3D models, and depth maps, making it the largest image-level AIS dataset in terms of both the number of images and instances. Additionally, we propose a new method for aggregating representative features across different instances and views, which demonstrates promising results in accurately predicting occluded objects from one viewpoint by leveraging information from other viewpoints. Besides, we also demonstrate that MUVA can benefit the AIS task in real-world scenarios. ¹

1. Introduction

The amodal instance segmentation (AIS) task aims to determine an object’s entire shape, encompassing its visible and occluded components. AIS task is more challenging than the visible instance segmentation task [23], [11] as it lacks occluded region appearance. Despite its complexity, the AIS task has significant implications for various industrial applications that encounter occlusion problems, such as robotic arm grasping [1], pedestrian re-identification [34], [36], [29], automatic driving [27], [28], and self-checkout systems in supermarkets [8].