PanoGlassNet: Glass Detection With Panoramic RGB and Intensity Images

Glass detection is an important and challenging task for many vision systems, such as 3-D reconstruction, autonomous driving, and depth estimation. However, to the best of our knowledge, almost all existing glass detection methods based on deep learning are trained on perspective images that contain very few transparent glasses very close to the camera. This is not suitable for some tasks that require context relationships, wide field of view (FOV), and simultaneous detection of multiple objects at a certain distance, such as 3-D reconstruction, autonomous driving, and pedestrian trajectory prediction. To tackle this problem, we build a panoramic dataset for glass detection called PanoGlass, which contains panoramic images and intensity images; the panoramic images are manually labeled and provide an FOV four times larger than a perspective image and contain more transparent glasses. Furthermore, based on the PanoGlass dataset, we propose a glass detection method named PanoGlassNet, which captures the wide FOV and twisted boundary of panoramic images using our novel module large-field deformable contextual features (LDCFs). The module consists of four branches with different kernel sizes and deformable convolutions. Through extensive experiments, we demonstrate that PanoGlassNet not only achieves 86.14, 0.0069, and 0.9255 of IoU, MAE, and
$F$
-score on PanoGlass, respectively, but also achieves 94.10, 0.029, and 0.9690 of IoU, MAE, and
$F$
-score on RGBT. Besides, PanoGlassNet is comparable on some of the glass detection datasets and SOD-D datasets. Code and dataset are available at
https://github.com/AnyaTracy/PanoGlass
.