SOSNet: Real-Time Small Object Segmentation via Hierarchical Decoding and Example Mining

Real-time semantic segmentation plays an important role in auto vehicles. However, most real-time small object segmentation methods fail to obtain satisfactory performance on small objects, such as cars and sign symbols, since the large objects usually tend to devote more to the segmentation result. To solve this issue, we propose an efficient and effective architecture, termed small objects segmentation network (SOSNet), to improve the segmentation performance of small objects. The SOSNet works from two perspectives: methodology and data. Specifically, with the former, we propose a dual-branch hierarchical decoder (DBHD) which is viewed as a small-object sensitive segmentation head. The DBHD consists of a top segmentation head that predicts whether the pixels belong to a small object class and a bottom one that estimates the pixel class. In this situation, the latent correlation among small objects can be fully explored. With the latter, we propose a small object example mining (SOEM) algorithm for balancing examples between small objects and large objects automatically. The core idea of the proposed SOEM is that most of the hard examples on small-object classes are reserved for training while most of the easy examples on large-object classes are banned. Experiments on three commonly used datasets show that the proposed SOSNet architecture greatly improves the accuracy compared to the existing real-time semantic segmentation methods while keeping efficiency. The code will be available at https://github.com/StuLiu/SOSNet.

I. Introduction

Semantic segmentation has been a fundamental problem in computer vision since the early days, which plays a central role in a broad range of applications including autonomous vehicles [1], [2], [3], [4], medical image analysis [5], [6], [7], [8], remote sensing [9], [10], [11], and augmented reality [12]. The results of semantic segmentation have very high potential value for subsequent applications. It aims to assign a categorical label to each pixel of the input image, thus, the foreground objects and background areas are segmented. In the past decades, extensive efforts have been made to develop semantic segmentation methods. Among these methods, deep-learning-based segmentation methods have achieved excellent performance in speed and accuracy. However, the small objects containing only a few pixels are still hard to segment accurately, such as cars in the remote sensing scenarios and traffic signs in automatic driving scenarios.