Is Heuristic Sampling Necessary in Training Deep Object Detectors?

To train accurate deep object detectors under the extreme foreground-background imbalance, heuristic sampling methods are always necessary, which either re-sample a subset of all training samples (hard sampling methods, e.g. biased sampling, OHEM), or use all training samples but re-weight them discriminatively (soft sampling methods, e.g. Focal Loss, GHM). In this paper, we challenge the necessity of such hard/soft sampling methods for training accurate deep object detectors. While previous studies have shown that training detectors without heuristic sampling methods would significantly degrade accuracy, we reveal that this degradation comes from an unreasonable classification gradient magnitude caused by the imbalance, rather than a lack of re-sampling/re-weighting. Motivated by our discovery, we propose a simple yet effective Sampling-Free mechanism to achieve a reasonable classification gradient magnitude by initialization and loss scaling. Unlike heuristic sampling methods with multiple hyperparameters, our Sampling-Free mechanism is fully data diagnostic, without laborious hyperparameters searching. We verify the effectiveness of our method in training anchor-based and anchor-free object detectors, where our method always achieves higher detection accuracy than heuristic sampling methods on COCO and PASCAL VOC datasets. Our Sampling-Free mechanism provides a new perspective to address the foreground-background imbalance. Our code is released at https://github.com/ChenJoya/sampling-free.

I. Introduction

With the development of deep learning [1], [2], recent years have witnessed remarkable advancement in object detection [3]. Among them, representative successes include two-stage R-CNN detectors [4]–[15]: their first stage uses a region proposal network (RPN [4]) to generate some candidates from dense, predefined bounding-boxes (i.e. anchors), then the second stage uses a region-of-interest subnetwork (RoI-subnet) for object classification and localization. To pursue higher efficiency, one-stage approaches [16]–[23] directly recognize objects from dense anchors rather than generating candidate proposals. Both two-stage and one-stage detectors adopt the anchoring scheme, where massive anchors (~10⁵) are uniformly sampled over an image.