BgSubNet: Robust Semisupervised Background Subtraction in Realistic Scenes

Background subtraction (BgS) is a problem for handling pixel-level identification of changing or moving entities in the field of view of a static camera system. Recent works have discovered superior generalization to unseen realistic scenarios by an approach called deep BgS, which employs deep neural networks (DNNs) on concatenations of image inputs and their backgrounds. However, due to a lack of large-scale foreground-background image datasets, the challenges of manually creating background masks for existing Internet-scale image datasets, and a general lack of attention to circumvent them, studies to exploit the generalization potentials of deep BgS have yet to be thoroughly conducted for unseen realistic scenes. Hence, our work proposes an analysis regarding the generalization capabilities of deep BgS, particularly by their binary constraints of pixel feature representations. Then, to address the lack of large-scale labeled data to induce generalization, we propose a strategy to generate a virtually limitless freeform dataset, follow up with the first self-supervised (SS) pretraining strategy for deep BgS, then extend it to a semisupervised framework called BgSubNet, which can support inference on unseen scenarios by self-supervision and mixed-domain training with real and synthetic data. Our results are experimentally validated at competitive results with CAMO-UOW and CDnet2014 datasets to show considerably high performance against existing frameworks.

I. Introduction

Background subtraction (BgS), or change detection, is a fundamental branch of semantic segmentation that handles the identification of changing or moving entities in the field of view of a visual camera sensor. In particular, models in BgS seek to classify regions of semantically meaningful changes (e.g., walking people and moving vehicles) as foreground (FG), and classify static regions (e.g., buildings and roads) or nonsemantic motions (e.g., wavy water and rain droplets) as background (BG). In practice, BgS methods have been utilized as an important step in intelligent surveillance systems such as traffic analyses, motion segmentation, or behavior analysis [1], [2]. Hence, accuracy and generalization are stressed as essential properties for BgS adoption across various visual challenges, such as dynamic backgrounds, camera motion, illumination changes, dual-camera sensors, camouflaged objects, and especially on unseen scenarios (or scene-agnostic evaluation). Because the processed results of BgS are typically pipelined toward higher-level analytic components (e.g., tracking and activity recognition), any erroneous predictions would significantly limit practical applicability.