Information-Theoretic Segmentation by Inpainting Error Maximization

We study image segmentation from an information-theoretic perspective, proposing a novel adversarial method that performs unsupervised segmentation by partitioning images into maximally independent sets. More specifically, we group image pixels into foreground and background, with the goal of minimizing predictability of one set from the other. An easily computed loss drives a greedy search process to maximize inpainting error over these partitions. Our method does not involve training deep networks, is computationally cheap, class-agnostic, and even applicable in isolation to a single unlabeled image. Experiments demonstrate that it achieves a new state-of-the-art in unsupervised segmentation quality, while being substantially faster and more general than competing approaches.¹

1. Introduction

Deep neural networks have significantly advanced a wide range of computer vision capabilities, including image classification [38], [55], [56], [27], object detection [22], [50], [40], and semantic segmentation [8], [73]. Nonetheless, neural networks typically require massive amounts of manually labeled training data to achieve state-of-the-art performance. Applicability to problems in which labeled data is scarce or expensive to obtain often depends upon the ability to transfer learned representations from related domains.