Reinforcing Self-expressive Representation with Constraint Propagation for Face Clustering in Movies

The ability to robustly cluster faces in movies is a necessary step in understanding media content representations of people along dimensions such as gender and age. Building upon the successes of sparse subspace clustering (SSC) in uncovering the underlying structure of the data, in this paper we propose an algorithm called Constraint Propagation Sparse Subspace Clustering (CP-SSC) for applications such as face clustering in videos where pairwise sample constraints (must-link and cannot-link sample pairs) are available in the processing pipeline since detected faces can be tracked locally in time. We learn the subspace structure while simultaneously incorporating the pairwise constraints to construct a similarity matrix needed for clustering. Our joint formulation uses low-rank matrix completion to propagate the initial pairwise constraints, that are used to reinforce the subspace representation during optimization. We evaluate CP-SSC for clustering faces in movies with pre-trained neural network embeddings as features. We first analyze CP-SSC with synthetic data and then show that it can be effectively used to cluster faces in movie videos. We evaluate our method for two movies annotated in-house and two benchmark movies released publicly. We also compare the performance of our algorithm with other clustering approaches that use pairwise constraint information.