Hierarchical Multivariate Representation Learning for Face Sketch Recognition

Face Sketch Recognition (FSR) is extremely challenging because of the heterogeneous gap between sketches and images. Relying on the ability to generative models, prior generation-based works have dominated FSR for a long time by decomposing FSR into two steps, namely, heterogeneous data synthesis and homogeneous data matching. However, decomposing FSR into two steps introduces noise and uncertainty, and the first step, heterogeneous data synthesis, is an even general and challenging problem. Solving a specific problem requires solving a more general one is to put the cart before the horse. In order to solve FSR smoothly and circumvent the above problems of generation-based methods, we propose a multi-view representation learning (MRL) framework based on Multivariate Loss and Hierarchical Loss (MvHi). Specifically, by using triplet loss as a bridge to connect the augmented representations generated by InfoNCE, we propose Multivariate Loss (Mv) to construct a more robust common feature subspace between sketches and images and directly solve FSR in this subspace. Moreover, Hierarchical Loss (Hi) is proposed to improve the training stability by utilizing the hidden states of the feature extractor. Comprehensive experiments on two commonly used datasets, CUFS and CUFSF, show that the proposed approach outperforms state-of-the-art methods by more than 7%. In addition, visualization experiments show that the proposed approach can extract the common representations among multi-view data compared to the baseline methods.

I. Introduction

A landscape can be painted or photographed. Different recording methods and different sensors bring rich multi-view data. In recent times, multi-view image retrieval has gained significant importance owing to its immense commercial value. Matching multi-view data, however, poses a formidable challenge due to the inherent heterogeneous gap between multi-view data. A typical example of the multi-view data matching problem is Face Sketch Recognition (FSR), which plays a crucial role in law enforcement [1]. When a crime occurs, law enforcement officials often encounter situations where poor resolution from crime scene surveillance videos or even surveillance equipment is missing or vandalized by the suspect. These factors make it difficult to obtain a clear facial image of a suspect. In this case, the common approach to solving the crime involves creating a sketch of the suspect based on the witnesses' descriptions. Then, the sketch is compared with the police mugshot database, one entry at a time. A successfully matched mugshot is pivotal in criminal investigations. However, the heterogeneous gap between images and sketches makes solving FSR smoothly a challenge, and conventional face recognition methods lack the ability to bridge this gap [2]. Over the past few decades, many scholars have devoted their efforts to FSR. Many methods have been proposed, mostly falling into two categories: photo-sketch generation (PSG) [3], [4], [5], [6], [7] and multi-view representation learning (MRL) [8], [9], [10], [11].