Mining Latent Attributes From Click-Through Logs for Image Recognition

Attribute-based image representation, which represents an image by projecting it into a space spanned by attributes, has attracted increasing attention from both computer vision and multimedia communities for its compactness and potential to bridge the semantic gap. While many works focus on learning attribute models and utilizing them in image recognition and retrieval, few touch on the problem of how to effectively construct a vocabulary of attributes, which is an essential part of effective attribute-based representation. Most existing approaches define the attribute vocabulary by human experts or through existing ontology, which is often limited in coverage of general concept space. In this paper, we propose automatically constructing the attribute vocabulary by mining latent topics from the click-through log of a commercial image search engine. These attributes are referred to as latent topic attributes (LTA), which take advantage of tens of millions of interactions between user submitted queries and images, thereby providing better coverage for the concept space than existing approaches. The mining of latent topics from the click log is formulated as a matrix factorization problem, and further improved by weighted terms-based matrix factorization to address the extreme sparsity of the click-through matrix. Both qualitative results of the mined LTA and quantitative results on the standard image recognition benchmark demonstrate the mined LTA's effectiveness.