Specrank: Ranking by Pairwise Comparisons from Specialty Workers in a Crowdsourced Setting

This work proposes SpecRank, an algorithm for the ranking of objects from pairwise comparisons provided by workers with different specialties in a crowdsourced setting. We assume that the objects may belong to different latent classes that are unknown to the requester, and the workers' ability to provide reliable labels may depend on their specialty or familiarity with these classes. We first propose a specialty Bradley-Terry-Luce (specialty-BTL) model that incorporates workers' specialties in the quality of their pairwise comparison labels. Moreover, we adopt a latent feature model where the scores of objects are modeled as inner products between the latent feature vectors of the objects and their associated classes. The workers' ability to provide accurate pairwise labels is also determined by the correlation between their specialty feature vectors and the class feature vectors. By adopting a mixture model on the objects' class labels, our proposed SpecRank adopts variational expectation maximization (EM) algorithm to perform joint ranking and clustering of the objects. The effectiveness of the proposed specialty-BTL model and the SpecRank algorithm is demonstrated on both synthetic and real-world datasets.