Real-time quality control for crowdsourcing relevance evaluation

In recent years, crowdsourcing has become an effective method in many fields, such as relevance evaluation of search engine results. However, the problems of bad workers and quality control always stand among the major challenges within the realm. Based on our experiments carried out in TREC 2011 Crowdsourcing track, this paper demonstrates a realtime strategy in recruiting workers and monitoring the quality of their relevance and rank judgments. The effectiveness of our strategy has been verified by empirical results.