Towards Relevance and Diversity in Crowdsourcing Worker Recruitment With Insufficient Information

Nowadays, crowdsourcing has become an increasingly popular paradigm in real-world applications. It provides a fundamental mechanism for inviting thoughts and perspectives from a wide swath of people. One of the critical issues in crowdsourcing lies in worker recruitment. However, the existing literature mainly focuses on maximizing the relevance between workers and tasks, ignoring the diversity among workers. Such neglect may lead to poor quality of task completion. In this article, we present an online worker recruitment mechanism with relevance and diversity. To comprehensively evaluate the capability of a worker group, we propose an enhanced metric called utility. We also consider a practically motivated setting in which the observed side-information of workers is insufficient due to privacy concerns. Then, we utilize contextual combinatorial multi-armed bandit to model the online worker selection process, and propose a novel Diversified Hidden Upper Confidence Bound (DH-UCB) algorithm to address the problem. Most importantly, we rigorously prove that our DH-UCB algorithm achieves significant sublinear upper regret bound with high probability. In other words, it can effectively recruit workers under insufficient information while maintaining sustainable relevance and diversity. Extensive experiments on both synthetic and large-scale real-world datasets empirically validate the advantages of our proposed mechanism.

I. Introduction

As a common practice, crowdsourcing has been extensively applied in a vast array of real-world applications, including Internet of Things [1], drug discovery [2], and cloud computing [3]. Typically, via a central platform (i.e., an organization or company), the crowdsourcing system collects information, opinions and efforts from a mass of people. A representative crowdsourcing application is InnoCentive¹

https://www.innocentive.com/

, which recruits workers to solve challenging scientific problems in areas such as chemistry, mathematics and physical sciences. By soliciting the wisdom of the crowd, over 2,500 challenges have been solved on the website, while more than 200 thousand solutions have been proposed²

https://www.innocentive.com/about-us/about/

.