Crowdsourcing with Multi-Dimensional Trust and Active Learning

We consider a typical crowdsourcing task that aggregates input from multiple workers as a problem in information fusion. To cope with the issue of noisy and sometimes malicious input from users, trust is used to model workers expertise. We propose a probabilistic model to jointly infer multi-dimensional trust of workers, multi-domain properties of questions,and true labels of questions.Our model is flexible and extensible to incorporate metadata associated with questions. To show that, we further propose two extended models, one of which handles input tasks with real-valued features and the other handles tasks with text features by incorporating topic models. In order to decrease entropies and reduce error rates more quickly with fewer annotations from workers, we further propose strategies for selecting which questions to ask and which workers to assign the questions to based on multidimension characteristics of questions and workers trust values in those dimensions. We evaluate our models and algorithms on real-world data sets. These results can be applied for fusion of information from multiple data sources like sensors, human input, machine learning results, or a hybrid of them.