An in-depth analysis of robustness and accuracy of recommendation systems

Recommender systems are extensively deployed in e-commerce, social networks, app markets, etc., as it facilitates the human decision-making process due to its capability in delivering tailored content for each individual. Among the evaluations, there are two key performance indicators, namely, robustness, and accuracy. The paper proposes a generalized evaluation methodology for accuracy and robustness evaluation of recommendation models and conducts an in-depth analysis of the inter-relation between these two indicators. This paper reveals that when the recommender is more efficient in using data with higher accuracy, it may lead to the instability of the model when the data is disturbed. In this paper, we examine the interplay between robustness and accuracy for both rating prediction and click-through-rate (ctr) prediction–the two mainstream recommendation tasks. In particular, for rating prediction task, we consider techniques based on k-nearest neighbor (KNN) that utilizes 2-nd order (i.e. user-item) pairs, to make recommendations. For ctr prediction task, we then consider a recommendation algorithm that emphasizes both low- and high-order feature interactions (DeepFM) by combining the power of factorization machines for recommendation and deep learning for feature learning in a new neural network architecture. The experiments have been performed on two widely used datasets, i.e., MovieLens, and Criteo, with a detailed sensitivity analysis w.r.t model parameters. Our results show that the better the model is at capturing the user preferences in the raw data, the more easily the model is affected by the noises/perturbations.