ZREC: Robust Recovery of Mean and Percentile Opinion Scores

Observer screening and subject opinion score recovery is essential for collecting a reliable QoE database. This paper proposes a new method, ZREC*, which uses Z-scores to estimate subject bias, inconsistency, and content ambiguity. Additionally, we propose Mean Opinion Score (MOS) recovery and Percentile Opinion Score (POS) recovery scheme based on the three estimated parameters. ZREC does not fully reject subjects, rather adjust their coefficients in the MOS/POS recovery, allowing for more efficient use of data collection. The estimated parameters of ZREC are highly correlated with more complex solver-based methods and standards. In addition, ZREC recovers MOS with smaller confidence intervals than the state of the art. Experimental results also demonstrate that using recovered p
th
POS as ground truth during training improves the performance of Satisfied User Ratio (SUR) prediction.