Comparison of ACR Methods for 360° Video Quality Assessment Subject to Participants’ Experience with Immersive Media

Watching 360° videos on head-mounted displays (HMDs) has become more popular in recent years. As such, subjective quality assessment of 360° videos on HMDs is needed to obtain a ground truth on the quality as perceived by users. Due to the lack of standardized methods, the absolute category rating (ACR) with hidden reference (ACR-HR) method for conventional videos has also been used for subjective quality assessment of 360° videos on HMDs. A modified ACR (M-ACR) method tailored for assessing 360° videos on HMDs has recently been proposed. In this paper, we compare the ACR-HR and M-ACR with hidden reference (M-ACR-HR) method regarding subjective quality assessment of 360° videos on HMDs. The comparison is conducted subject to participants’ experience with watching immersive media on HMDs. The findings include: 1) Average rating times to cast opinion scores is lower for the ACR-HR method irrespective of participants’ experience, 2) Mean opinion scores are similar for both methods irrespective of participants’ experience, 3) The M-ACR-HR method is more reliable, especially with experts and participants with no experience, 4) Discomfort is significantly lower for the ACR-HR method, especially, for participants with no experience.