Analysis of Public Image and Video Databases for Quality Assessment

Databases of images or videos annotated with subjective ratings constitute essential ground truth for training, testing, and benchmarking algorithms for objective quality assessment. More than two dozen such databases are now available in the public domain; they are presented and analyzed in this paper. We propose several criteria for quantitative comparisons of source content, test conditions, and subjective ratings, which are used as the basis for the ensuing analyses and discussion. This information will allow researchers to make more well-informed decisions about databases, and may also guide the creation of additional test material and the design of future experiments.

I. Introduction

Ground truth is one of the most important and useful components for the evaluation and benchmarking of new algorithms. In the field of image and video quality, ground truth means databases of test clips annotated with subjective ratings. The lack of annotated databases used to be a major hurdle for researchers working on quality assessment algorithms. Even uncompressed video content was hard to find. In recent years, an increasing a number of databases have been released into the public domain, to the point where it has become be hard to keep track of them or to choose the most suitable one. For this paper, we have compiled a list of 27 image and video databases that are publicly available and relevant to quality assessment.