How to complete performance graphs in content-based image retrieval: add generality and normalize scope

The performance of a content-based image retrieval (CBIR) system, presented in the form of precision-recall or precision-scope graphs, offers an incomplete overview of the system under study: the influence of the irrelevant items (embedding) is obscured. We propose a comprehensive and well-normalized description of the ranking performance compared to the performance of an ideal retrieval system defined by ground-truth for a large number of predefined queries. We advocate normalization with respect to relevant class size and restriction to specific normalized scope values (the number of retrieved items). We also propose new three and two-dimensional performance graphs for total recall studies in a range of embeddings.

1 The Need for Reevaluation of CBIR Performance Characterization

The motivation for this work came up during the experiments carried out on our Leiden 19th-Century Portrait Database (LCPD). This database presently contains 21, 094 photographic portraits with as many studio logos (on the separately scanned backsides). These logos are manually grouped into 1, 856 nonoverlapping logo classes with an average size of almost eight relevant unordered items; relevant class size is in the range [2, 308]. A more extensive description of this ground-truth test set can be found in [8].