AxIoU: An Axiomatically Justified Measure for Video Moment Retrieval

Evaluation measures have a crucial impact on the direction of research. Therefore, it is of utmost importance to develop appropriate and reliable evaluation measures for new applications where conventional measures are not well suited. Video Moment Retrieval (VMR) is one such application, and the current practice is to use R@K, $\theta$ for evaluating VMR systems. However, this measure has two disadvantages. First, it is rank-insensitive: It ignores the rank positions of successfully localised moments in the top-K ranked list by treating the list as a set. Second, it binarizes the Intersection over Union (IoU) of each retrieved video moment using the threshold $\theta$ and thereby ignoring fine-grained localisation quality of ranked moments. We propose an alternative measure for evaluating VMR, called Average Max IoU (AxIoU), which is free from the above two problems. We show that AxIoU satisfies two important axioms for VMR evaluation, namely, Invariance against Redundant Moments and Monotonicity with respect to the Best Moment, and also that R@ K, $\theta$ satisfies the first axiom only. We also empirically examine how Ax-IoU agrees with R@K, $\theta$, as well as its stability with respect to change in the test data and human-annotated temporal boundaries.

1. Introduction

Video Moment Retrieval (VMR) has been explored to find relevant fragments of videos (i.e. video moments) based on a user's textual query [10], [15]. Most existing VMR systems [10], [22], [38], [39], [42] cast the problem of finding video moments into a ranking problem. For evaluating ranked lists of video moments, R@K, is widely adopted in the literature [10]. R@K, for a query is defined as 1 if at least one relevant video moment in the top of the ranked list has an Intersection over Union (IoU) larger than with the ground truth for .