From Leaderboard To Operations: DIVA Transition Experiences

The IARPA Deep Intermodal Video Analytics (DIVA) program has sponsored the development of systems that detect and recognize activities in security video. During the period from September 2017 to March 2021, the development and evaluation of these systems was focused on optimizing accuracy, embodied in quantified metrics, against a large but relatively static corpus of video collected and annotated by the program. This focus was aided by various software engineering decisions collaboratively reached by the program performers and Test & Evaluation (T&E) team, which established a common software framework enabling ongoing quantitative evaluation via software submissions to a leaderboard. While continuing to support the leaderboard, in March 2021 the program began efforts, still in progress, to transition capabilities developed on DIVA from the research environment to operational evaluation and deployment. As an operational system is a different use case than a research environment, it is not surprising that design decisions favoring the former will not always align with the latter. This paper discusses our work to transition DIVA systems into an operational setting, particularly identifying and resolving conflicts between the evaluation framework and operational requirements. We describe transition efforts to date, propose future work, and conclude with lessons learned from the overall transition effort.

1. Introduction

It has been estimated that in 2019, 180 million security cameras were shipped worldwide [9], while the attention span of a human camera operator has been estimated at only 20 minutes of continuous manual monitoring [4, 3]. The gap between the massive volume of data available and the scarce capacity of human analysts has been closed, but not eliminated, by the rapid advancement of computer vision techniques, particularly deep-learning based methods.