Object Detection in Real-Time Systems: Going Beyond Precision

Applications like autonomous driving, industrial robotics, surveillance, and wearable assistive technology rely on object detectors as an integral part of the system. Thus, an increase in performance of object detectors directly affects the quality of such systems. In the recent years, convolutional neural networks (CNNs) and its variants emerged as the state of art in object detection, where performance is usually measured either in terms of mean average precision (mAP) or number of frames processed per second (fps). Many applications which use object detectors are resource constrained in practice. Even though it is clear from the published results, that a frame-level analysis of the system in terms of mAP or fps proves the superiority of one algorithm over the other, we observe that such metrics do not necessarily apply to real time applications with resource constraints. A slower algorithm even though highly accurate may need to drop frames to maintain the necessary frame rate and lose on the accuracy. We propose a closer look at the metrics used for performance in real-time applications, and suggest some new evaluation criterion. Our comparison of state of the art detectors on these metrics has also thrown some surprises in terms of conventional wisdom, which we present in this paper. Our framework is available at https://www.github.com/anupamsobti/object-detectionreal-time-systems.

1. Introduction

Object Detection is the task of localizing and classifying objects in a given image. Applications such as assisted and autonomous driving [9], navigation aids for visually impaired [21], and robotics applications [13] use object detector module as an integral step for motion planning, landmark detection etc. The efficacy of these detectors on a set of images is typically measured using conventional metrics like precision and recall. For evaluating performance on video input, True positives (TP), False Positives (FP), True Negatives (TN) and False Negatives (FN) determined using the Jaccard index criteria are simply accumulated over all frames of the video. Jaccard index, more commonly known as IoU is the intersection over union of the bounding box predicted using the object detector under consideration with that given in the ground truth. A detection is considered as positive if the Jaccard Index is greater than a threshold. Precision-Recall curves are then plotted for different values of the threshold of the detector and Mean Average Precision (mAP) is often used as a measure of how well the detector performs. Figure 1:

There are a number of factors which affect the performance of detectors under different constraints. We discuss some of these constraints and situations. We also suggest an evaluation criterion which can capture this information and take the evaluation of object detectors closer to the application performance.