Joint Detection and Online Multi-object Tracking

Most multiple object tracking methods rely on object detection methods in order to initialize new tracks and to update existing tracks. Although strongly interconnected, tracking and detection are usually addressed as separate building blocks. However both parts can benefit from each other, e.g. the affinity model from the tracking method can reuse appearance features already calculated by the detector, and the detector can use object information from past in order to avoid missed detection. Towards this end, we propose a multiple object tracking method that jointly performs detection and tracking in a single neural network architecture. By training both parts together, we can use optimized parameters instead of heuristic decisions over the track lifetime. We adapt the Single Shot MultiBox Detector (SSD)[14] to serve single frame detection to a recurrent neural network (RNN), which combines detections into tracks. We show initial prove of concept on the DETRAC[26] benchmark with competitive results, illustrating the feasibility of learnable track management. We conclude with a discussion of open problems on the MOT16[15] benchmark.

1. Introduction

The main challenge in tracking multiple objects in a video sequence stems from its open world nature. Specifically, the unknown number of present objects as well as uncertainties and ambiguities in assigning tracks to detection (data association) make it difficult to decide if a detection is a false positive or a new track, or when a track should be established or terminated. Even more challenging is tracking objects in an online setting, where a globally optimal solution cannot be achieved. Locality in time means that once an error is made it can hardly be corrected in the future, e.g., confusing a false positive detection on a background structure with a new track could result in a false track, which will constantly be confirmed by the background structure.