Machine Learning Multi-Modality Fusion Approaches Outperform Single-Modality & Traditional Approaches

This paper presents results from two machine learning components within the upstream multi-modal fusion and decision-making framework introduced previously (Garagić et al., 2018 IEEE Aerospace Conference, doi: 10.1109/AERO.2018.8396737, Garagić et al., 2019 IEEE Aerospace Conference, doi: 10.1109/AERO.2019.8742221). The first combines Bayesian change-point motion-based detection to discover new targets with online transfer learning over Convolutional Neural Networks (CNNs) to learn models of previously seen targets and adapt those models as target appearance changes. CNN detection overcomes motion-based detection problems that arise when targets stop or operate in close proximity to each other. Motion-based detection overcomes the CNN problem of initializing target appearance and adapting to target appearance changes. This novel combination of motion and CNN detectors obviates the need for prior knowledge about targets to stimulate target detection. This detection approach improves performance over that achieved without an appearance-based detector on challenging data. A closed-loop between the tracking component and the detector component modules drives detector model adaptation. The second component combines multi-sensor, multi-modality features from upstream detection and tracking components to learn dynamic compact probabilistic representations of targets as manifolds in a joint space over all inputs. This Multi-Modal Deep Belief Network (MMDBN) approach incorporates supervised deep autoencoders and unsupervised Dirichlet process mixture models. Supervision within MMDBN arises from consistent associations between sensor and modality inputs for a target of interest. MMDBN thus learns and optimally leverages inter-sensor, inter-modality features of targets. This approach obviates the need for explicit, hand-crafted intermodal features because it implicitly learns both inter-modal features and a likelihood model to evaluate those features. This fusion approach improves tracking performance over a traditional fusion approach on challenging data. This overall approach offers multiple benefits in addition to outperforming single-modality and traditional fusion approaches. It is feature agnostic by accommodating any data source that provides a feature set of pre-determined dimensionality for each target at each time step. This paper describes novel methods for enhancing motion-based tracking by adding an appearance-based sidecar that provides to improve DTC performance in difficult conditions. It also showed the benefits of a feature-level fusion approach in performing DTC on several challenging multi-modality, multi-platforms AFRL ESCAPE data scenarios.