Visualizations of Fusion of Electro Optical (EO) and Passive Radio-Frequency (PRF) Data

In machine learning, the ability to reliably determine potential pitfalls in the decision making process of an algorithm is essential. In previous research, the fusion of passive radio frequency (P-RF) histograms generated from in-phase quadrature component (I/Q) data and enhanced electro-optical (EO) data are fused together in order to implement classification and tracking of different vehicle targets using the AFRL's ESCAPE dataset. In previous research, the impact of the P-RF data was confirmed to essential for achieving a higher performance when fused with EO data. This research, however, did not provide an intuitive means by which inferences and explanations for a human expert could interpret. In this paper, saliency maps are implemented in order to visualize the impact of P-RF data in the fusion model and thereby confirm the role the different modalities play in the heterogeneous sensor fusion in an intuitive manner. These visualizations provide context for which pixels activate neurons in the final layer of the model. Overlaying the two modality inputs with respect to time, the method presented in this paper is able to provide explainability for the fusion model while also achieving an F1 score of 0.9. The research in this paper uses the distribution and frequency of the appearance of different types of visualizations. Combined this with context from the scenarios with respect to the timeline of events, it becomes possible to draw inferences for how the two modalities are utilized by the fusion model.