I. Introduction
Recent advances in computer vision and deep learning have enabled autonomous systems to employ vision-based controllers to perceive their environment and react to it for accomplishing various tasks, including agile navigation [1], manipulation [2], autonomous driving [3], and autonomous aircraft landing and taxiing [4]. Despite their success, such vision-based controllers can fall prey to issues when they are subjected to inputs that are scarcely encountered in the training dataset or are outside the training distribution. For example, a visual policy trained exclusively with well-illuminated images might fail to predict good actions in dark conditions; an autonomous car that is predominately shown to take right turns in an expert dataset, may fail to learn to make left turns. Such vision failures can cascade to catastrophic system failures and compromise system safety. Thus, to successfully adopt vision-based neural network controllers in safety-critical applications, it is vital to analyze them and understand when and why they result in a system failure. In addition to reasoning about system safety, these failure modes might be useful in engineering corrective measures for the system.