A Semantically Informed Benchmark Dataset for Computer Vision in Aviation Systems

Deep learning and convolutional neural networks have greatly improved the accuracy of computer vision technologies. As a result, computer vision for aviation systems is a rapidly expanding field, with systems being developed to detect aircraft, improve aircraft maintenance, and even assist in navigation tasks as Unmanned Aerial Vehicles (UAVs) move towards autonomous flight. Despite the rapid expansion of technology in terms of accuracy and novel applications, there is a lack of trust and adequate testing in computer vision research, making it difficult to implement these models within the highly regulated aviation industry. Verification techniques, specifically testing metrics traditionally applicable to software systems are no longer applicable to the non-deterministic and black-box nature of deep learning models. In alignment with current developments in testing techniques for deep learning, this work proposes a system for creating and managing a novel benchmark dataset of images to test all computer vision models. The images in this dataset are embedded with semantic knowledge from a general knowledge graph and a Bidirectional Encoder Representations from Transformers model to represent the semantic relationships between objects within an image. As a result, the system can validate the performance of computer vision models, even when the objects being detected are unlabeled in the original testing set. This novel benchmark dataset demonstrates an accurate testing system that works with UAV-based detection systems currently in development. This work can be incredibly beneficial to developing the concept of trust surrounding computer vision, allowing the expanded use of computer vision in aviation operations.