Virtual Reality System using Explainable AI for Identification of Specific Expert Refinery Inspection Skills

In this study, we propose a virtual reality system for identifying expert-specific skills in a visual inspection task in a refinery by using an eXplainable Artificial Intelligence (XAI) technique. Most previous studies have applied statistical analysis such as t-tests to the mean value of the experimental data, and there is a consequent lack of specificity in the results (i.e., when and where expert skill appears within a long inspection duration). It is thus difficult to provide feedback based on the most important part of the collected experts’ data to the novices. To address this issue, we introduce a Convolutional Neural Network (CNN) with Class Activation Map (CAM) technique, an XAI method, to analyze the experimental data of experienced and novice field operators, and identify the most significant contributors for classifying expert and novice behavior for 120 seconds inspections. The resulting model can classify field operators as expert or novice with an accuracy of 99.1% on average, and visualize the classification criteria as a heat map for each experimental trial. Based on those results, we propose a virtual reality training system for learning expert inspection skills by referencing the CNN results. The contribution of our study is the proposition of a new analytical framework, as well as a training system beyond the limitations of conventional statistical analysis.