Design and Validation of a Pilot Stress Monitoring System based on a Deep Convolutional Neural Network Model

This article presents the Design and Validation of a Stress Model based on a Deep Convolutional Neural Network (DCNN) included in the Pilot Monitoring System (PMS) developed by Airbus Defence Space.Starting by introducing the whole system, PMS obtains pilot’s vital data in real time with two wearable devices (smartwatch and ear sensor). It is able to determine stress, fatigue, workload, health status and pilot capacity with new models based on DCNN encoded in Python which are able to interpret these vital data.Additionally, PMS is able to launch alerts when the vital parameters of the crew reach pre-established limits, giving the chance to detect an abnormal situation (e.g. excessive workload scenario with high stress level). In this way, it will improve situational awareness by making crew aware of their own conditionGoing deeper into the details of the Stress Model, its main objective is to determine Pilot stress in real time.For achieving this target, several design stages were taken into consideration while building the Stress Model DCNN: raw data acquisition, data cleaning, feature engineering, algorithm selection, model training, model evaluation, model testing, and result analysis. Further details of these stages are explained below.Regarding data acquisition, Stress Model obtains pilot’s vital data from a smartwatch (one of the wearable devices of PMS), which communicates via Bluetooth Low Energy (BLE) to a laptop. Selected device was Analog Devices Inc. Gen II watch, which sends the following signals: Heart Rate (HR), Electro-Dermal Activity (EDA), Skin temperature and 3D Acceleration.At Cleaning stage, Inter-Quartile Range (IQR) was used to normalize our raw data, being less sensitive to outliers and therefore more robust.At Feature engineering stage, several signal combinations were explored. The selected signal set which gave the best Stress Model accuracy was comprised of two EDA parameters (Real Admittance [Siemens], Impedance Magnitude [Ohm]) and wrist Skin Temperature [deg C].At Algorithm selection stage, three convolutional layers (128, 256 and 128 filters respectively) were chosen to extract a feature map from the inputs. After them, a Global Average Pooling (GAP) was chosen to compress the convolutions output. Downstream GAP, a decision layer based on softmax activation function (stress vs non-stress) was chosen.At Training stage, it was necessary to implement a Stress Test recording campaign. Participants were asked to fulfill several individual demanding tasks (memory, multi-task, stroop, vigilance) which included relaxing periods among them. The Stress Model was fed with the vital data obtained in this campaign.In between the tasks, NASA-TLX questionnaires were filled by test subjects, to obtain their subjective rating of Mental demand, Physical demand, Temporal demand, Performance, Effort and Frustration. Therefore, the Stress Model considered both vital data and self-report questionnaires as inputs.At result analysis stage, the Stress Model accuracy was refined, understanding accuracy as the ratio between correct predicted stress scenarios versus total cases, and considering ground truth self-report questionnaires.Finally, Stress Model DCNN was validated considering three data sources, whose details are included below:A.Vital data from fifteen (N=15) subjects of Wearable Stress and Affect Detection (WESAD) Database available online.Stress Model gave an average test accuracy of 99%,B.Vital data from four (N=4) Airbus Defence and Space (ADS) pilots while calibrating their NASA-TLX scale.Stress Model in this validation gave an average test accuracy of 90%.C.Vital data from six (N=6) ADS pilots while performing simulator evaluations using an Enhanced Light Weight Visor (ELWEV).Stress Model in this validation gave an average test accuracy of 87%.