Pilot Workload Prediction from ECG Using Deep Convolutional Neural Networks

Pilot workload monitoring plays an important role in aviation safety. Heart rate, heart rate variability, and respiration rate have been shown to correlate with pilot workload and can be measured using electrocardiograms (ECG). Traditional machine learning approaches rely on manually extracting features from ECG, which is a difficult and time-consuming process. Recent years witnessed the success of deep neural networks, especially deep convolutional neural networks (CNNs), in computer vision and related domains; however, the application of deep CNNs onto the ECG data faces challenges on both data insufficiency and lack of tailored CNN architectures. With a small training set, this work proposes the use of transfer learning with pre-trained deep CNNs for the prediction of pilot workload. Two ECG-derived visual representations, spectrograms and scalo-grams, are compared for their performance on the prediction. Experimental results indicate that the scalograms perform better (at 51:35%) than spectrograms (at 45:85%) in predicting three levels of pilot workload. With the scalograms, using the pre-trained deep CNNs as "off-the-shelf" feature extractors yields better performance than fine-tuning the deep CNNs (at 42:44%) for the ECG data. The deep features are visualized using dimension reduction with t-sne.