Standardization of Agnostic Learning Techniques in Neuroimaging: a Case Study in EEG

Over the last few years, the use of multivariate pattern analysis (MVPA) has been growing exponentially in cognitive neuroscience. This methodology offers the possibility to perform more detailed studies in electroencephalography (EEG) and other neuroimaging modalities. Usually, machine learning-based algorithms along with empirical out-of sample generalization approaches, e.g. K-fold cross validation (CV), are implemented. In this work we applied a validation approach based on resubstitution with upper bound correction to address the instability problem concerning limited sample sizes, particularly typical in neuroimaging, when estimating accuracy via CV. Moreover, a statistical assessment of the accuracies is conducted. To this end, the methodology underlying Statistical Agnostic Mapping (SAM) to perform spatial detection of regions of interest (ROIs) in neuromaging modalities, such as magnetic resonance imaging (MRI), has been applied to a temporal EEG study. Instead of estimating ROIs by means of a significance maps, the time points where trials are most significant are highlighted. The analyses were implemented on an EEG dataset obtained from 48 participants that performed a simple go-nogo task, in which they were presented names and faces. Here, we classified name and face perception at the subject level. The findings suggest that the use of resubstitution-based approaches can reduce the computational burden and obtain similar, albeit more conservative, results than CV.