Journal Article

Assessing and tuning brain decoders: Cross-validation, caveats, and guidelines

Varoquaux, Gaël, Raamana, Pradeep Reddy, Hoyos-Idrobo, Andrés, Thirion, Bertrand, Schwartz, Yannick, Engemann, Denis A.

2017

Overview

Decoding, i.e. prediction from brain images or signals, calls for empirical evaluation of its predictive power. Such evaluation is achieved via cross-validation, a method also used to tune decoders' hyper-parameters. This paper is a review on cross-validation procedures for decoding in neuroimaging. It includes a didactic overview of the relevant theoretical considerations. Practical aspects are highlighted with an extensive empirical study of the common decoders in within- and across-subject predictions, on multiple datasets –anatomical and functional MRI and MEG– and simulations. Theory and experiments outline that the popular “leave-one-out” strategy leads to unstable and biased estimates, and a repeated random splits method should be preferred. Experiments outline the large error bars of cross-validation in neuroimaging settings: typical confidence intervals of 10%. Nested cross-validation can tune decoders' parameters while avoiding circularity bias. However we find that it can be favorable to use sane defaults, in particular for non-sparse decoders. [Display omitted] •We give a primer on cross-validation to measure decoders predictive power.•We assess on many datasets its practical use for decoding selection and tuning.•Cross-validation displays large confidence intervals, in particular leave one out.•Default parameters on standard decoders can outperform parameter tuning.

• Publisher: Elsevier Inc,Elsevier
• Subject: Bagging / Bioengineering / Brain Diseases - diagnostic imaging / Cognitive science / Cognitive Sciences / Computer Science / Cross-validation / Decoding / FMRI / Humans / Life Sciences / Machine Learning / Medical Imaging / Model selection / MVPA / Neuroimaging - methods / Neuroimaging - standards / Neurons and Cognition / Neuroscience / Sparse / Statistics