Journal Article

Statistical learning and selective inference

Robert J. Tibshirani, Taylor, Jonathan

2015

Overview

We describe the problem of “selective inference.” This addresses the following challenge: Having mined a set of data to find potential associations, how do we properly assess the strength of these associations? The fact that we have “cherry-picked”—searched for the strongest associations—means that we must set a higher bar for declaring significant the associations that we see. This challenge becomes more important in the era of big data and complex statistical modeling. The cherry tree (dataset) can be very large and the tools for cherry picking (statistical learning methods) are now very sophisticated. We describe some recent new developments in selective inference and illustrate their use in forward stepwise regression, the lasso, and principal components analysis. Significance Most statistical analyses involve some kind of ”selection”—searching through the data for the strongest associations. Measuring the strength of the resulting associations is a challenging task, because one must account for the effects of the selection. There are some new tools in selective inference for this task, and we illustrate their use in forward stepwise regression, the lasso, and principal components analysis.

• Publisher: National Academy of Sciences,National Acad Sciences
• Subject: Datasets as Topic / Fruit trees / INAUGURAL ARTICLE / inference / lasso / Learning / methodology / Models, Statistical / P values / Physical Sciences / principal component analysis / Principal components analysis / Regression analysis / Statistical methods / Statistical models