Journal Article

False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant

Simmons, Joseph P., Nelson, Leif D., Simonsohn, Uri

2011

Overview

In this article, we accomplish two things. First, we show that despite empirical psychologists' nominal endorsement of a low rate of false-positive findings (< .05), flexibility in data collection, analysis, and reporting dramatically increases actual false-positive rates. In many cases, a researcher is more likely to falsely find evidence that an effect exists than to correctly find evidence that it does not. We present computer simulations and a pair of actual experiments that demonstrate how unacceptably easy it is to accumulate (and report) statistically significant evidence for a false hypothesis. Second, we suggest a simple, low-cost, and straightforwardly effective disclosure-based solution to this problem. The solution involves six concrete requirements for authors and four guidelines for reviewers, all of which impose a minimal burden on the publication process.

• Publisher: SAGE Publications,Sage Publications,SAGE PUBLICATIONS, INC
• Subject: Adult / Ambiguity / Biological and medical sciences / Childrens songs / Cognition & reasoning / Computer Simulation / Data analysis / Data collection / Data Collection - standards / Data Interpretation, Statistical / Degrees of freedom / Empirical research / Experiments / False positive errors / False positive results / Flexibility / Fundamental and applied biological sciences. Psychology / Humans / Hypotheses / Methodology / Methodology. Experimentation / Motivation / Observational research / Peer Review, Research - standards / Positive psychology / Practice Guidelines as Topic / Psychologists / Psychology / Psychology. Psychoanalysis. Psychiatry / Psychology. Psychophysiology / Psychometrics. Statistics. Methodology / Publications - standards / Reasoning / Research Design - standards / Research Personnel - psychology / Research Personnel - standards / Sample size / Simulation / Statistical results / Statistical significance / Statistics as Topic / Statistics. Mathematics / T tests / Young Adult