Journal Article

Does Peer Review Work? An Experiment of Experimentalism

Ho, Daniel E.

2017

Overview

Ensuring the accuracy and consistency of highly decentralized and discretionary decisionmaking is a core challenge for the administrative state. The widely influential school of \"democratic experimentalism\" posits that peer review—the direct and deliberative evaluation of work product by peers in the discipline—provides a way forward, but systematic evidence remains limited. This Article provides the first empirical study of the feasibility and effects of peer review as a governance mechanism based on a unique randomized controlled trial conducted with the largest health department in Washington State (Public Health—Seattle and King County). We randomly assigned half of the food safety inspection staff to engage in an intensive peer review process for over four months. Pairs of inspectors jointly visited establishments, separately assessed health code violations, and deliberated about divergences on health code implementation. Our findings are threefold. First, observing identical conditions, inspectors disagreed 60% of the time. These joint inspection results in turn helped to pinpoint challenging code items and to develop training and guidance documents efficiently during weekly sessions. Second, analyzing over 28,000 independently conducted inspections across the peer review and control groups, we find that the intervention caused an increase in violations detected and scored by 17% to 19%. Third, peer review appeared to decrease variability across inspectors, thereby improving the consistency of inspections. As a result of this trial, King County has now instituted peer review as a standard practice. Our study has rich implications for the feasibility, promise, practice, and pitfalls of peer review, democratic experimentalism, and the administrative state.

• Publisher: students of the Stanford Law School,Stanford Law School,Stanford University, Stanford Law School
• Subject: ADMINISTRATIVE LAW / Attitudes / CITIZENSHIP / Collective bargaining / Company business management / Court decisions and opinions / Emigration and immigration law / Evaluation / Examiners (Administrative procedure) / Food adulteration and inspection / Food inspection / Food safety / IMMIGRATION / Inspections / Judges / JUDICIARY / Labor contracts / LANGUAGE / Management / Occupational safety / Peer review / Public administration / PUBLIC HEALTH / Salmonella / Scholars / Social security / U.S. states / United States. Social Security Administration