Explore the vast range of titles available.

Ensuring the accuracy and consistency of highly decentralized and discretionary decisionmaking is a core challenge for the administrative state. The widely influential school of \"democratic experimentalism\" posits that peer review—the direct and deliberative evaluation of work product by peers in the discipline—provides a way forward, but systematic evidence remains limited. This Article provides the first empirical study of the feasibility and effects of peer review as a governance mechanism based on a unique randomized controlled trial conducted with the largest health department in Washington State (Public Health—Seattle and King County). We randomly assigned half of the food safety inspection staff to engage in an intensive peer review process for over four months. Pairs of inspectors jointly visited establishments, separately assessed health code violations, and deliberated about divergences on health code implementation. Our findings are threefold. First, observing identical conditions, inspectors disagreed 60% of the time. These joint inspection results in turn helped to pinpoint challenging code items and to develop training and guidance documents efficiently during weekly sessions. Second, analyzing over 28,000 independently conducted inspections across the peer review and control groups, we find that the intervention caused an increase in violations detected and scored by 17% to 19%. Third, peer review appeared to decrease variability across inspectors, thereby improving the consistency of inspections. As a result of this trial, King County has now instituted peer review as a standard practice. Our study has rich implications for the feasibility, promise, practice, and pitfalls of peer review, democratic experimentalism, and the administrative state.

COVID-19 exposed and exacerbated health disparities, and a core challenge has been how to adapt pandemic response and public health in light of these disproportionate health burdens. Responding to this challenge, the County of Santa Clara Public Health Department designed a model of “high-touch” contact tracing that integrated social services with disease investigation, providing continued support and resource linkage for clients from structurally vulnerable communities. We report results from a cluster randomized trial of 5,430 cases from February to May 2021 to assess the ability of high-touch contact tracing to aid with isolation and quarantine. Using individual-level data on resource referral and uptake outcomes, we find that the intervention, randomized assignment to the high-touch program, increased the referral rate to social services by 8.4% (95% confidence interval, 0.8%-15.9%) and the uptake rate by 4.9% (-0.2%-10.0%), with the most pronounced increases in referrals and uptake of food assistance. These findings demonstrate that social services can be effectively combined with contact tracing to better promote health equity, demonstrating a novel path for the future of public health.

Although published works rarely include causal estimates from more than a few model specifications, authors usually choose the presented estimates from numerous trial runs readers never see. Given the often large variation in estimates across choices of control variables, functional forms, and other modeling assumptions, how can researchers ensure that the few estimates presented are accurate or representative? How do readers know that publications are not merely demonstrations that it is possible to find a specification that fits the author's favorite hypothesis? And how do we evaluate or even define statistical properties like unbiasedness or mean squared error when no unique model or estimator even exists? Matching methods, which offer the promise of causal inference with fewer assumptions, constitute one possible way forward, but crucial results in this fast-growing methodological literature are often grossly misinterpreted. We explain how to avoid these misinterpretations and propose a unified approach that makes it possible for researchers to preprocess data with matching (such as with the easy-to-use software we offer) and then to apply the best parametric techniques they would have used anyway. This procedure makes parametric models produce more accurate and considerably less model-dependent causal inferences.

A comprehensive overview of medical AI devices approved by the US Food and Drug Administration sheds new light on limitations of the evaluation process that can mask vulnerabilities of devices when they are deployed on patients.

The Food and Drug Administration's menu labeling rule requires chain restaurants to prominently display calories, while leaving other nutritional information (e.g., fat, sodium, sugar) to the request of consumers. We use rich micronutrient data from 257 large chain brands and 24,076 menu items to examine whether calories are correlated with widely used \"nutrient profile\" scores that measure healthfulness based on nutrient density. We show that calories are indeed statistically significant predictors of nutrient density. However, as a substantive matter, the correlation is highly attenuated (partial R2 < 0.01). Our findings (a) suggest that the promise of calorie labeling to improve nutrient intake quality at restaurants is limited and (b) clarify the basis for transparency of nutrient composition beyond calories to promote healthy menu choices.

Goldberg v. Kelly and its progeny imposed a judicial model for decisionmaking on much of the administrative state. The linchpin of procedural due process was accuracy: Goldberg’s premise was that agencies could improve the accuracy of their decisionmaking by giving individuals the sort of procedural rights enjoyed in court. In the wake of the due process revolution, federal agencies now adjudicate more cases than all Article III courts combined, and state adjudicators handle millions of cases with court-like procedures in their administrative systems. Yet despite Goldberg’s premise, mass adjudication has struggled to achieve an adequate threshold of accuracy. In much of the administrative state, this struggle has deepened into an urgent crisis. The leading academic response argues for a turn to “internal administrative law” and management techniques, not external law, to improve the quality of agency adjudication. Many agencies in turn have responded with such quality assurance programs, but we know next to nothing about how such programs have evolved, how they function, and whether they work. Our Article is the first to rigorously investigate the promise and pitfalls of quality assurance as a guarantor of accuracy in agency adjudication. We make three contributions. First, we use in-depth interviews with senior agency officials and a wide array of internal agency materials to document the evolving use of quality assurance at three federal agencies whose mass adjudication epitomizes Goldberg’s domain. This history documents years of fits and starts, as agencies tried to manage what is commonly referred to as a “quantity-quality” tradeoff. It also reveals deep tensions and ambiguities in what the agencies intend as the purpose of quality assurance. Second, we provide the first rigorous test of quality assurance, the leading academic response to Goldberg’s limitations. We use a rich dataset, never before available to outside academics, of over 500,000 cases decided by the Board of Veterans’ Appeals (BVA) to craft a rigorous evaluation of a natural experiment created by its “Quality Review” program. Under this program, cases were randomly selected for review of draft decisions by an elite squadron of attorneys to correct substantive legal errors. BVA used this program ostensibly to reduce appeals to and remands from the courts reviewing its decisions. We show that the program failed on its own terms: Cases selected for Quality Review fared no better than cases that were not. BVA used the program not to vindicate Goldberg’s premise, but to mollify external oversight bodies, most notably Congress, with the appearance of accuracy. Third, our historical and empirical evidence has substantial implications for major theoretical debates about “internal administrative law” and the emerging crisis in mass adjudication. We show that conventional scholarly accounts are in need of much refinement. Deficiencies in mass adjudication will not be fixed solely through external constitutional law, with courts imposing remedies from outside. Nor will they be fixed solely by internal administrative law. Goldberg’s original premise of decisional accuracy requires a hybrid of external intervention, stakeholder oversight, and internal agency management. We offer concrete policy prescriptions, based on a pilot one of us designed as BVA’s Chief of the Office for Quality Review, for how quality assurance might be reenvisioned to solve the looming crisis of decisional quality.

Contact tracing is a pillar of COVID-19 response, but language access and equity have posed major obstacles. COVID-19 has disproportionately affected minority communities with many non–English-speaking members. Language discordance can increase processing times and hamper the trust building necessary for effective contact tracing. We demonstrate how matching predicted patient language with contact tracer language can enhance contact tracing. First, we show how to use machine learning to combine information from sparse laboratory reports with richer census data to predict the language of an incoming case. Second, we embed this method in the highly demanding environment of actual contact tracing with high volumes of cases in Santa Clara County, CA. Third, we evaluate this language-matching intervention in a randomized controlled trial.We show that this low-touch intervention results in 1) significant time savings, shortening the time from opening of cases to completion of the initial interview by nearly 14 h and increasing same-day completion by 12%, and 2) improved engagement, reducing the refusal to interview by 4%. These findings have important implications for reducing social disparities in COVID-19; improving equity in healthcare access; and, more broadly, leveling language differences in public services.

Concentrated Animal Feeding Operations (CAFOs) house livestock at high densities for prolonged periods of time, posing substantial risks to environmental and human health. However, limited public information on CAFOs has constrained efforts to quantify their impacts on proximate communities. Gaps in permitting and reporting have severely limited studies that rely primarily on administrative records. We introduce Cal-FF, a near-complete census of CAFOs in California, a large and agriculturally significant state in the United States, with richer facility data than existing administrative data. Cal-FF was constructed using computer vision on satellite imagery, along with extensive human validation. We focus on California, which accounts for about 20% of US livestock production and has been at the forefront of CAFO regulatory innovation. We estimate that our 2,121 facility dataset captures 98% (95% CI [82%, 98%]) of all California CAFOs as of 2017, identifying 222 locations not present in state regulatory records. In addition to improved accuracy, Cal-FF adds a wealth of information about each facility, including validated permit information, land parcel data, satellite imagery, and annotated facility features. These data provide numerous opportunities for research, analysis, and monitoring.

Decades of research and advocacy have underscored the imperative of surfacing – as the first step towards mitigating – racial disparities, including among subgroups historically bundled into aggregated categories. Recent U.S. federal regulations have required increasingly disaggregated race reporting, but major implementation barriers mean that, in practice, reported race data continues to remain inadequate. While imputation methods have enabled disparity assessments in many research and policy settings lacking reported race, the leading name algorithms cannot recover disaggregated categories, given the same lack of disaggregated data from administrative sources to inform algorithm design. Leveraging a Wikidata sample of over 300,000 individuals from six Asian countries, we extract frequencies of 25,876 first names and 18,703 surnames which can be used as proxies for U.S. name-race distributions among six major Asian subgroups: Asian Indian, Chinese, Filipino, Japanese, Korean, and Vietnamese. We show that these data, when combined with public geography-race distributions to predict subgroup membership, outperform existing deterministic name lists in key prediction settings, and enable critical Asian disparity assessments.

Abstract Do large language models (LLMs) know the law? LLMs are increasingly being used to augment legal practice, education, and research, yet their revolutionary potential is threatened by the presence of “hallucinations”—textual output that is not consistent with legal facts. We present the first systematic evidence of these hallucinations in public-facing LLMs, documenting trends across jurisdictions, courts, time periods, and cases. Using OpenAI’s ChatGPT 4 and other public models, we show that LLMs hallucinate at least 58% of the time, struggle to predict their own hallucinations, and often uncritically accept users’ incorrect legal assumptions. We conclude by cautioning against the rapid and unsupervised integration of popular LLMs into legal tasks, and we develop a typology of legal hallucinations to guide future research in this area.