Journal Article

Problematic meta-analyses: Bayesian and frequentist perspectives on combining randomized controlled trials and non-randomized studies

Linden, Ariel, Moran, John L.

2024

Overview

Purpose In the literature, the propriety of the meta-analytic treatment-effect produced by combining randomized controlled trials (RCT) and non-randomized studies (NRS) is questioned, given the inherent confounding in NRS that may bias the meta-analysis. The current study compared an implicitly principled pooled Bayesian meta-analytic treatment-effect with that of frequentist pooling of RCT and NRS to determine how well each approach handled the NRS bias. Materials & methods Binary outcome Critical-Care meta-analyses, reflecting the importance of such outcomes in Critical-Care practice, combining RCT and NRS were identified electronically. Bayesian pooled treatment-effect and 95% credible-intervals (BCrI), posterior model probabilities indicating model plausibility and Bayes-factors (BF) were estimated using an informative heavy-tailed heterogeneity prior (half-Cauchy). Preference for pooling of RCT and NRS was indicated for Bayes-factors > 3 or < 0.333 for the converse. All pooled frequentist treatment-effects and 95% confidence intervals (FCI) were re-estimated using the popular DerSimonian-Laird (DSL) random effects model. Results Fifty meta-analyses were identified (2009–2021), reporting pooled estimates in 44; 29 were pharmaceutical-therapeutic and 21 were non-pharmaceutical therapeutic. Re-computed pooled DSL FCI excluded the null (OR or RR = 1) in 86% (43/50). In 18 meta-analyses there was an agreement between FCI and BCrI in excluding the null. In 23 meta-analyses where FCI excluded the null, BCrI embraced the null. BF supported a pooled model in 27 meta-analyses and separate models in 4. The highest density of the posterior model probabilities for 0.333 < Bayes factor < 1 was 0.8. Conclusions In the current meta-analytic cohort, an integrated and multifaceted Bayesian approach gave support to including NRS in a pooled-estimate model. Conversely, caution should attend the reporting of naïve frequentist pooled, RCT and NRS, meta-analytic treatment effects.

• Publisher: BioMed Central,BioMed Central Ltd,Springer Nature B.V,BMC
• Subject: Bayes factors / Bayes Theorem / Bayesian / Bayesian statistical decision theory / Bias / Clinical trials / Critical care / Estimates / Frequentist / Health Sciences / Humans / Intensive care / Medicine / Medicine & Public Health / Meta-analysis / Meta-Analysis as Topic / Methods / Models, Statistical / Non-Randomized Controlled Trials as Topic - methods / Posterior probability / Probability / Randomized Controlled Trials as Topic - methods / Randomized Controlled Trials as Topic - statistics & numerical data / Statistical Theory and Methods / Statistics for Life Sciences / Theory of Medicine/Bioethics