Journal Article

How much does the typical ecological meta‐analysis overestimate the true mean effect size?

Fox, Jeremy W.

2022

Overview

Many primary research studies in ecology are underpowered, providing very imprecise estimates of effect size. Meta‐analyses partially mitigate this imprecision by combining data from different studies. But meta‐analytic estimates of mean effect size may still remain imprecise, particularly if the meta‐analysis includes a small number of studies. Imprecise, large‐magnitude estimates of mean effect size from small meta‐analyses likely would shrink if additional studies were conducted (regression towards the mean). Here, I propose a way to estimate and correct this regression to the mean, using meta‐meta‐analysis (meta‐analysis of meta‐analyses). Hierarchical random effects meta‐meta‐analysis shrinks estimated mean effect sizes from different meta‐analyses towards the grand mean, bringing those estimated means closer on average to their unknown true values. The intuition is that, if a meta‐analysis reports a mean effect size much larger in magnitude than that reported by other meta‐analyses, that large mean effect size likely is an overestimate. This intuition holds even if different meta‐analyses of different topics have different true mean effect sizes. Drawing on a compilation of data from hundreds of ecological meta‐analyses, I find that the typical (median) ecological meta‐analysis overestimates the absolute magnitude of the true mean effect size by ~10%. Some small ecological meta‐analyses overestimate the magnitude of the true mean effect size by >50%. Meta‐meta‐analysis is a promising tool for improving the accuracy of meta‐analytic estimates of mean effect size, particularly estimates based on just a few studies. Meta‐analytic estimates of mean effect size can be imprecise and overestimate effect magnitude, particularly if the meta‐analysis includes few studies. Here, I use meta‐meta‐analysis (meta‐analysis of meta‐analyses) to quantify and correct for overestimation of the magnitude of mean effect sizes in ecological meta‐analyses. The typical (median) ecological meta‐analysis overestimates the magnitude of the mean effect size by ~10%, and some meta‐analyses overestimate the magnitude of the mean effect size by >50%.

• Publisher: John Wiley & Sons, Inc,John Wiley and Sons Inc,Wiley
• Subject: Agroecology / Applied Ecology / Autecology / Behavioural Ecology / best linear unbiased predictions / Biodiversity Ecology / Biogeochemistry / Biogeography / Chemical Ecology / Community Ecology / Conservation Ecology / Demography / Disease Ecology / Ecological effects / Ecological Genetics / Ecophysiology / Ecosystem Ecology / Ecosystem Services Studies / Ecotoxicology / effect size / Estimates / Evolutionary Ecology / Functional Ecology / Global Change Ecology / Global Ecology / hierarchical model / Invasion Ecology / Landscape Ecology / Life History Ecology / Macroecology / Meta-analysis / meta‐meta‐analysis / Microbial Ecology / Movement Ecology / Normal distribution / Phenology / Population Ecology / random effects / Regression to mean effect / Restoration Ecology / Sampling error / Seed Ecology / Sensory Ecology / shrinkage / Soil Ecology / Spatial Ecology / Trophic Interactions / Urban Ecology