Journal Article

Key concepts in clinical epidemiology: detecting and dealing with heterogeneity in meta-analyses

Cordero, Cynthia P., Dans, Antonio L.

2021

Overview

In a meta-analysis, a question always arises. Is it worthwhile to combine estimates from studies of different populations using various formulations of an intervention, evaluating outcomes measured differently? Sometimes even study designs differ. Differences are expected in a meta-analysis. These may be negligible, and a pooled estimate of effect can guide the clinical decision. However, when the differences are large, this estimate may mislead. Effect estimates from study to study differ because of real differences (between-study variability) and because of chance (within-study variability). To combine estimates when there is heterogeneity (between-study differences are large) may not be sensible. Two complementary methods may be used to detect heterogeneity: visual inspection of the forest plot and calculating numerical measures of heterogeneity (I2 and Q). Visual inspection can show effects that are different from the rest. A large I2 (proportion of overall variability attributed to between-study variation) or a small P-value associated with Q may suggest heterogeneity. Large P-values, however, do not mean the absence of heterogeneity. It is more informative to report the confidence interval of the I2. If there is no heterogeneity, a pooled estimate of the true effect may be generated using only within-study variation (fixed-effect model). If there is substantial heterogeneity, reasons should be sought. Subgroup analysis or meta-regression using study-level characteristics may be done. Although more involved and potentially challenging, individual-level data (Individual Participant Data, IPD) may also be used. In the case of unexplained heterogeneity, both within- and between-study variation should be used to generate a pooled estimate (random-effects model). This estimate does not estimate a single true effect but estimates the average of a range of effects of the intervention on populations represented by the studies. If precise enough (narrow confidence interval), this estimate, together with the prediction interval (a measure of uncertainty in the effect one might see in a particular context), can guide clinical and policy decisions. •While differences are expected in a meta-analysis, these may be negligible, and a pooled estimate can guide the clinical decision. However, when the differences are large, this estimate may mislead.•The danger of reporting pooled estimates is that readers may overlook the overall picture—some studies having bigger effects than the other studies, some effects with different directions (harm) from the benefit shown by most studies. A careful inspection of the forest plot can help detect these differences; we refer to as heterogeneity.•Visual inspection should be used together with measures of heterogeneity–I2 and Q. High values of I2 and small P-values associated with Q may suggest heterogeneity. But large P-values do not mean the absence of heterogeneity. It is more informative to report the confidence interval of I2.•If heterogeneity is detected, an explanation must be sought, and analysis using study-level characteristics (subgroup analysis or meta-regression) may be done. Although intensive, analysis using individual-level data (Individual Participant Data) may also be done.•In case of unexplained heterogeneity, a pooled estimate using the random-effects model may be used. This estimate no longer estimates a single unknown effect but the average of the effects of the intervention in the populations represented by the studies. If precise enough (narrow confidence interval), this estimate, together with the prediction interval (a measure of uncertainty in the effect one might see in a particular context), can guide clinical and policy decisions.

• Publisher: Elsevier Inc,Elsevier Limited
• Subject: Age / Confidence intervals / Diarrhea / Epidemiology / Estimates / Fixed-effect model / Heterogeneity / Individual participant data / Inspection / Internal Medicine / Mathematical analysis / Meta-analyses / Meta-analysis / Meta-regression / Population studies / Populations / Prediction interval / Q-statistic / Random-effects model / Regression analysis / Statistical analysis / Subgroup analyses / Subgroups / Variability / Variation / Visual effects