Journal Article

The performance of small sample correction methods for controlling type I error when analyzing parallel cluster randomized trials: a systematic review of simulation studies

Bowden, R., Watson, S., Forbes, A., McKenzie, J.E., Thompson, J., Kasza, J., Hemming, K., Kristunas, C.

2025

Overview

Most cluster randomized trials (CRTs) include fewer than 50 clusters, yet the assumption of a “large sample” is relied upon to derive the sampling distributions of treatment effects. We review the current simulation study literature pertaining to small sample corrections for common analytical approaches for parallel CRTs. We searched Ovid Medline and Web of Science up to August 30 2024 for simulation studies evaluating the performance of small sample corrections. We include binary and continuous outcomes analyzed using generalized linear mixed models, generalized estimating equations, or cluster-level approaches. Full-text screening and data abstraction was performed independently in duplicate. Fourteen studies evaluated binary outcomes and 6 studies continuous outcomes. The number of clusters ranged from 4 to 200; the median smallest intracluster correlation coefficient was 0.001 [range 0.000–0.200]; largest intracluster correlation coefficient was 0.10 [range 0.05–0.70]; lowest prevalence was 0.25 [range 0.05–0.50]; and median coefficient of variation of cluster sizes 1.00 [range: 0.80–1.50]. For continuous outcomes, a cluster-level analysis (either unweighted or inverse-variance weighted) with a t-distribution (with between-within degree of freedom); a linear mixed model with a Satterthwaite correction; or a generalized estimating equation with the Fay and Graubard correction mostly preserve nominal type I error with as few as six clusters (although up to 40 clusters in some settings). Other approaches work less favorably (eg, Kenward-Roger is conservative even with 30 clusters). For binary outcomes, an unweighted or inverse-variance weighted cluster-level analysis can achieve nominal type I error (but can be anticonservative with small cluster sizes or low prevalence); as can a generalized linear mixed model with a between-within correction with as few as 10 clusters (but sometimes conservative with up to 30 clusters). Other corrections such as the Kenward-Roger or Satterthwaite correction are more conservative. For generalized estimating equations, the Mancl and DeRouen correction mostly seems to preserve nominal errors but can be anticonservative. The literature on the performance of small sample corrections for parallel CRTs is complex. While the available corrections can maintain type I error with a very small number of clusters, more than 40 clusters are required to guarantee nominal type I error across all settings. •With fewer than 50 clusters, analysis of data from a cluster trial requires a small sample correction.•Small sample corrections mostly maintain type I error close to 5% or are conservative.•For continuous outcomes, Satterthwaite and Fay/Graubard maintain type I error with 6 clusters.•For binary outcomes, the between-within and Mancl and DeRouen can sometimes maintain type I error.

• Publisher: Elsevier Inc,Elsevier Limited
• Subject: Analysis / Approximation / Bias / Cluster Analysis / Cluster randomized trials / Cluster-level analysis / Clusters / Coefficient of variation / Computer Simulation / Control methods / Correlation coefficient / Correlation coefficients / Error correction / Generalized estimating equations / Generalized linear mixed models / Humans / Independent sample / Internal Medicine / Performance evaluation / Randomized Controlled Trials as Topic - methods / Randomized Controlled Trials as Topic - statistics & numerical data / Research Design / Sample Size / Small sample corrections / Statistical models / Statistics / Variance