Journal Article

The AMSTAR 2 publication lacks explicit instructions on how to assess the appropriateness of statistical methods (item 11) and publication bias (item 15) and does not reflect advances in meta-analysis

Spineli, Loukia M.

2025

Overview

As the quantity of published systematic reviews has increased substantially over the years, concerns about the methodological quality of this growing body of literature have been validly raised. AMSTAR 2, the updated version of AMSTAR, is an endorsed appraisal instrument aiming to critically assess the methodological aspects of a systematic review from conception to conduct and interpretation of the findings. However, since the publication of AMSTAR 2, several critiques have been expressed targeting various aspects of the instrument. The present commentary focuses on the AMSTAR 2 items that involve the appropriateness of statistical methods (item 11) and publication bias (item 15). The refinements are based on the methodological advances in meta-analysis as summarized in the Cochrane Handbook and delineated in review methodological studies. Initially, the commentary outlines further issues and challenges with the formulation and implementation of AMSTAR 2 items 11 and 15, beyond those already raised by other authors. Then, refinements to the corresponding decision points of items 11 and 15 are suggested, with explanations for their importance in facilitating an evidence-based, transparent, and consistent evaluation among the involved appraisers. The commentary strongly recommends that appraisers consult with meta-analysts when assessing the statistical methods of a systematic review and refer to the Cochrane Handbook, as it is regularly updated with recent methodological advances in meta-analysis. The appraisal teams could use the suggested refinements as a basis to predetermine the decision points for items 11 and 15 that align with the statistical expectations of the assessed systematic review, thereby preventing any ambiguity during the rating process. Systematic reviews have been established as an essential research design to detect knowledge gaps and prepare evidence-based clinical guidelines, among others. High reporting and methodological quality are crucial for systematic reviews to meet the intended goals. The rapid advances in systematic review methodology and software availability have led to a staggering number of published systematic reviews. Several empirical studies have already revealed limitations in the methodological and reporting quality of many systematic reviews from various medical fields. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses statement and AMSTAR (A MeaSurement Tool to Assess systematic Reviews) instrument were developed in response to the concerning number of systematic reviews with questionable quality. AMSTAR was updated to AMSTAR 2 in 2017 to meet the advances in the systematic review methodology. AMSTAR 2 has been widely used by several researchers to critically appraise the methodological quality of published systematic reviews. However, several authors have raised valid points regarding the wording and completeness of several items underlying the AMSTAR 2 tool. In the present commentary, I focus on items 11 and 15 of AMSTAR 2 that revolve around meta-analysis and publication bias. I propose significant refinements to meet the advances in meta-analysis and align with the Cochrane Handbook, which is a recognized standard in preparing, maintaining, and promoting high-quality systematic reviews. The proposed refinements for item 11 elaborate on the appropriateness of the random-effects model for a typical systematic review that is based on literature searches, the interpretation of the summary treatment effect based on the Hartung-Knapp adjustment and prediction interval (when at least five studies populate the meta-analysis and statistical heterogeneity is not excessive), and the diligent assessment of sources of statistical heterogeneity. The refinements of item 15 emphasize the assessment of the risk of missing studies and outcomes using a validated tool, the appropriate interpretation of the relevant statistical analyses (if applied), and the acknowledgment of the implications for the conclusions. I use two published systematic reviews to help the appraisers apply the refined items 11 and 15 correctly. [Display omitted] •The proposed refinements to items 11 and 15 are built on best practices in meta-analysis.•They warrant an evidence-based assessment of the appropriateness of statistical methods.•The decision points for items 11 and 15 should align with the statistical expectations of the review.•AMSTAR 2 appraisers should assess the risk of missing evidence and use a sensitivity analysis.•AMSTAR 2 appraisers should consult with meta-analysts and the Cochrane Handbook.

• Publisher: Elsevier Inc,Elsevier Limited
• Subject: AMSTAR 2 / Appraisers / Bias / Data Interpretation, Statistical / Generalized linear models / Handbooks / Heterogeneity / Humans / Internal Medicine / Knowledge management / Literature reviews / Meta-analysis / Meta-Analysis as Topic / Methodological quality / Parameter estimation / Publication bias / Publication Bias - statistics & numerical data / Research design / Research Design - standards / Risk assessment / Sensitivity analysis / Statistical analysis / Statistical heterogeneity / Statistical methods / Statistics / Systematic review / Systematic Reviews as Topic