Explore the vast range of titles available.

Background Methodological studies – studies that evaluate the design, analysis or reporting of other research-related reports – play an important role in health research. They help to highlight issues in the conduct of research with the aim of improving health research methodology, and ultimately reducing research waste. Main body We provide an overview of some of the key aspects of methodological studies such as what they are, and when, how and why they are done. We adopt a “frequently asked questions” format to facilitate reading this paper and provide multiple examples to help guide researchers interested in conducting methodological studies. Some of the topics addressed include: is it necessary to publish a study protocol? How to select relevant research reports and databases for a methodological study? What approaches to data extraction and statistical analysis should be considered when conducting a methodological study? What are potential threats to validity and is there a way to appraise the quality of methodological studies? Conclusion Appropriate reflection and application of basic principles of epidemiology and biostatistics are required in the design and analysis of methodological studies. This paper provides an introduction for further discussion about the conduct of methodological studies.

AbstractObjectiveTo compare estimated treatment effects of physical therapy (PT) between Patient-Reported Outcome Measures (PROMs) and outcomes measured in other ways. Study Design and SettingWe selected randomized trials of PT with both a PROM and a non-PROM included in Cochrane Systematic Reviews (CSRs). Two reviewers independently extracted data and risk-of-bias assessments. Our primary outcome was the ratio of odds ratios (ROR), used to quantify how effect vary between PROMs and non-PROMs; an ROR > 1 indicates larger effect when assessed by PROMs. We used REML-methods to estimate associations of trial characteristics with effects and between-trial heterogeneity. ResultsFrom 90 relevant CSRs, 205 PT trials were included. The summary ROR across all the comparisons was not statistically significant (ROR, 0.88 [95% CI: 0.70-1.12]; P=0.30); however, the heterogeneity was substantial (I 2=88.1%). When stratifying non-PROMs further into clearly objective non-PROMs (e.g., biomarkers) and other non-PROMs (e.g., aerobic capacity), the PROMs appeared more favorable than did clearly objective non-PROMs (ROR, 1.92 [95% CI: 0.99-3.72]; P=0.05). ConclusionEstimated treatment effects based on PROMs are generally comparable to treatment effects measured in other ways. However, in our study, PROMs indicate a more favorable treatment effect compared to treatment effects based on clearly objective outcomes.

To assess the characteristics and core statistical methodology specific to network meta-analyses (NMAs) in clinical research articles. We searched MEDLINE, EMBASE, and the Cochrane Database of Systematic Reviews from inception until April 14, 2015, for NMAs of randomized controlled trials including at least four different interventions. Two reviewers independently screened potential studies, whereas data abstraction was performed by a single reviewer and verified by a second. A total of 456 NMAs, which included a median (interquartile range) of 21 (13–40) studies and 7 (5–9) treatment nodes, were assessed. A total of 125 NMAs (27%) were star networks; this proportion declined from 100% in 2005 to 19% in 2015 (P = 0.01 by test of trend). An increasing number of NMAs discussed transitivity or inconsistency (0% in 2005, 86% in 2015, P < 0.01) and 150 (45%) used appropriate methods to test for inconsistency (14% in 2006, 74% in 2015, P < 0.01). Heterogeneity was explored in 256 NMAs (56%), with no change over time (P = 0.10). All pairwise effects were reported in 234 NMAs (51%), with some increase over time (P = 0.02). The hierarchy of treatments was presented in 195 NMAs (43%), the probability of being best was most commonly reported (137 NMAs, 70%), but use of surface under the cumulative ranking curves increased steeply (0% in 2005, 33% in 2015, P < 0.01). Many NMAs published in the medical literature have significant limitations in both the conduct and reporting of the statistical analysis and numerical results. The situation has, however, improved in recent years, in particular with respect to the evaluation of the underlying assumptions, but considerable room for further improvements remains.

To evaluate the characteristics of therapeutic meta-analyses including both observational studies and randomized controlled trials (RCTs), how these studies were combined and whether there were differences in treatment effects. Meta-epidemiological study of meta-analyses, including both observational studies and RCTs. We searched MEDLINE for the five leading journals of each medical category according to Journal Citation Reports) and Cochrane Database of Systematic Reviews, from 2014 to 2018 for eligible meta-analyses and extracted how observational studies and RCTs were combined and results for each study. Of the 102 included meta-analyses, observational studies and RCTs were combined together without a subgroup analysis in 39 (38%) and with subgroup analysis in 15 (15%); they were pooled separately for the same outcome in 11 (11%) and not for the same outcome in 9 (9%). In 28 (27%) meta-analyses, only RCTs were combined, with a qualitative description of observational studies. Treatment effect estimates did not differ between observational studies and RCTs (ratio of estimates = 0.98 [95% confidence interval 0.80–1.21]), with substantial heterogeneity (I2 = 59%). Many meta-analyses, including both observational studies and RCTs pool results from both study types. Although treatment effects did not differ between them on average, we identified situations for which estimates differed. •Many meta-analyses combined observational studies with RCTs in meta-analysis. Overall, there was no difference in effects between RCTs and observational studies.•Heterogeneity across topics was substantial.•There were a few situations with significant differences between both study types.

•Trials published before 2000 represented one fourth of all trials included in meta-analyses and trials published before 1990 almost 10%. The oldest trial was published in 1951.•Intervention effects were, on average, significantly larger for older than recent trials.•Results were consistent in sensitivity analyses adjusted on risk of bias and sample size.•It is generally recommended to include all trials within a meta-analysis whatever their publication date but this may have an influence on external validity and intervention effect.•With a biomedical literature that is increasing exponentially, we wonder whether it is reasonable to consider the results of old trials sometimes conducted more than 50 years ago. To assess whether meta-analyses include older randomized controlled trials (RCTs) and whether intervention effect differ between older and recent RCTs. In this meta-epidemiological study of 295 meta-analyses (2940 RCTs) published in 2017–2018, we evaluated the difference in intervention effects between older (i.e., published before 2000) and recent RCTs. We also compared effects by quarters of publication year within each meta-analysis (from quarter 1 including the 25% oldest trials to quarter 4 including the 25% most recent trials). A ratio of odds ratio (ROR) <1 indicates larger effects in older than recent RCTs. Trials published before 2000 and before 1990 represented 25% and 10% of all trials, respectively. Intervention effects were significantly larger for old than recent RCTs (ROR = 0.92, 95% confidence interval [CI] 0.85–1.00, I2 = 22%). Compared with the most recent trials (quarter 4), intervention effects were significantly larger for the oldest trials (quarter 1) (ROR = 0.85, 95% CI 0.79–0.92) and for trials in quarter 2 (ROR = 0.89, 95% CI 0.83–0.96) but not for trials in quarter 3 (ROR = 0.98, 95% CI 0.91–1.05). Intervention effects were larger for older than recent RCTs. Meta-analyses including older trials should be interpreted cautiously.

To assess to what extent the overall quality of evidence indicates changes to observe intervention effect estimates when new data become available. We conducted a meta-epidemiological study. We obtained evidence from meta-analyses of randomized trials of Cochrane reviews addressing the same health-care question that was updated with inclusion of additional data between January 2016 and May 2021. We extracted the reported effect estimates with 95% confidence intervals (CIs) from meta-analyses and corresponding GRADE (Grading of Recommendations Assessment, Development, and Evaluation) assessments of any intervention comparison for the primary outcome in the first and the last updated review version. We considered the reported overall quality (certainty) of evidence (CoE) and specific evidence limitations (no, serious or very serious for risk of bias, imprecision, inconsistency, and/or indirectness). We assessed the change in pooled effect estimates between the original and updated evidence using the ratio of odds ratio (ROR), absolute ratio of odds ratio (aROR), ratio of standard errors (RoSE), direction of effects, and level of statistical significance. High CoE without limitations characterized 19.3% (n = 29) out of 150 included original Cochrane reviews. The update with additional data did not systematically change the effect estimates (mean ROR 1.00; 95% CI 0.99–1.02), which deviated 1.06-fold from the older estimates (median aROR; interquartile range [IQR]: 1.01–1.15), gained precision (median RoSE 0.87; IQR 0.76–1.00), and maintained the same direction with the same level of statistical significance in 93% (27 of 29) of cases. Lower CoE with limitations characterized 121 original reviews and graded as moderate CoE in 30.0% (45 of 150), low CoE in 32.0% (48 of 150), and very low CoE in 18.7% (28 of 150) reviews. Their update had larger absolute deviations (median aROR 1.12 to 1.33) and larger gains in precision (median RoSE 0.78–0.86) without clear and consistent differences between these categories of CoE. Changes in effect direction or statistical significance were also more common in the lower quality evidence, again with a similar extent across categories (without change in 75.6%, 64.6%, and 75.0% for moderate, low, very low CoE). As limitations increased, effect estimates deviated more (aROR 1.05 with zero, 1.11 with one, 1.25 with two, 1.24 with three limitations) and changes in direction or significance became more frequent (93.2% stable with no limitations, 74.5% with one, 68.2% with two, and 61.5% with three limitations). High-quality evidence without methodological deficiencies is trustworthy and stable, providing reliable intervention effect estimates when updated with new data. Evidence of moderate and lower quality may be equally prone to being unstable and cannot indicate if available effect estimates are true, exaggerated, or underestimated.

To assess the proportion of the recent Cochrane reviews that included outcomes in their literature search strategy, how often they acknowledged these limitations, and how qualitatively different the results of outcomes included and not included in the search strategy were. We identified all the Cochrane reviews of the interventions published in 2020 that used a search strategy connecting outcome terms with “AND.” Reviews were defined as acknowledging the limitations of searching for outcomes if they mentioned them in the discussion. We compared the characteristics of outcomes included and not included in the search strategy. Of the 523 Cochrane reviews published in 2020, 51 (9.8%) included outcomes in their search strategy. Only one review acknowledged it as a limitation. Forty-seven (92%) assessed outcomes not included in the search strategy. Outcomes included in the search strategies tended to include a larger number of studies and show their effects in favor of the intervention. Around ten percent of the recent Cochrane reviews included outcomes in their search, which may have resulted in more outcomes significantly in favor of the intervention. Reviewers should be more explicit in acknowledging the potential implications of searching for outcomes.

To investigate the meta-epidemiology and reporting characteristics of mapping reviews. We conducted a scoping review of a sample of recent mapping reviews (2022–2023) by searching nine electronic databases and eleven institutional websites up to January 2024. A 28-item reporting checklist, developed by our team and based on existing guidance and methodological studies of mapping reviews, was employed to assess reporting characteristics. The median (interquartile range [IQR]) was calculated based on the number of studies that reported each specified reporting item. A total of 451 mapping reviews were included, with 197 published in 2022 and 254 in 2023. The USA published the most studies (80, 17.74%) and the University of London being the most active institution (23, 5.10%). Barbosa JLV from Universidade do Vale do Rio dos Sinos authored the highest number of studies, with significant collaborations primarily between Chinese and Canadian teams. The studies spanned 11 fields within health and social sciences, Environmental Evidence and Campbell Systematic Reviews published the most studies. A random sample of 200 studies was further assessed for reporting characteristics, where items such as title, author, abstract, background, conclusions, acknowledgments, contributions of authors, declarations of interest, and sources of support were reported by over 50% of the studies. Median number of studies reporting items related to methods, results, and discussion was 89.5 (IQR: 45.5, 165.25), 94.0 (IQR: 61.5, 183.5), and 164 (IQR: 124.0, 179.25), respectively. Items such as the definition of stakeholders, registration information, deviations from protocol, strategy for adequacy and priority setting, and plans for map updates were reported by less than 20% of the studies. Mapping reviews receive contributions from authors across various countries and institutions. However, collaboration between teams is limited, and the reporting of included studies, especially in the methods and results sections, needs improvement to better emphasize their unique attributes and functions compared to other methods. •Meta-epidemiological analysis of 451 mapping reviews conducted from 2022 to 2023.•Application of a 28-item reporting checklist to assess reporting characteristics.•Found gaps in reporting stakeholder, registration, and adequacy/priority setting.•Noted the rapid growth of mapping reviews across 69 countries.•Provided recommendations to enhance the future mapping reviews' comprehensiveness.

To undertake meta-analysis and compare treatment effects estimated by the intention-to-treat (ITT) method and per-protocol (PP) method in randomized controlled trials (RCTs). PP excludes trial participants who are non-adherent to trial protocol in terms of eligibility, interventions, or outcome assessment. Five high impact journals were searched for all RCTs published between July 2017 to June 2019. Primary outcome was a pooled estimate that quantified the difference between the treatment effects estimated by the two methods. Results are presented as ratio of odds ratios (ROR). Meta-regression was used to explore the association between level of trial protocol non-adherence and treatment effect. Sensitivity analyses compared results with varying within-study correlations and across various study characteristics. Random-effects meta-analysis (N = 156) showed that PP estimates were on average 2% greater compared to the ITT estimates (ROR: 1.02, 95% CI: 1.00–1.04, P = 0.03). The divergence further increased with higher degree of protocol non-adherence. Sensitivity analyses reassured consistent results with various within-study correlations and across various study characteristics. There was evidence of larger treatment effect with PP compared to ITT analysis. PP analysis should not be used to assess the impact of protocol non-adherence in RCTs. Instead, in addition to ITT, investigators should consider randomization based casual method such as Complier Average Causal Effect (CACE).

Flaws in the conduct of randomized trials can lead to biased estimation of the intervention effect. Methods for adjustment of within-trial biases in meta-analysis include the use of empirical evidence from an external collection of meta-analyses, and the use of expert opinion informed by the assessment of detailed trial information. Our aim is to present methods to combine these two approaches to gain the advantages of both. We make use of the risk of bias information that is routinely available in Cochrane reviews, by obtaining empirical distributions for the bias associated with particular bias profiles (combinations of risk of bias judgements). We propose three methods: a formal combination of empirical evidence and opinion in a Bayesian analysis; asking experts to give an opinion on bias informed by both summary trial information and a bias distribution from the empirical evidence, either numerically or by selecting areas of the empirical distribution. The methods are demonstrated through application to two example binary outcome meta-analyses. Bias distributions based on opinion informed by trial information alone were most dispersed on average, and those based on opinions obtained by selecting areas of the empirical distribution were narrowest. Although the three methods for combining empirical evidence with opinion vary in ease and speed of implementation, they yielded similar results in the two examples.