Meta-analysis

This article is about Meta-analysis. For other uses of the term Analysis , please see Analysis (disambiguation).

Meta-analysis is defined as "a quantitative method of combining the results of independent studies (usually drawn from the published literature) and synthesizing summaries and conclusions which may be used to evaluate therapeutic effectiveness, plan new studies, etc., with application chiefly in the areas of research and medicine."

A meta-analyses is a subset of systematic reviews in which the results of the studies are numerically pooled.

Standards for the reporting of meta-analyses exist.

Validity of meta-analysis

Studies on the validity of meta-analyses conflict. Some of the conflict may be due to the methods used to compare the meta-analyses.

Methods of meta-analysis

Guidelines are available for the conduct and reporting of meta-analyses.

Searching for studies

Meta-analyses vary in the extent of their searches for underlying studies. No individual database contains all the existing randomized controlled trials; however, the Cochrane database may be the most comprehensive.

Machine learning and text categorization has been use for searching.

There is not a consensus on what details of searching should be reported in a meta-analysis.

Stopping rules

There is debate on how extensive should be the search for studies as there is are diminishing returns with extensive searching. Some studies suggest limiting searches while other studies advocate exhaustive searches including unpublished studies. The role of databases other than MEDLINE is not clear.

Various methods have been proposed for when to stop searchers.

Selecting studies for inclusion

Conflict in selection of trials to be included in the meta-analysis can affect the conclusions of a meta-analysis.

Although meta-analyses in general are very inclusive, arguments exist for only including the best trials.

Assessing the quality of trials

Assessing the quality of a trial by only using the published report may lead to inaccurate conclusions.

The formal methods below have some difficulty with reproducibility.

Cochrane bias scale

The Cochrane Collaboration uses a six item tool.

Jadad score

The Jadad score may be used to assess quality and contains three items:

1. Was the study described as randomized (this includes the use of words such as randomly, random, and randomization)?
2. Was the study described as double blind?
3. Was there a description of withdrawals and dropouts?

Each question is scored one point for a yes answer. In addition, for questions and 2, a point is added if the method was appropriate and a point is deducted if the method is not appropriate (e.g. not effectively randomized or not effectively double-blinded).

Statistical methods

Measuring consistency of study results

Consistency can be statistically tested using either the Cochran's Q or I². The I² is the "percentage of total variation across studies that is due to heterogeneity rather than chance." These numbers are usually displayed for each group of studies on a Forest plot.

In interpreting of the Cochran's Q, heterogeneity exists if its p-value is < 0.05 or possibly if < 0.10.

The following has been proposed for interpreting I²:

• Low heterogeneity is I² = 25%
• Moderate heterogeneity is I² = 50%
• High heterogeneity is I² = 75%

or according to the Handbook of the Cochrane Collaboration:

• 0%-40%: might not be important
• 30%-60%: may represent moderate heterogeneity
• 50%-90%: may represent substantial heterogeneity
• 75%-100%: considerable heterogeneity

However, I², even when the value is 0%, can be misleading if the confidence intervals around the value are not provided.

Statistical methods exist for assessing the importance of subgroups.

Comparing rates of dichotomous outcomes

Studies are usually statistically combined by a method such as the DerSimonian and Laird. The DerSimonian and Laird weight for pooling studies is a type of inverse variance weight and creates a random effect model. Determining prediction intervals for random effects may help apply results to clinical practice.

Statistical packages are available from the Cochrane Collaboration (http://www.cc-ims.net/revman) and for R (programming language) (rmeta and HSAUR2).

Studies with groups having zero events

Excluding studies with zero events total events (zero-total-event trials) or zero events in one treatment group (zero-event trials) may exaggerate effect sizes. An alternative is to use a continuity correction. Rather than using a constant continuity correction, less bias may occur by correcting with either

• "empirical estimate of the pooled effect size from the remaining studies in the meta-analysis."
• "a function of the reciprocal of the opposite group arm size"

For an example of continuity correction using the second method above:

• S is the sum of corrections for event and no event cells (usually S=1 in a zero-event trial and S=2 in a zero-total-event trial)
• R is the ratio of group sizes (R=1 if both groups are the same)
• For a zero-event trial with equal group sizes
  • The correction in the larger experimental group is R/S*(R + 1). This becomes 1/1*(1 + 1) = 1
  • The correction in the smaller experimental group is 1/S*(R + 1). This becomes 1/1*(1 + 1) = 1
• For a zero-event-total trial with equal group sizes
  • The correction in the larger experimental group is R/S*(R + 1). This becomes 1/2*(1 + 1) = 0.5
  • The correction in the smaller experimental group is 1/S*(R + 1). This becomes 1/2*(1 + 1) = 0.5

Comparing rates of continuous outcomes

The standardized mean difference (SMD), also called the effect size (ES), is used. The SMD is:

${\text{Standardized mean difference}}=\left({\frac {\text{Difference in mean outcome between groups}}{\text{Standard deviation of outcome among participants}}}\right)$

In the interpretation of SMD:

• 0.2 represents a small effect
• 0.5 a moderate effect
• 0.8 a large effect

Transforming standardized mean difference to odds ratio

${\text{Odds ratio}}={\text{exp}}^{\left({\text{2 * 1.96}}\right)}$

Subgroup analysis

There are two types of interactions:

• Qualitative interaction interaction exists if the direction of effect is reversed in subgroups.
• Quantitative interaction is when the size of the effect varies but not the direction.

If the subgrouping accounts for all heterogeneity, interaction can be sought using an inverse-variance method for a fixed-effect model.

If the subgrouping does not account for all heterogeneity, interaction can be tested with meta-regression to avoid false-positive results. Metagression is detailed in a section below.

Software

Software available for meta-analysis includes:

• Review Manager (RevMan) by the Cochrane Collaboration
• Meta-analyst
• R programming language:
  • metaLik package with reference manual. Allows for the profile likelihood for computing confidence bounds.
  • rmeta package with reference manual
  • metafor package with manual
  • HSAUR2 interactive package with a chapter containing sample demonstrations, "Meta-Analysis: Nicotine Gum and Smoking Cessation and the Effcacy of BCG Vaccine in the Treatment of Tuberculosis" in "A Handbook of Statistical Analyses Using R".
  • For meta-analysis of diagnostic tests
    • bamdit. "Bayesian meta-analysis of diagnostic test data based on a scale mixtures bivariate random-effects model. Summary statistics are based on pooled and predictive sensitivities and specificities."
    • HSROC. " implements a model for joint meta-analysis of sensitivity and specificity of the diagnostic test under evaluation, while taking into account the possibly imperfect sensitivity and specificity of the reference test. This hierarchical model accounts for both within and between study variability. Estimation is carried out using a Bayesian approach, implemented via a Gibbs sampler."

Displaying results

Study results may be grouped and displayed with a Forest plot.

(CC) Photo: Robert Badgett
Forest Plot showing meta-analysis of randomized controlled trials of differing target glucose control and mortality for diabetes mellitus type 2. Note the heterogeneity (P<0.05 and high I² in circled in red) due to increased death when the glycosylated hemoglobin A (Hb A1c) target was 6.0% in the ACCORD trial

Variations on meta-analysis

Cumulative meta-analysis

Cumulative meta-analysis has been used to show that 25 off 33 randomized controlled trials of streptokinase not necessary and have shown the delay in adoption of evidence by experts.

Cumulative meta-analyses may be prone to false positive results due to repeated tests of statistical significance. This may be avoided by use of trial sequential analysis.

Individual patient data meta-analysis

An individual patient data meta-analysis is "where analyses are done using original data and outcomes for each person enrolled in relevant studies; these results are then pooled in one analysis as if patients were in a single large study."

Standards exist for conduct and reporting.

Individual patient data meta-analysis (IPD meta-analysis) may have more long lasting results than other meta-analyses.

Living or continuous meta-analyses

Because of the above problems the need for living meta-analyses has been made. Websites to support collaborative meta-analyses is available.

• Systematic Review Data Repository of the United States of America Agency for Healthcare Research and Quality
• OpenMetaAnalysis at OpenCPU

Meta-regression

Meta-regression allows simultaneous comparison of multiple sources of heterogeneity.

Meta-regression can examine relationships between predictor and outcome variables including non-linear relationships.

Meta-regression can analyze subgroups.A permutation test may reduce the chance of a false positive subgroup analysis.

When analyzing a meta-regression of dichotomous independent variables, the "results of meta-regression analyses are most usefully expressed as ratios of odds ratios (or risk ratios)."

Meta-regression can be performed with the rmeta package of the R programming language as described by Everitt and Hothorn.

Meta-regression is not as powerful as individual patient data meta-analysis, especially when the distributions of covariates are heterogeneous across studies.

Examples of meta-regression analysis are:

1. McAlister FA, Wiebe N, Ezekowitz JA, Leung AA, Armstrong PW (2009). "Meta-analysis: beta-blocker dose, heart rate reduction, and death in patients with heart failure.". Ann Intern Med 150 (11): 784-94. PMID 19487713.

1. Briel M, Ferreira-Gonzalez I, You JJ, Karanicolas PJ, Akl EA, Wu P et al. (2009). "Association between change in high density lipoprotein cholesterol and cardiovascular disease morbidity and mortality: systematic review and meta-regression analysis.". BMJ 338: b92. DOI:10.1136/bmj.b92. PMID 19221140. PMC PMC2645847. Research Blogging.

1. Emerging Risk Factors Collaboration. Erqou S, Kaptoge S, Perry PL, Di Angelantonio E, Thompson A et al. (2009). "Lipoprotein(a) concentration and the risk of coronary heart disease, stroke, and nonvascular mortality.". JAMA 302 (4): 412-23. DOI:10.1001/jama.2009.1063. PMID 19622820. Research Blogging.

Network meta-analysis

A network meta-analysis and Bayesian hierarchical models pool studies in order to compare to treatments that have not been directly compared. Network meta-analyses are commonly not well performed. Network meta-analyses of both randomized controlled trials and diagnostic test assessments can have misleading results. Network meta-analyses have been conducted by the Cochrane Collaboration.

Network meta-analyses can be conducted with Bugs and OpenBugs software.

Meta-analysis of diagnostic tests

Standards exists for the meta-analysis of diagnostic tests. The traditional summary receiver operating characteristic curve (SROC curve) should be replaced by either the hierarchical summary receiver operating characteristic curve(HSROC curve). or bivariate random-effects model. Discussions of HSROC and bivariate random-effects meta-analysis are available. An example of a meta-analysis using bivariate mixed-effects binomial regression model is available. Examples of using the HSROC and diagnostic odds ratio are available.

Update of existing a systematic review

Reviewers may elect to conserve resources by updating an existing review.

Overview of reviews

A systematic review may review other systematic reviews. The reviews may address different treatments of the same disease or different diseases that can be treated with an intervention.

Assessing the quality of meta-analysis

The quality of a meta-analysis can be assess with:

• Oxman and Guyatt instrument
• Assessment of Multiple Systematic Reviews (AMSTAR) tool

Factors associated with higher quality meta-analyses

Meta-analyses by the Cochrane Collaboration tend to be of higher quality.

Individual data meta-analyses, in which the records from individual patients are pooled together into one dataset, tend to have more stable conclusions.

Factors associated with lower quality meta-analyses

About a third of meta-analyses that happen to precede large randomized controlled trials will conflict with the results of the trial.

Conflict of interest

Meta-analyses produced with a conflict of interest are more likely to interpret results as positive.

Small study effect and publication bias

The small study effect is the observation that small studies tend to report more positive results. This is especially a threat when the original studies in a meta-analysis are less than 50 patients in size.

Publication bias against negative studies is part of the small study effect and may threaten the validity of meta-analyses that are positive and all the studies included within the meta-analysis are small.

In performing a meta-analysis, a file draweror a funnel plot analysis may help detect underlying publication bias among the studies in the meta-analysis.

Outcome reporting bias

Meta-analyses in which a smaller proportion of included trials provide raw data for inclusion in the meta-analysis are more likely to be positive. This may be due a bias against reporting negative results.

Problems with meta-analyses

Disagreement with major clinical trials

Meta-analyses may not agree with major clinical trials. Some of the disagreement may be due to the methods used in selecting and comparing meta-analyses and trials. Publication bias may be a factor.

The disagreements lead to debate as to whether truth is the meta-analysis or a dominant, large trial.

Conflict of interest

Meta-analyses may not consider or report conflict of interests among the studies included in the analysis.

Obsolescent meta-analyses

The conclusions of meta-analyses may be mitigated by research published after the search date of the meta-analysis. This may occur by the time the meta-analysis has been published. Strategies have been developed for identifying potentially outdated analyses and their updating.

Small meta-analyses more be prone to obsolescence and disagreement with larger, subsequent trials.

Publication bias

Prospero is prospective registry of systematic reviews to reduce publication bias.

Redundant meta-analyses

Overlapping and redundant meta-analyses have become a problem.