What technique is used to combine the statistics from studies included in a systematic review?

Meta-Analyses and Systematic Reviews

When multiple RCTs have been conducted on a given topic, a meta-analysis can be employed. In a meta-analysis, statistical techniques are used to measure and compare the outcomes of multiple studies. By combining the outcomes of multiple studies, the overall sample size and statistical power are increased.17 Meta-analyses are particularly useful when there are small clinical trials that are inconclusive, perhaps because of size, and when different RCTs come to different conclusions. The averages of the study results are often weighted, placing more emphasis on larger studies. Forest plots are a convenient way of summarizing the statistical analyses performed on the studies used for the meta-analyses (Fig. 10.34.1).18 Meta-analyses can be useful for reducing the biases and improving the precision that is associated with RCTs.8

An example of a meta-analysis in glaucoma therapy is the study by Maier et al.,13 which addressed whether lowering of IOP delays the progression of visual field defects in patients with open-angle glaucoma. This study is of particular note because it was published in theBritish Medical Journal, not in an ophthalmology journal. It gives us insight into the limited conclusions that nonophthalmologists are willing to draw from our ophthalmological data. Maier et al. were willing to only conclude that lowering IOP in patients with ocular hypertension or manifest glaucoma lowers the risk of visual field loss and has unclear effects in patients with normal-tension glaucoma, whereas the glaucoma community has drawn more widespread conclusions from these data regarding IOP lowering in glaucoma.

Another meta-analysis compared the effectiveness and adverse effects of adding a second IOP-lowering agent to an eye already being treated with a topical prostaglandin.19 The authors of that meta-analysis appropriately followed established guidelines (Quality of Reporting of Meta-analyses [QUORUM]) for the performance of a meta-analysis, used predetermined inclusion and exclusion criteria for selecting trials, and reported on how well the included studies met the quality criterion. They found that the effectiveness of the three classes of medications was similar but that they each had different side-effect profiles. Li et al.20 performed a systematic review and network meta-analysis of 114 RCTs comparing the effectiveness of first-line IOP-lowering eyedrops in patients with ocular hypertension or primary open-angle glaucoma (POAG). Those authors found that prostaglandin medications were the most efficacious class, with within-class differences being small and probably not clinically meaningful.20 Mean reductions in IOP (mm Hg) at 3 months compared with that in placebo group were as follows: bimatoprost (5.61), latanoprost (4.85), travoprost (4.83), levobunolol (4.51), tafluprost (4.37), timolol (3.70), brimonidine (3.59), carteolol (3.44), levobetaxolol (2.56), apraclonidine (2.52), dorzolamide (2.49), brinzolamide (2.42), betaxolol (2.24), and unoprostone (1.91).20

Calculating an Average Effect Size

The statistical analyses in a meta-analysis are guided by a statistical model that must be previously assumed. The main task of the statistical model is to establish the properties of the effect-size population from which the individual effect-size estimates have been selected. To accomplish the first purpose in a meta-analysis, that is, to calculate an average effect size, two statistical models can be assumed: the fixed- and the random-effects models.

Suppose there are k independent empirical studies about a given topic and Ti is the effect-size estimate obtained in the ith study (here Ti refers to any of the different effect-size indices presented above, both from the d and the r families.) In a fixed-effects model, it is assumed that all of the effect-size estimates come from a population with a common parametric effect size, θ, and as a consequence the only error source is that produced by sampling error, ei. Thus, the model can be formulated as Ti = θ + ei, the sampling errors, ei, being normally distributed with mean 0 and sampling variance σi2, ei ∼ N(0,σi2). Therefore, the effect-size estimates, Ti, are also normally distributed with mean θ and sampling variance σi2, Ti ∼ N(θ,σi2).

In a random-effects model, it is assumed that the effect-size estimates, Ti, estimate different population effect sizes, θi, that is, Ti = θi + ei, and θi pertains to a distribution of parametric effect sizes with mean μ and variance τ2, usually called between-studies variance. The parametric effect sizes can be modeled as θi = μ + ϵi, ϵi being the errors of the parameters around its mean, μ. Therefore, the random-effects model is formulated as Ti = μ +εi + ei. Assuming normality, Ti has as mean μ and variance τ2+σi2,Ti∼N(μ,τ2+σi2). Thus, the fixed-effects model can be considered a particular case of the random-effects model when the between-studies variance is zero (τ2 = 0) and, as a consequence, all the parametric effect sizes are equal (θ1 = θ2 =… = θi… = θ = μ).

To calculate an average effect size from a set of studies, each effect-size estimate must be weighted by its precision. Both in a fixed- and a random-effects model, the uniformly minimum variance unbiased estimator (UMVUE) of the average effect size, μ, is that obtained by weighting each effect-size estimate by its inverse variance:

[8]TUMVUE=∑i=1kwiTi∑i=1kwi

where wi is the optimal weight for the ith study and, depending on the statistical model assumed, it is defined as wi FE=1/σi2 or as wiRE=1/(τ2+σi2), for the fixed- and the random-effects models, respectively.

In practice, the optimal weights cannot be used, because the within-study sampling variances, σi2 , and the between-studies variance, τ2, are unknown. For each effect-size index, formulas have been devised to estimate σi2 and τ2. Thus, the estimated weights are defined as wˆi FE=1/σˆi2 and wˆiRE=1/(τˆ2+σˆi2) for fixed- and random-effects models, respectively. (Another option consists of simply weighting each effect-size estimate by its sample size (Hunter and Schmidt, 2004).) Here σˆi2 is the estimated within-study sampling variance for the ith study (e.g., eqn [4] for the d index), and τˆ2 can be calculated by applying one of the different estimators of the between-studies variance proposed in the meta-analytic literature. The τ2 estimator most usually applied in meta-analysis is that based on the moments method:

[9]τˆ2=Q−(k−1)c

where k is the number of studies, Q is a heterogeneity statistic defined in eqn [17], and c is obtained by:

[10]c=∑i=1kwˆiFE −∑i=1k(wˆiFE)2∑i=1kwˆiFE

When Q < (k − 1), then τˆ2 is negative and must be truncated to zero. Other τ2 estimators can be consulted in Viechtbauer (2005).

With the respective estimated variances, the population effect size, μ, is then estimated by:

[11]TFE=∑i= 1kwˆiFETi∑i =1kwˆiFE

[12]TRE=∑i=1kwˆiRETi ∑i=1kwˆiRE

for fixed- and random-effects models, respectively. When a fixed-effects model is assumed, TFE is approximately normally distributed and its sampling variance defined as:

[13]V(TFE)=1/ ∑i=lkwˆiFE

Thus, a confidence interval for the average effect size can be obtained by (cf. e.g., Cooper et al. 2009):

[14]TFE±zα/2V(T FE)

where zα/2 is the 100(α/2) percentile of the standard normal distribution and α is a significance level.

Under a random-effects model, a better approach for obtaining a confidence interval for the overall effect size consists of assuming a Student t reference distribution with k − 1 degrees of freedom, instead of the standard normal distribution:

[15]TRE±tk−1,α/2V(TRE)

where tk-1,α/2 is the 100(α/2) percentile of the Student t distribution with k − 1 degrees of freedom, and V(TRE) is an estimate of the sampling variance for TRE, which is obtained by (cf. Sánchez-Meca and Marín-Martínez, 2008)

[16]V(TRE)=∑i=1kwˆiRE(Ti−TRE)2(k−1)∑i=1 kwˆiRE

Alternative approaches to those presented in eqns [14] and [15] have been proposed (cf. e.g., Sánchez-Meca and Marín-Martínez, 2008).

Systematic Review and Meta-Analysis

Systematic review andmeta-analysis are two related but different terms, and they are often confused. A systematic review is a scientific investigation that focuses on a specific question and uses explicit, planned methods to identify, select, assess, and summarize the findings of similar but separate studies. It may or may not include a quantitative synthesis of the results from separate studies.16 A meta-analysis is the process of using statistical methods to quantitatively combine the results of similar studies identified in a systematic review in an attempt to allow inferences to be made from the sample of studies. In a meta-analysis, the results of a number of randomized clinical trials or observational studies may be statistically combined to obtain a summary estimate for the effect of a given treatment.17 Systematic reviews and meta-analyses should be differentiated from other, less data-driven review articles in which authors present their own interpretation of data. The strength of a meta-analysis comes from its being an analysis of combined results from multiple studies, thereby increasing power to detect differences. This is an especially important methodology in obstetrics, as here there are few large randomized clinical trials to guide treatments.

Numerous meta-analyses have been performed for topics in obstetrics,18,19 and many appear in theCochrane Database of Systematic Reviews.20 Two such analyses (Figs. 17.2 and17.3) are taken from the Cochrane Library meta-analysis of the effect on neonatal outcome of antibiotics given antenatally to women with preterm prematurely ruptured amniotic membranes.21Fig. 17.2 shows a comparison of neonatal infectious complications between women who received antibiotics and women who did not, and data are pooled for all available studies. Each of 11 randomized trials that met inclusion criteria for this analysis is listed, with the number of subjects and the frequency of the outcome in the treatment and control groups noted. The relative risk and 95% confidence interval (seeAssessing Random Error, later) for each study, weighted for their sample size, are shown. The total number of subjects with the outcome of interest is summed, and the combined relative risk and 95% confidence interval are calculated. In this example, a number of small trials show a nonsignificant trend in favor of antibiotic treatment. The pooled (i.e., statistically combined) relative risk was 0.67, with a 95% confidence interval of 0.52 to 0.85. The point estimate (i.e., the relative risk) suggests that the “best guess” is that antibiotics reduce the risk for neonatal infection by 33%. The confidence interval suggests that data are consistent with as much as a 48% reduction in risk (1 − 0.52) or as little as a 15% (1 − 0.85) reduction in risk. Even the upper bound of the confidence interval suggests a protective effect of antibiotics on neonatal infection.

Meta-Analysis

•

Meta-analysis is a statistical method to combine results of different studies, especially those with small sample size or with conflicting results.

Meta-analysis is often an important component of systematic reviews.

Literature search is the first step, and is very important for meta-analysis, as incomplete literature search may bring incorrect results.

Meta-analysis is done by identifying a common statistical measure that is shared among studies, and calculating a weighted average of that common measure.

The statistics of meta-analysis could be conducted with software such as Stata or Review manager (RevMan).

6.4 Meta-analysis for GWAS

Meta-analysis combines information from multiple GWAS and can increase the chances of finding true positives among the identified associations (Cantor et al., 2010). Hundreds of studies involving GWAS meta-analysis have been published for humans (Evangelou & Ioannidis, 2013), but there seems to be no published report of meta-analysis for GWAS in plants. Therefore, it will be desirable to conduct meta-analysis using results of several GWAS involving the same trait in the same crop. While doing so, one should recognize that several factors may influence the results of GWAS meta-analysis. First, different studies may be based on heterogeneity in data, due to genetic and environmental factors, making interpretation of the results of meta-analysis difficult, although methods have been suggested to deal with this problem (Han & Eskin, 2012). Second, sample size and design may be different in different studies included in meta-analysis (Moonesinghe, Khoury, Liu, & Ioannidis, 2008; Spencer, Su, Donnelly, & Marchini, 2009). Third, some studies included in a meta-analysis may be based on imputed data, which should be taken into consideration (de Bakker et al., 2008). Fourth, meta-analysis for a complex trait involving rare variants may create some problems (Evangelou & Ioannidis, 2013). Some of these issues have also been addressed by Thompson, Attia, and Minelli (2011).

Abstract

Meta-analysis has become a popular approach for summarizing a large number of clinical trials and resolving discrepancies raised by these trials. In this chapter, we introduce the general procedures for meta-analysis: formulating the question, defining eligibility, identifying studies, abstracting data, statistical analysis, and reporting the results. One key issue determining whether studies can be combined is the extent of heterogeneity among individual studies. We review graphical and statistical tools for assessing heterogeneity, describing the fixed-effect and random-effect models commonly used in meta-analysis, and providing some general recommendations regarding when fixed-effect or random-effect approach is appropriate. Publication bias is an inherent issue with meta-analysis, since studies (especially smaller ones) with “negative” results are frequently unpublished. Funnel plot, Begg and Mazumdar's rank correlation, and Egger regression are useful tools for assessing publication bias. As an illustration of the concepts discussed, we apply meta-analysis techniques to studies examining the use of antiinflammatory therapies in sepsis.

SECONDARY STUDY DESIGN

Meta-analysis is an analytical tool that permits the evaluation of a diagnostic or therapeutic modality through the appropriate use of previously published smaller studies.21 Meta-analysis is not the simple pooling of data reported in numerous small studies, a notion that has often caused investigational errors. The simple pooling of data from multiple small studies often compounds biases and may further reduce the ability to detect important differences among study groups.22 A meta-analysis of available data is a rigorous, systematic, and quantitative review. To undertake a meta-analysis to address a research question or hypothesis, the investigator must first define the appropriate study population and methodology. The literature is then reviewed exhaustively for the identification of all relevant studies that meet the established criteria. These studies are then evaluated critically to determine and correct for possible bias in data collection. Reports in which data collection did not adhere to the established meta-analysis criteria, or in which an unquantified bias was introduced, are not included. The raw data from the selected studies are then combined and analyzed. This methodology permits the detection of statistically significant differences among study groups that may not have been possible in individual reports due to their small size.

Introduction

Meta-analysis is a set of techniques used “to combine the results of a number of different reports into one report to create a single, more precise estimate of an effect” (Ferrer, 1998). The aims of meta-analysis are “to increase statistical power; to deal with controversy when individual studies disagree; to improve estimates of size of effect, and to answer new questions not previously posed in component studies” (Hunter and Schmidt, 1990). All definitions stress that there must be a valid reason to combine the studies. Egger et al. (2002) wrote “Indeed, it is our impression that reviewers often find it hard to resist the temptation of combining studies when such meta-analysis is questionable or clearly inappropriate.” Although the frequency at which meta-analysis is used is increasing (Egger and Smith, 1997), meta-analysis has its detractors. In reality, if carefully performed, it yields useful information, but a meta-analysis of badly designed studies produces erroneous statistics and may be misleading. Ignoring heterogeneity and combining apples and oranges is a pervasive error in meta-analysis (Eysenck, 1995) and techniques exist to assess it (Ferrer, 1998; Tang and Liu, 2000). Other forms of bias also interfere with effective meta-analysis (Egger and Smith, 1998).

There are several advantages to meta-analysis. It allows investigators to pool data from many trials that are too small by themselves to allow for secure conclusions. Although ideally any clinical trial should plan an adequate sample size, historically most trials have been underpowered. In 2002, a study of 5503 clinical trials (McDonald et al., 2002) identified 69% as having fewer than 100 subjects. Small trials make it more difficult to reject the null hypothesis because they lead to larger standard deviations and standard errors. There is also a risk of bias. A small trial that does not show a significant effect might not be submitted for publication, whereas the same sized trial that reached significance (whether warranted or not) will probably be published (Stern and Simes, 1997). Egger et al. (2002) concluded that on average unpublished trials underestimate treatment effects by 10%. Furthermore, Stanbrook et al. (2006) found that clinical trials named with an acronym were more likely to be published in a major journal or to be cited than trials not named, independent of whether the results were positive or negative.

Meta-Analysis

Meta-analysis combines findings from many different – yet related – studies to foster empirical knowledge about causal associations that are more trustworthy than those possible from any single study. This benefit arises for two main reasons. First, combining findings from parallel studies promises to increase statistical power and precision for estimating the magnitude of a causal association. More importantly, however, is the potential of meta-analysis to strengthen external validity by identifying the realm of application of a causal association – that is, meta-analyses are most useful when they allow us to examine whether a causal association (1) holds with specific populations of persons, settings, times, and ways of varying the cause or measuring the effect; (2) holds across different populations of people, settings, times, and ways of operationalizing a cause and effect; and (3) can even be extrapolated to other populations of people, settings, times, causes, and effects than those that have been studied to date – that is, meta-analyses offer opportunities to probe external validity questions 1, 2, and 3.

In 1978, Glass and Smith published a noteworthy meta-analysis examining the relationship between class-size and academic achievement – an issue that was previously surrounded by inconclusive research findings. This meta-analysis synthesized the results of 77 separate studies that included 725 comparisons of academic achievement in smaller versus larger classes. It received considerable attention because it was the first meta-analysis to find clear evidence that reduced student-to-instructor ratios significantly improved academic achievement (Cooper, 1989). Examples from this meta-analysis will be presented to illustrate how Cook’s five principles justify generalization in the absence of formal sampling.

3 Replication of GWAS findings and meta-analysis

Regardless of the method applied to the nominal P-values in order to correct for multiple testing, statistically significant SNPs should be replicated and therefore validated in an independent cohort. Replicating findings provides credibility to the initial discovery efforts and is an essential step in establishing genotype-phenotype associations by evaluating whether or not the discovered SNPs in the initial GWAS are false positives. Samples from the replication cohort should be collected in the same manner as in the original (discovery) cohort and the statistical analysis needs to be based on the same genetic model employed in the discovery cohort to ensure consistency and robustness of association. Just as important, the effect sizes of the SNPs should be in accordance between the discovery and replication cohorts. For example, if the minor allele of a SNP in the discovery cohort is associated with increased drug requirement then it should also be associated with increased drug response in the replication cohort. But failure to replicate the original findings does not necessarily mean that the association is a false positive finding. Factors that may influence failures of replication include hidden population substructure, very small marker effect size that cannot be replicated easily, small sample size for the replication cohort which can lead to low power to confirm the initial association. Nonetheless, replicating significant associations in GWAS is considered the gold standard in establishing genotype-phenotype associations.