Chapter 12

Researchers in the social sciences often already have access to completed studies in the literature that relate to, or address their hypotheses.  How best, then, to organize and summarize findings from these studies in order to identify and exploit what is known, and focus research on promising areas? While narrative summaries and analyses of the literature are important (and the norm), quantitative research synthesis or meta-analysis is currently considered a best practice across many disciplines (see Cooper, & Hedges, 1994; Hedges & Olkin, 1985).

Meta-analysis refers to quantitative methods of synthesizing empirical research evidence from a sample of studies that examine a certain topic and test comparable hypotheses (Hedges & Olkin, 1985). The first step in meta-analysis involves describing the results of each study via numerical indicators (e.g., estimates of effect sizes such as a standardized mean difference, a correlation coefficient, or an odds ratio). These effect size estimates reflect the magnitude of the association of interest in each study. The second step involves combining the effect size estimates from each study to produce a single indicator that summarizes the relationship of interest across the sample of studies. Hence, meta-analytic procedures produce summary statistics, which are then tested to determine their statistical significance and importance.

The specific analytic techniques involved will depend on the question the meta-analytic summary is intended to address.  Sometimes the question of interest concerns the typical or average study result, such as the effect of some treatment or intervention, where the average effect of the treatment is often of interest (see, e.g., Smith & Glass, 1977).  In other cases the degree of variation in results across studies will be of primary interest, where meta-analysis can be utilized to study the generalizability of employment test validities across situations (see, e.g., Schmidt & Hunter, 1977).  Meta-analysis is also frequently used to identify the contexts in which a treatment or intervention is most successful or has the largest effect (see, e.g., Cooper, 1989).

Meta-analytic reviews are designed to integrate empirical research with the objective to create research generalizations, and hence, one substantial advantage of meta-analysis is the generality of the summary estimates (Cooper & Hedges, 1994). This constitutes a unique aspect of meta-analysis that is crucial for the external validation of the estimates (see Shadish, Cook, & Campbell, 2002). Generally, the estimates that are produced from meta-analyses have higher external validity than estimates reported in single studies. Other advantages of meta-analytic reviews include that the summary estimates that are generated from such reviews can support or refute theories (and hence facilitate the improvement of substantive theory), and can guide future research by identifying important issues (Copper, 1989). In addition, from a statistical point of view the results of meta-analytic procedures have higher statistical power than indicators obtained from individual studies, which increases the probability of detecting associations of interest (Cohn & Becker, 2003).