Chapter 31. Analysis of Moderator Effects in Meta-Analysis

Meta-analysis is a quantitative methodology for leveraging the proliferation of published research to more scientifically and comprehensively synthesize bodies of research ( e.g., Chalmers, Hedges, & Cooper, 2002). Social policy decisions and best practices various fields are increasingly influenced not by the results from single isolated studies, but by the findings from meta-analyses (Cook, et al., 1992).
A meta-analysis not only helps to determine whether a particular treatment is actually effective or whether there is indeed an association between variables, but also allows the reviewer to examine whether the treatment effectiveness or relationship strength is influenced by the characteristics of the studies. For example, it is conceivable that the effectiveness of a treatment observed in a particular study depends on the treatment duration or intensity (e.g., the length of the psychotherapy or the medication dosage), the characteristics of the sample, the study setting, or the type of outcome measure used. Examining these hypotheses is difficult when conducting a traditional narrative literature review, but such moderator analyses constitute an integral and important aspects of a meta-analysis (Lau, Ioannidis, & Schmid, 1998; Thompson, 1994).

Scope of this Chapter
One can roughly break the process of a meta-analysis down into five stages (Cooper, 1998): (a) problem formulation, (b) data collection, (c) data evaluation, (d) analysis and interpretation, and (e) presentation of results. The majority of the time and effort will typically be spent on the first three stages, which are briefly outlined in Chapter XX and elsewhere. For the purposes this chapter, we will assume that these steps have already been completed. Instead, the present chapter is meant to provide some guidelines on how to conduct the statistical analysis, once the first three steps have been completed. Again, several books deal extensively with this topic (e.g., Hedges & Olkin, 1985; Cooper & Hedges, 1994) and a single chapter cannot replace these references. However, the statistical methods that should be used for a meta-analysis are constantly being improved and extended. The goal is then to highlight those methods that currently represent best practices.

Example data set (click here to download excel file with this data)

Consider Table 1, which provides the results from k = 16 studies examining the effectiveness of massage therapy for reducing state anxiety. In each study, the amount of anxiety was measured among subjects randomly assigned to either a massage therapy or a control/standard treatment group. For each study (i being the index for the studies), the table lists the sample size of the control/comparison and the treatment group ( and , respectively), the effect size estimate (Yi) in the form of a standardized mean difference (to be discussed in more detail below), the estimated sampling variance () of the effect size estimate, the minutes per session of massage therapy provided, whether a fully trained massage therapist or layperson provided the therapy (coded as 1 and 0, respectively), the mean age of the sample, and whether the study was conducted by the Touch Research Institute (TRI) or not (coded as 1 and 0, respectively). The last four variables are examples of moderator variables that may influence the effectiveness of massage therapy for reducing state anxiety.

Table 1: Results from 16 Studies on the Effectiveness of Massage Therapy for Reducing State Anxiety

Note. Adopted from Moyer, Rounds, and Hannum (2004), leaving out three studies with missing data and two studies where the duration of massage therapy provided was less than 10 minutes. Minutes per Session = minutes of therapy provided per session; Trained Therapist = 0 for a layperson providing the therapy and 1 for a trained therapist; Mean Age = mean age of the sample; TRI Study = 1 when study was conducted by the Touch Research Institute (TRI) and 0 otherwise.