A Welch-type test for homogeneity of contrasts under heteroscedasticity with application to meta-analysis

A common problem that arises in the meta-analysis of several studies, each with independent treatment and control groups, is to test for the homogeneity of effect sizes without the assumptions of equal variances of the treatment and the control groups and of equal variances among the separate studies. A commonly used test statistic, frequently denoted as Q, is the weighted sum of squares of the differences of the individual effect sizes from the mean effect size, with weights inversely proportional to the variances of the effect sizes. The primary contributions of this article are the presentation of improved and very accurate approximations to the distributions of the Q statistic when the effect size is a linear contrast such as the difference between the treatment and control means. Our improved approximation to the distribution of Q under the null hypothesis is based on a multiple of an F-distribution; its use yields a substantial reduction in the type I error rate of the homogeneity test. Our improved approximation to the distribution of Q under an alternative hypothesis is based on a shift of a chi-square distribution; its use allows for much greater accuracy in the computation of the power of the homogeneity test. These two improved approximate distributions are developed using the Welch methodology of approximating the moments of Q by the use of multivariate Taylor expansions. The quality of these approximations is studied by simulation. A secondary contribution of this article is a study of how best to combine the variances of the treatment and control groups (needed for the calculation of weights in the Q statistic). Our conclusion, based on simulations, is that use of pooled variances can result in substantially erroneous conclusions.