Gang Chen
2010-Feb-05 22:07 UTC
[R] metafor package: effect sizes are not fully independent
In a classical meta analysis model y_i = X_i * beta_i + e_i, data
{y_i} are assumed to be independent effect sizes. However, I'm
encountering the following two scenarios:
(1) Each source has multiple effect sizes, thus {y_i} are not fully
independent with each other.
(2) Each source has multiple effect sizes, each of the effect size
from a source can be categorized as one of a factor levels (e.g.,
happy, sad, and neutral). Maybe better denote the data as y_ij, effect
size at the j-th level from the i-th source. I can code the levels
with dummy variables into the X_i matrix, but apparently the data from
the same source are correlated with each other. In this case, I would
like to run a few tests one of which is, for example, whether there is
any difference across all the levels of the factor.
Can metafor handle these two cases?
Thanks,
Gang
Mike Cheung
2010-Feb-07 15:41 UTC
[R] metafor package: effect sizes are not fully independent
Dear Gang, Here are just some general thoughts. Wolfgang Viechtbauer will be a better position to answer questions related to metafor. For multivariate effect sizes, we first have to estimate the asymptotic sampling covariance matrix among the effect sizes. Formulas for some common effect sizes are provided by Gleser and Olkin (2009). If a fixed-effects model is required, it is quite easy to write your own GLS function to conduct the multivariate meta-analysis (see e.g., Becker, 1992). If a random-effects model is required, it is more challenging in R. SAS Proc MIXED can do the work (e.g., van Houwelingen, Arends, & Stijnen, 2002). Sometimes, it is possible to transform the multivariate effect sizes into independent effect sizes (Kalaian & Raudenbush, 1996; Raudenbush, Becker, & Kalaian, 1988). Then univariate meta-analysis, e.g., metafor(), can be performed on the transformed effect sizes. This approach works if it makes sense to pool the multivariate effect sizes as in your case (2)- the effect sizes are the same but in different conditions (happy, sad, and neutral). However, this approach does not work if the multivariate effect sizes are measuring different concepts, e.g., verbal achievement and mathematical achievement. Hope this helps. Becker, B. J. (1992). Using results from replicated studies to estimate linear models. Journal of Educational Statistics, 17, 341-362. Gleser, L. J., & Olkin, I. (2009). Stochastically dependent effect sizes. In H. Cooper, L. V. Hedges, and J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis, 2nd edition (pp. 357-376). New York: Russell Sage Foundation. Kalaian, H. A., & Raudenbush, S. W. (1996). A multivariate mixed linear model for meta-analysis. Psychological Methods, 1, 227-235. Raudenbush, S. W., Becker, B. J., & Kalaian, H. (1988). Modeling multivariate effect sizes. Psychological Bulletin, 103, 111-120. van Houwelingen, H.C., Arends, L.R., & Stijnen, T. (2002). Advanced methods in meta-analysis: multivariate approach and meta-regression. Statistics in Medicine, 21, 589-624. Regards, Mike -- --------------------------------------------------------------------- Mike W.L. Cheung Phone: (65) 6516-3702 Department of Psychology Fax: (65) 6773-1843 National University of Singapore http://courses.nus.edu.sg/course/psycwlm/internet/ --------------------------------------------------------------------- On Sat, Feb 6, 2010 at 6:07 AM, Gang Chen <gangchen6 at gmail.com> wrote:> In a classical meta analysis model y_i = X_i * beta_i + e_i, data > {y_i} are assumed to be independent effect sizes. However, I'm > encountering the following two scenarios: > > (1) Each source has multiple effect sizes, thus {y_i} are not fully > independent with each other. > (2) Each source has multiple effect sizes, each of the effect size > from a source can be categorized as one of a factor levels (e.g., > happy, sad, and neutral). Maybe better denote the data as y_ij, effect > size at the j-th level from the i-th source. I can code the levels > with dummy variables into the X_i matrix, but apparently the data from > the same source are correlated with each other. In this case, I would > like to run a few tests one of which is, for example, whether there is > any difference across all the levels of the factor. > > Can metafor handle these two cases? > > Thanks, > Gang > > ______________________________________________ > R-help at r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. >-- --------------------------------------------------------------------- Mike W.L. Cheung Phone: (65) 6516-3702 Department of Psychology Fax: (65) 6773-1843 National University of Singapore http://courses.nus.edu.sg/course/psycwlm/internet/
Mike Cheung
2010-Feb-08 01:59 UTC
[R] metafor package: effect sizes are not fully independent
Dear Gang, It seems that it is possible to use a univariate meta-analysis to handle your multivariate effect sizes. If you want to calculate a weighted average first, Hedges and Olkin (1985) has discussed this approach. Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, FL: Academic Press. Regards, Mike -- --------------------------------------------------------------------- Mike W.L. Cheung Phone: (65) 6516-3702 Department of Psychology Fax: (65) 6773-1843 National University of Singapore http://courses.nus.edu.sg/course/psycwlm/internet/ --------------------------------------------------------------------- On Mon, Feb 8, 2010 at 6:48 AM, Gang Chen <gangchen6 at gmail.com> wrote:> Dear Mike, > > Thanks a lot for the kind help! > > Actually a few months ago I happened to read a couple of your posts on > the R-help archive when I was exploring the possibility of using lme() > in R for meta analysis. > > First of all, I didn't specify the meta analysis model for my cases > correctly in my previous message. Currently I'm only interested in > random- or mixed-effects meta analysis. So what you've suggested is > directly relevant to what I've been looking for, especially for case > (2). I'll try to gather those references you listed, and figure out > the details. > > Also I think I didn't state my case (1) clearly in my previous post. > In that case, all the effect sizes are the same and in the same > condition too (e.g., happy), but each source has multiple samples of > the measurement (and also measurement error, or standard error). Could > this still be handled as a multivariate meta analysis since the > samples for the the same source are correlated? Or somehow the > multiple measures from the same source can be somehow summarized > (weighted average?) before the meta analysis? > > Your suggestions are highly appreciated. > > Best wishes, > Gang > > > On Sun, Feb 7, 2010 at 10:39 AM, Mike Cheung <mikewlcheung at gmail.com> wrote: >> Dear Gang, >> >> Here are just some general thoughts. Wolfgang Viechtbauer will be a >> better position to answer questions related to metafor. >> >> For multivariate effect sizes, we first have to estimate the >> asymptotic sampling covariance matrix among the effect sizes. Formulas >> for some common effect sizes are provided by Gleser and Olkin (2009). >> >> If a fixed-effects model is required, it is quite easy to write your >> own GLS function to conduct the multivariate meta-analysis (see e.g., >> Becker, 1992). If a random-effects model is required, it is more >> challenging in R. SAS Proc MIXED can do the work (e.g., van >> Houwelingen, Arends, & Stijnen, 2002). >> >> Sometimes, it is possible to transform the multivariate effect sizes >> into independent effect sizes (Kalaian & Raudenbush, 1996; Raudenbush, >> Becker, & Kalaian, 1988). Then univariate meta-analysis, e.g., >> metafor(), can be performed on the transformed effect sizes. This >> approach works if it makes sense to pool the multivariate effect sizes >> as in your case (2)- the effect sizes are the same but in different >> conditions (happy, sad, and neutral). However, this approach does not >> work if the multivariate effect sizes are measuring different >> concepts, e.g., verbal achievement and mathematical achievement. >> >> Hope this helps. >> >> Becker, B. J. (1992). Using results from replicated studies to >> estimate linear models. Journal of Educational Statistics, 17, >> 341-362. >> Gleser, L. J., & Olkin, I. (2009). Stochastically dependent effect >> sizes. In H. Cooper, L. V. Hedges, and J. C. Valentine (Eds.), The >> handbook of research synthesis and meta-analysis, 2nd edition (pp. >> 357-376). New York: Russell Sage Foundation. >> Kalaian, H. A., & Raudenbush, S. W. (1996). A multivariate mixed >> linear model for meta-analysis. Psychological Methods, 1, 227-235. >> Raudenbush, S. W., Becker, B. J., & Kalaian, H. (1988). Modeling >> multivariate effect sizes. Psychological Bulletin, 103, 111-120. >> van Houwelingen, H.C., Arends, L.R., & Stijnen, T. (2002). Advanced >> methods in meta-analysis: multivariate approach and meta-regression. >> Statistics in Medicine, 21, 589-624. >> >> Regards, >> Mike >> -- >> --------------------------------------------------------------------- >> ?Mike W.L. Cheung ? ? ? ? ? ? ? Phone: (65) 6516-3702 >> ?Department of Psychology ? ? ? Fax: ? (65) 6773-1843 >> ?National University of Singapore >> ?http://courses.nus.edu.sg/course/psycwlm/internet/ >> --------------------------------------------------------------------- >> >> On Sat, Feb 6, 2010 at 6:07 AM, Gang Chen <gangchen6 at gmail.com> wrote: >>> In a classical meta analysis model y_i = X_i * beta_i + e_i, data >>> {y_i} are assumed to be independent effect sizes. However, I'm >>> encountering the following two scenarios: >>> >>> (1) Each source has multiple effect sizes, thus {y_i} are not fully >>> independent with each other. >>> (2) Each source has multiple effect sizes, and each of the effect size >>> from a source can be categorized as one of a factor levels (e.g., >>> happy, sad, and neutral). Maybe better denote the data as y_ij, effect >>> size at the j-th level from the i-th source. I can code the levels >>> with dummy variables into the X_i matrix, but apparently the data from >>> the same source are correlated with each other. In this case, I would >>> like to run a few tests one of which is, for example, whether there is >>> any difference across all the levels of the factor. >>> >>> Can metafor handle these two cases? >>> >>> Thanks, >>> Gang >
Viechtbauer Wolfgang (STAT)
2010-Feb-08 09:22 UTC
[R] metafor package: effect sizes are not fully independent
Hello Gang, Mike has already given you some excellent advice and references. I just want to add some information about the metafor package. To be precise, the meta-analytic mixed-effects model is given by: y_i = X_i * beta_i + u_i + e_i where X_i and beta_i have the usual interpretation, u_i is a random effect (for "residual heterogeneity"), and e_i is sampling error. Var(e_i) = v_i is assumed to be known for each study. Moreover, assume Cov(e_i, e_i') = 0, Cov(u_i, u_i') = 0, and Cov(u_i, e_i) = 0. This is the model underlying the rma() function. Correlation can arise between the e_i's (i.e., Cov(e_i, e_i') != 0), between the u_i's (i.e., Cov(u_i, u_i') != 0), or both. For example, if you have multiple estimates from the same study calculated from the same sample of subjects, then the e_i's for that study will certainly be correlated and it usually makes sense then to assume that the corresponding u_i's are correlated as well. For the correlation between the e_i's, there are often equations that you can use to calculate those (or to be precise: to estimate those correlations). So, instead of just assuming that v_i is known, we then have an entire variance-covariance matrix of the e_i's that is assumed to be known. The tricky thing then is to figure out what structure to use for the variance-covariance matrix of correlated u_i's. Currently, the metafor package cannot handle situations where some of the effect size estimates are correlated. I am currently working on functions that will allow for correlated estimates, but it will take some time before those will become part of the package. If it makes sense to do so in your case, you could consider aggregating dependent estimates into a single estimate and then using that for the meta-analysis. Best, -- Wolfgang Viechtbauer http://www.wvbauer.com/ Department of Methodology and Statistics Tel: +31 (43) 388-2277 School for Public Health and Primary Care Office Location: Maastricht University, P.O. Box 616 Room B2.01 (second floor) 6200 MD Maastricht, The Netherlands Debyeplein 1 (Randwyck) ----Original Message---- From: r-help-bounces at r-project.org [mailto:r-help-bounces at r-project.org] On Behalf Of Gang Chen Sent: Friday, February 05, 2010 23:08 To: r-help at r-project.org Subject: [R] metafor package: effect sizes are not fully independent> In a classical meta analysis model y_i = X_i * beta_i + e_i, data > {y_i} are assumed to be independent effect sizes. However, I'm > encountering the following two scenarios: > > (1) Each source has multiple effect sizes, thus {y_i} are not fully > independent with each other. (2) Each source has multiple effect > sizes, each of the effect size from a source can be categorized as > one of a factor levels (e.g., happy, sad, and neutral). Maybe better > denote the data as y_ij, effect size at the j-th level from the i-th > source. I can code the levels with dummy variables into the X_i > matrix, but apparently the data from the same source are correlated > with each other. In this case, I would like to run a few tests one of > which is, for example, whether there is any difference across all the > levels of the factor. > > Can metafor handle these two cases? > > Thanks, > Gang > > ______________________________________________ > R-help at r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide > http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code.