R help - Aug 2005 - [SPAM] - Re: How to assess significance of random effect in lme4 - Bayesian Filter detected spam

Actually, I re-read the post and think it needs clarification. We may
both be right. If the question is "I am building a model and want to
know if I should retain this random effect?" (or something like that)
then the LRT should be used to compare the fitted model against another
model. This would be accomplished via anova().

In other multilevel programs, the variance components are often
associated with a chi-square statistic and a test statistic associated
with the variance. lmer() does not report this test statistic.

Now, if the question is something like "I want to know if a specific
realization of the random variable (i.e., a specific empirical Bayes
estimate) is different from a population value?" then one would need the
posterior means.

So, Shige, I hope this hasn't been confusing, but there are many things
happening in these models and it is easy to get confused. Maybe if you
could clarify your question. 

-----Original Message-----
From: Spencer Graves [mailto:spencer.graves at pdf.com] 
Sent: Thursday, August 18, 2005 8:26 AM
To: Doran, Harold
Cc: Shige Song; r-help at stat.math.ethz.ch
Subject: [SPAM] - Re: [R] How to assess significance of random effect in
lme4 - Bayesian Filter detected spam

Hi, Harold:  Thanks for the clarification.  I thought I had read the
original post.  Obviously, I had misread it.  Thanks again.  spencer
graves

Doran, Harold wrote:
> Yes, it is a different issue. ranef() extracts the empirical Bayes 
> estimates, which are the empirical posterior modes. The bVar slot 
> holds the corresponding posterior variances of these modes.
> 
> Technically, (according to D. Bates) the values in the bVar slot are 
> the the diagonal elements of (Z'Z+\Omega)^{-1}.
> 
> The original post was asking how to test and compare a specific random
> effect, not a general assessment of how much information is provided 
> by the data via LRT.
> 
> Shige asked how to test whether a specific EB estimate is different 
> than some other value.
> LRT doesn't answer this question, but the values in the bVar slot do.
> 
> 
> -----Original Message-----
> From:   Spencer Graves [mailto:spencer.graves at pdf.com]
> Sent:   Wed 8/17/2005 10:08 PM
> To:     Doran, Harold
> Cc:     Shige Song; r-help at stat.math.ethz.ch
> Subject:        Re: [R] How to assess significance of random effect in
lme4
> 
>           Is there some reason you are NOT using "anova", as in
"Examples"
> section of "?lmer"?
> 
>           Permit me to summarize what I know about this, and I'll be 
> pleased if someone else who thinks they know different would kindly 
> enlighten me and others who might otherwise be misled if anything I 
> say is inconsistent with the best literature available at the moment:
> 
>           1.  Doug Bates in his PhD dissertation and later in his book
> with Don Watts (1988) Nonlinear Regression Analysis and Its 
> Applications (Wiley) split approximation errors in nonlinear least 
> squares into "intrinsic curvature" and "parameter effects
curvature".
> He quantified these two problems in the context of roughly three dozen
> published examples, if my memory is correct, and found that in not 
> quite all cases, the parameter effects were at least an order of 
> magnitude greater than the intrinsic curvature.
> 
>           2.  In nonnormal situations, maximum likelihood is subject 
> to more approximation error -- intrinsic curvature -- than
"simple"
> nonlinear least squares.  However, I would expect this comparison to 
> still be fairly accurate, even if the differences may not be quite as
stark.
> 
>           3.  The traditional use of "standard errors" to judge 
> statistical significance is subject to both intrinsic and parameter 
> effects errors, while likelihood ratio procedures such as anova are 
> subject only to the intrinsic curvature (assuming there are no 
> substantive problems with nonconvergence).  Consequently, to judge 
> statistical significance of an effect, anova is usually substantially 
> better than the so-called Wald procedure using approximate standard
errors, and is almost never worse.
>   If anyone knows of a case where this is NOT true, I'd like to know.
> 
>           4.  With parameters at a boundary as with variance 
> components, the best procedure seems to double the p-value from a 
> nested anova (unless the reported p-value is already large).  This is 
> because the 2*log(likelihood ratio) in such cases is roughly a 50-50 
> mixture of 0 and chi-square(1) [if testing only 1 variance component
parameter].
> This is supported by a substantial amount of research, including 
> simulations discussed in a chapter in Pinheiro and Bates (2000) 
> Mixed-Effects Models in S and S-Plus (Springer).  The may be more 
> accurate procedures available in the literature, but none so simple as
> this as far as I know.
> 
>           Comments?
>           spencer graves
> p.s.  It looks like fm at bVars is a list containing vectors of length 29
> and 6 in your example.  I don't know what they are, but I don't see
> how they can be standard errors in the usual sense.
> 
> Doran, Harold wrote:
> 
>  > These are the posterior variances of the random effects (I think 
> more  > properly termed "empirical" posteriors).  Your model 
> apparently includes  > three levels of random variation (commu, 
> bcohort, residual). The first  > are the variances associated with 
> your commu random effect and the  > second are the variances
associated with the bcohort random effect.
>  >
>  > Accessing either one would require
>  >
>  > fm at bVar$commu or fm at bVar$bcohort
>  >
>  > Obviously, replace "fm" with the name of your fitted model.
>  >
>  > -----Original Message-----
>  > From: r-help-bounces at stat.math.ethz.ch  > 
> [mailto:r-help-bounces at stat.math.ethz.ch] On Behalf Of Shige Song  > 
> Sent: Wednesday, August 17, 2005 7:50 AM  > To: 
> r-help at stat.math.ethz.ch  > Subject: Re: [R] How to assess 
> significance of random effect in lme4  >  > Hi Harold,  >  >
Thanks
> for the reply. I looked at my outputs using str() as you  > suggested,
> here is the part you mentioned:
>  >
>  >   ..@ bVar     :List of 2
>  >   .. ..$ commu  : num [1, 1, 1:29] 5e-10 5e-10 5e-10 5e-10 5e-10
...
>  >   .. ..$ bcohort: num [1, 1, 1:6] 1.05e-05 7.45e-06 6.53e-06
8.25e-06
>  > 7.11e-06 ...
>  >
>  > where commu and bcohort are the two second-level units. Are these  
> > standard errors? Why the second vector contains a series of 
> different  > numbers?
>  >
>  > Thanks!
>  >
>  > Shige
>  >
>  > On 8/17/05, Doran, Harold <HDoran at air.org> wrote:
>  >
>  >>
>  >>
>  >>You can extract the posterior variance of the random effect from 
> the  >>bVar slot of the fitted lmer model. It is not a hidden option,
> but a  >>part of the fitted model. It just doesn't show up when
you
> use  >  > summary().
>  >
>  >>
>  >> Look at the structure of your object to see what is available 
> using  >  > str().
>  >
>  >>
>  >> However, your comment below seems to imply that it is incorrect 
> for  >>lmer to report SDs instead of the standard error, which is not
true.
>  >>That is a quantity of direct interest.
>  >>
>  >> Other multilevel programs report the same exact statistics (for 
> the  >>most part). For instance, HLM reports the variances as well.
If
> you  >>want the posterior variance of an HLM model you need to
extract
it.
>  >>
>  >>
>  >>
>  >> -----Original Message-----
>  >> From:   r-help-bounces at stat.math.ethz.ch on behalf of
>  >>Shige Song
>  >> Sent:   Wed 8/17/2005 6:30 AM
>  >> To:     r-help at stat.math.ethz.ch
>  >> Cc:   
>  >> Subject:        [R] How to assess significance of random effect
in
>  >
>  > lme4
>  >
>  >>
>  >> Dear All,
>  >>
>  >> With kind help from several friends on the list, I am getting
close.
>  >> Now here are something interesting I just realized: for random  
> >>effects, lmer reports standard deviation instead of standard error!
> Is  >  >  >>there a hidden option that tells lmer to report
standard
> error of  >>random effects, like most other multilevel or mixed 
> modeling software,  >  >  >>so that we can say something like
"randome
> effect for xxx is  >>significant, while randome effect for xxx is not
> significant"? Thanks!
>  >>
>  >> Best,
>  >> Shige
>  >>
>  >> ______________________________________________
>  >> R-help at stat.math.ethz.ch mailing list  
> >>https://stat.ethz.ch/mailman/listinfo/r-help
>  >> PLEASE do read the posting guide!
>  >>http://www.R-project.org/posting-guide.html
>  >>
>  >>
>  >>
>  >>
>  >>
>  >
>  >
>  > ______________________________________________
>  > R-help at stat.math.ethz.ch mailing list  > 
> https://stat.ethz.ch/mailman/listinfo/r-help
>  > PLEASE do read the posting guide!
>  > http://www.R-project.org/posting-guide.html
>  >
>  > ______________________________________________
>  > R-help at stat.math.ethz.ch mailing list  > 
> https://stat.ethz.ch/mailman/listinfo/r-help
>  > PLEASE do read the posting guide! 
> http://www.R-project.org/posting-guide.html
> 
> --
> Spencer Graves, PhD
> Senior Development Engineer
> PDF Solutions, Inc.
> 333 West San Carlos Street Suite 700
> San Jose, CA 95110, USA
> 
> spencer.graves at pdf.com
> www.pdf.com <http://www.pdf.com>
> Tel:  408-938-4420
> Fax: 408-280-7915
> 
> 
--
Spencer Graves, PhD
Senior Development Engineer
PDF Solutions, Inc.
333 West San Carlos Street Suite 700
San Jose, CA 95110, USA

spencer.graves at pdf.com
www.pdf.com <http://www.pdf.com>
Tel:  408-938-4420
Fax: 408-280-7915