R help - Sep 2005 - model comparison and Wald-tests (e.g. in lmer)

Dear expeRts,

there is obviously a general trend to use model comparisons, LRT and AIC
instead of Wald-test-based significance, at least in the R community.
I personally like this approach. And, when using LME's, it seems to be 
the preferred way (concluded from postings of Brian Ripley and Douglas 
Bates' article in R-News 5(2005)1), esp. because of problems with the 
d.f. approximation.

But, on the other hand I found that not all colleagues are happy with the
resulting AIC/LRT tables and the comparison of multiple models.

As a compromise, and after a suggestion in Crawley's "Statistical
computing" one may consider to supply "traditional" ANOVA tables
as an
additional explanation for the reader (e.g. field biologists).

An example:

one has fitted 5 models m1..m5 and after:

 >anova(m1,m2,m3,m4,m5) # giving AIC and LRT-tests

he selects m3 as the most parsimonious model and calls anova with the 
best model (Wald-test):

 >anova(m3)             # the additional explanatory table

My questions:

* Do people outside the S-PLUS/R world still understand us?

* Is it wise to add such an explanatory table (in particular when the 
results are the same) to make the results more transparent to the reader?

* Are such additional ANOVA tables *really helpful* or are they (in 
combination with a model comparison) just another source of confusion?


Thank you!

Thomas P.

On 9/5/05, Thomas Petzoldt <thpe at hhbio.wasser.tu-dresden.de>
wrote:
> Dear expeRts,
> 
> there is obviously a general trend to use model comparisons, LRT and AIC
> instead of Wald-test-based significance, at least in the R community.
> I personally like this approach. And, when using LME's, it seems to be
> the preferred way (concluded from postings of Brian Ripley and Douglas
> Bates' article in R-News 5(2005)1), esp. because of problems with the
> d.f. approximation.
> 
> But, on the other hand I found that not all colleagues are happy with the
> resulting AIC/LRT tables and the comparison of multiple models.
> 
> As a compromise, and after a suggestion in Crawley's "Statistical
> computing" one may consider to supply "traditional" ANOVA
tables as an
> additional explanation for the reader (e.g. field biologists).
> 
> An example:
> 
> one has fitted 5 models m1..m5 and after:
> 
>  >anova(m1,m2,m3,m4,m5) # giving AIC and LRT-tests
> 
> he selects m3 as the most parsimonious model and calls anova with the
> best model (Wald-test):
> 
>  >anova(m3)             # the additional explanatory table
Whether or not this is a good idea will depend on what the differences
in the models are.  Two mixed-effects models for the same data set can
differ in their random effects specification or in the fixed-effects
specification or both.  The anova() function applied to a single lmer
model produces a sequential anova table for the fixed-effects terms.

If the models differ in the random effects specification - say the
full model has random effects for slope and intercept but the
restricted model has a random effect for the intercept only - then a
Wald test is not appropriate (and it is not provided).  In those cases
I would use a likelihood ratio test and, if necessary, approximate the
p-value by simulating from the null hypothesis rather than assuming a
chi-squared distribution of the test statistic.

Recent versions of the mlmRev package have a vignette with extensive
analysis of the Exam data, including MCMC samples from the posterior
distribution of the parameters.  The marginal posterior distribution
of the variance components are quite clearly skewed (not surprisingly,
they look like scaled chi-squared distributions).  Testing whether
such a parameter could be zero by creating a z-statistic from the
estimate and its standard error is inappropriate.

Changing both the fixed-effects and the random-effects specification
is tricky.  I would try to do such changes in stages, settling on the
fixed-effects terms first then checking the random-effects
specification.
> 
> My questions:
> 
> * Do people outside the S-PLUS/R world still understand us?
> 
> * Is it wise to add such an explanatory table (in particular when the
> results are the same) to make the results more transparent to the reader?
> 
> * Are such additional ANOVA tables *really helpful* or are they (in
> combination with a model comparison) just another source of confusion?
> 
> 
> Thank you!
> 
> Thomas P.
> 
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide!
http://www.R-project.org/posting-guide.html
>

Hello,

I wish to thank Douglas Bates very much for clarification and pointing 
me to the MCMC simulation method to get p values even for cases where 
Wald tests are inappropriate.

One question however remains when publishing statistical results: does 
it help readers if we combine both,

- AIC based model selection
*and*
- null hypothesis based tests statistics

or should we focus on model selection only and try to reduce the amount 
of tables provided?

Apologies if this is question is too much off-topic, so you may decide 
to answer off-list. I will give a short summary at the end.

Thomas P.

An article explaining the background:

Johnson, J. & Omland, K.S., Model Selection in Ecology and Evolution. 
Trends in Ecology and Evolution, 2004, 19 , 101-108