R help - Apr 2005 - generalized linear mixed models - how to compare?

Dear all,

I want to evaluate several generalized linear mixed models, including the null
model, and select the best approximating one. I have tried glmmPQL (MASS
library) and GLMM (lme4) to fit the models. Both result in similar parameter
estimates but fairly different likelihood estimates.
My questions:
1- Is it correct to calculate AIC for comparing my models, given that they use
quasi-likelihood estimates? If not, how can I compare them?
2- Why the large differences in likelihood estimates between the two procedures?

Thanks,

Nestor

On Sunday 17 April 2005 08:39, Nestor Fernandez wrote:
> Dear all,
>
> I want to evaluate several generalized linear mixed models, including
> the null model, and select the best approximating one. I have tried
> glmmPQL (MASS library) and GLMM (lme4) to fit the models. Both result
> in similar parameter estimates but fairly different likelihood
> estimates.
> My questions:
> 1- Is it correct to calculate AIC for comparing my models, given that
> they use quasi-likelihood estimates? If not, how can I compare them?
> 2- Why the large differences in likelihood estimates between the two
> procedures?
The likelihood reported by glmmPQL is wrong, as it's the likelihood of 
an incorrect model (namely, an lme model that approximates the correct 
glmm model). GLMM uses (mostly) the same procedure to get parameter 
estimates, but as a final step calculates the likelihood for the 
correct model for those estimates (so the likelihood reported by it 
should be fairly reliable).

Deepayan

Hello All:

Should I conclude from this discussion that there is no practical
means by which nested generalized mixed models can be compared from
output produced through glmmPQL or GLMM? What is one then to do???

Andrew


On Sun, 17 Apr 2005, Deepayan Sarkar wrote:
> On Sunday 17 April 2005 08:39, Nestor Fernandez wrote:
>> I want to evaluate several generalized linear mixed models, including
>> the null model, and select the best approximating one. I have tried
>> glmmPQL (MASS library) and GLMM (lme4) to fit the models. Both result
>> in similar parameter estimates but fairly different likelihood
>> estimates.
>> My questions:
>> 1- Is it correct to calculate AIC for comparing my models, given that
>> they use quasi-likelihood estimates? If not, how can I compare them?
>> 2- Why the large differences in likelihood estimates between the two
>> procedures?
>
>
> The likelihood reported by glmmPQL is wrong, as it's the likelihood of
> an incorrect model (namely, an lme model that approximates the correct
> glmm model).

Actually glmmPQL does not report a likelihood.  It returns an object
of class "lme", but you need to refer to the reference for how to
interpret that.  It *is* support software for a book.
> GLMM uses (mostly) the same procedure to get parameter estimates, but as a
final step calculates the likelihood for the correct model for those estimates
(so the likelihood reported by it should be fairly reliable).

Well, perhaps but I need more convincing.  The likelihood involves
many high-dimensional non-analytic integrations, so I do not see how
GLMM can do those integrals -- it might approximate them, but that
would not be `calculates the likelihood for the correct model'.  It
would be helpful to have a clarification of this claim.  (Our
experiments show that finding an accurate value of the log-likelihood
is difficult and many available pieces of software differ in their
values by large amounts.)

Further, since neither procedure does ML fitting, this is not a
maximized likelihood as required to calculate an AIC value.  And even
if it were, you need to be careful as often one GLMM is a boundary
value for another, in which case the theory behind AIC needs
adjustment.

-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595

______________________________________________
R-help at stat.math.ethz.ch mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide! http://www.R-project.org/posting-guide.html



-- 
Andrew R. Criswell, Ph.D.
Graduate School, Bangkok University

Andrew Criswell asks:

	Hello All:

	Should I conclude from this discussion that there is no
practical
	means by which nested generalized mixed models can be compared
from
	output produced through glmmPQL or GLMM?

[WNV]  The picture is, in my view, not as bleak as this, but there are
are 
certainly many open questions in this area and much research left to do.

	 What is one then to do???

[WNV]  Research in statistics, perhaps?  Every little bit helps.  It is
a
mistake to assume that everything is known about even the common
approximations
used in statistical practice, and this area is still opening up.

	Andrew


	On Sun, 17 Apr 2005, Deepayan Sarkar wrote:

	> On Sunday 17 April 2005 08:39, Nestor Fernandez wrote:


	>> I want to evaluate several generalized linear mixed models,
including
	>> the null model, and select the best approximating one. I have
tried
	>> glmmPQL (MASS library) and GLMM (lme4) to fit the models.
Both result
	>> in similar parameter estimates but fairly different
likelihood
	>> estimates.
	>> My questions:
	>> 1- Is it correct to calculate AIC for comparing my models,
given that
	>> they use quasi-likelihood estimates? If not, how can I
compare them?
	>> 2- Why the large differences in likelihood estimates between
the two
	>> procedures?
	>
	>
	> The likelihood reported by glmmPQL is wrong, as it's the
likelihood of
	> an incorrect model (namely, an lme model that approximates the
correct
	> glmm model).


	Actually glmmPQL does not report a likelihood.  It returns an
object
	of class "lme", but you need to refer to the reference for how
to
	interpret that.  It *is* support software for a book.

	> GLMM uses (mostly) the same procedure to get parameter
estimates, but as a final step calculates the likelihood for the correct
model for those estimates (so the likelihood reported by it should be
fairly reliable).


	Well, perhaps but I need more convincing.  The likelihood
involves
	many high-dimensional non-analytic integrations, so I do not see
how
	GLMM can do those integrals -- it might approximate them, but
that
	would not be `calculates the likelihood for the correct model'.
It
	would be helpful to have a clarification of this claim.  (Our
	experiments show that finding an accurate value of the
log-likelihood
	is difficult and many available pieces of software differ in
their
	values by large amounts.)

	Further, since neither procedure does ML fitting, this is not a
	maximized likelihood as required to calculate an AIC value.  And
even
	if it were, you need to be careful as often one GLMM is a
boundary
	value for another, in which case the theory behind AIC needs
	adjustment.

	-- 
	Brian D. Ripley,                  ripley at stats.ox.ac.uk
	Professor of Applied Statistics,
http://www.stats.ox.ac.uk/~ripley/
	University of Oxford,             Tel:  +44 1865 272861 (self)
	1 South Parks Road,                     +44 1865 272866 (PA)
	Oxford OX1 3TG, UK                Fax:  +44 1865 272595

	______________________________________________
	R-help at stat.math.ethz.ch mailing list
	https://stat.ethz.ch/mailman/listinfo/r-help
	PLEASE do read the posting guide!
http://www.R-project.org/posting-guide.html



	-- 
	Andrew R. Criswell, Ph.D.
	Graduate School, Bangkok University

	______________________________________________
	R-help at stat.math.ethz.ch mailing list
	https://stat.ethz.ch/mailman/listinfo/r-help
	PLEASE do read the posting guide!
http://www.R-project.org/posting-guide.html