R help - Jun 2009 - Comparing model fits for NLME when models are not nested

Hi there,

I am looking to compare nonlinear mixed effects models that have different
nonlinear functions (different types of growth curve)embedded. Most of the
literature I can find focuses on comparing nested models with likelihood ratios
and AIC. Is there a way to compare model fits when models are not nested, i.e.
when the nonlinear functions are not the same?

Many thanks in advance!
Lindsay

Lindsay Banin <l.banin05 at leeds.ac.uk> wrote
>Hi there,
>
>I am looking to compare nonlinear mixed effects models that have different
nonlinear functions (different types of growth curve)embedded. Most of the
literature I can find focuses on comparing nested models with likelihood ratios
and AIC. Is there a way to compare model fits when models are not nested, i.e.
when the nonlinear functions are not the same?
>
What I like to do in these cases (and, in fact, for comparing model fits in
general, even if nested) is graph the results of one vs the other, and of each
vs. actual values.  If the predicted values from the two models are very
similar, then I can choose based on complexity or some other criterion; if they
are not similar, then which is closer to the actual values?  Is the difference
large? Is it "worth it"?

It's  not a formal test, but I often find it illuminating.

HTH

Peter



Peter L. Flom, PhD
Statistical Consultant
www DOT peterflomconsulting DOT com

At 05:42 AM 6/12/2009, Lindsay Banin wrote:
>Hi there,
>
>I am looking to compare nonlinear mixed effects models that have 
>different nonlinear functions (different types of growth 
>curve)embedded. Most of the literature I can find focuses on 
>comparing nested models with likelihood ratios and AIC. Is there a 
>way to compare model fits when models are not nested, i.e. when the 
>nonlinear functions are not the same?
Transform back into original units, if necessary, and compare 
distributions of and statistics of residuals from fitted values in 
original units.

This is not a significance-test, but instead a measure of the better 
approximation to the observed model.

Types of measures: 1) rms residual, 2) max absolute residual, 3) mean 
absolute residual.

In my opinion, models should be chosen based on the principles of 
causality (theory), degree of approximation and parsimony. None of 
these involve significance testing.

Choosing models based upon significance testing (which merely 
identifies whether or not the experiment is large enough to 
distinguish an effect clearly) amounts to admitting intellectually 
that you have no subject matter expertise, and you must therefore 
fall back on the crumbs of significance testing to get glimmers of 
understanding of what's going on. (Much like stepwise regression
techniques.)

As an example, suppose you have two models, one with 5 parameters and 
one with only 1. The rms residual error for the two models are 0.50 
and 0.53 respectively. You have a very large study, and all 4 
additional parameters are significant at p = 0.01 or less.  What 
should you do? What I would do is select the 1 parameter study as my 
baseline model. It will be easy to interpret physically, will 
generalize to other studies much better (stable), and is almost 
identical in degree of approximation as the 5 parameter model. I 
would be excited that a one parameter model could do this. The fact 
that the other 4 parameters have detectable effects at a very low 
level is not important for modeling the study, but may conceivably 
have some special significance on their own for future investigations.

So not being able to do significance testing on non-nested models is 
not that big a loss, in my opinion. Such tests encourage wrong 
thinking, in my opinion.

What I've expressed as an opinion here (which I am sure some will 
disagree with) is similar to the philosophy of choosing the number of 
principal components to use, or number of latent factors in factor 
analysis. What investigation do people ever do on the small 
eigenvalue principal components, even if their contributions are 
"statistically significant"?

===============================================================Robert A.
LaBudde, PhD, PAS, Dpl. ACAFS  e-mail: ral at lcfltd.com
Least Cost Formulations, Ltd.            URL: http://lcfltd.com/
824 Timberlake Drive                     Tel: 757-467-0954
Virginia Beach, VA 23464-3239            Fax: 757-467-2947

"Vere scire est per causas scire"