R help - Jul 2008 - Comparing differences in AUC from 2 different models

Hi,

I would like to compare differences in AUC from 2 different models, glm and gam
for predicting presence / absence. I know that in theory the model with a higher
AUC is better, but what I am interested in is if statistically the increase in
AUC from the glm model to the gam model is significant. I also read quite
extensive discussions on the list about ROC and AUC but I still didn't find
my answer.

To calculate the AUC and plot the ROC I used the package PresenceAbsence. The
help file for auc() says: " The standard errors from auc are only valid for
comparing an individual model to random assignment (i.e. AUC=.5). To compare two
models to each other it is necessary to account for correlation due to the fact
that they use the same test set. If you are interested in pair wise model
comparisons see the Splus ROC library from Mayo clinic. auc is a much simpler
function than what is available from the Splus ROC library from Mayo
clinic."

I did download this library but I don't have access to S-PLUS and even if
supposedly the code is very similar between S-PLUS and R I still don't quite
understand what is going on because I am a little bit confused what some
parameters represent ?. For example "markers" and "status",
although I think "status" represent my original data (all coded 0 and
1) and "markers" might be the probabilities obtained from my 2 models.
The confusion may also steam from the fact that I don't have a medical or
biological training and maybe "markers" and "status" do have
a special meaning for these 2 disciplines.

I will really appreciate if you can help in finding a way to compare differences
in AUC.

Thanks,

Monica



_________________________________________________________________


_WL_Refresh_messenger_video_072008

Monica Pisica wrote:
> Hi,
> 
> I would like to compare differences in AUC from 2 different models, glm and
gam for predicting presence / absence. I know that in theory the model with a
higher AUC is better, but what I am interested in is if statistically the
increase in AUC from the glm model to the gam model is significant. I also read
quite extensive discussions on the list about ROC and AUC but I still didn't
find my answer.
> 
> To calculate the AUC and plot the ROC I used the package PresenceAbsence.
The help file for auc() says: " The standard errors from auc are only valid
for comparing an individual model to random assignment (i.e. AUC=.5). To compare
two models to each other it is necessary to account for correlation due to the
fact that they use the same test set. If you are interested in pair wise model
comparisons see the Splus ROC library from Mayo clinic. auc is a much simpler
function than what is available from the Splus ROC library from Mayo
clinic."
> 
> I did download this library but I don't have access to S-PLUS and even
if supposedly the code is very similar between S-PLUS and R I still don't
quite understand what is going on because I am a little bit confused what some
parameters represent ?. For example "markers" and "status",
although I think "status" represent my original data (all coded 0 and
1) and "markers" might be the probabilities obtained from my 2 models.
The confusion may also steam from the fact that I don't have a medical or
biological training and maybe "markers" and "status" do have
a special meaning for these 2 disciplines.
> 
> I will really appreciate if you can help in finding a way to compare
differences in AUC.
> 
> Thanks,
> 
> Monica
> 
> 
> 
Comparison of ROC areas does not have sufficient power to detect 
important differences of two models.  See the following, for which I 
have R/S+ code.  Likelihood ratio tests for nested models are even more 
powerful.   -Frank


@Article{pen08eva,
   author = 		 {Pencina, Michael J. and {D'Agostino Sr}, Ralph B. and 
{D'Agostino Jr}, Ralph B. and Vasan, Ramachandran S.},
   title = 		 {Evaluating the added predictive ability of a new marker: 
{From} area under the {ROC} curve to reclassification and beyond},
   journal = 	 Stat in Med,
   year = 		 2008,
   volume =		 27,
   pages =        {157-172},
   annote =		 {discrimination;model performance;AUC;C-index;risk 
prediction;biomarker;small differences in ROC area can still be very 
meaningful;example of insignificant test for difference in ROC areas 
with very significant results from new method;Yates' discrimination 
slope;reclassification table;limiting version of this based on whether 
and amount by which probabilities rise for events and lower for 
non-events when compare new model to old;comparing two models}
}


-- 
Frank E Harrell Jr   Professor and Chair           School of Medicine
                      Department of Biostatistics   Vanderbilt University