R help - Jun 2008 - About Mcneil Hanley test for a portion of AUC!

Dear all

 I am trying to compare the performances of several methods using the AUC0.1
and
not the whole AUC. (meaning I wanted to compare to AUC's whose x axis only
goes to
0.1 not 1)

I came to know about the Mcneil Hanley test from Bernardo Rangel Tura
and I referred to the original paper for the calculation of "r" which
is an
argument of the function
cROC. I can only find the value of "r" for the whole AUC's .



> seROC<-function(AUC,na,nn){
> a<-AUC
> q1<-a/(2-a)
> q2<-(2*a^2)/(1+a)
> se<-sqrt((a*(1-a)+(na-1)*(q1-a^2)+(nn-1)*(q2-a^2))/(nn*na))
> se
> }
>
> cROC<-function(AUC1,na1,nn1,AUC2,na2,nn2,r){
> se1<-seROC(AUC1,na1,nn1)
> se2<-seROC(AUC2,na2,nn2)
>
> sed<-sqrt(se1^2+se2^2-2*r*se1*se2)
> zad<-(AUC1-AUC2)/sed
> p<-dnorm(zad)
> a<-list(zad,p)
> a
Could somebody kindly suggest me how to calculate the value of "r" or
some ways to calculate the statistical significance measure for the
differences of auc for a part of
the curve like AUC0.1.

Thank You
-- 
Dukka KC
UNCC

	[[alternative HTML version deleted]]

Dukka k.c. wrote:
> Dear all
> 
>  I am trying to compare the performances of several methods using the
AUC0.1
> and
> not the whole AUC. (meaning I wanted to compare to AUC's whose x axis
only
> goes to
> 0.1 not 1)
> 
> I came to know about the Mcneil Hanley test from Bernardo Rangel Tura
> and I referred to the original paper for the calculation of "r"
which is an
> argument of the function
> cROC. I can only find the value of "r" for the whole AUC's .
> 
> 
> 
> 
>> seROC<-function(AUC,na,nn){
>> a<-AUC
>> q1<-a/(2-a)
>> q2<-(2*a^2)/(1+a)
>> se<-sqrt((a*(1-a)+(na-1)*(q1-a^2)+(nn-1)*(q2-a^2))/(nn*na))
>> se
>> }
>>
>> cROC<-function(AUC1,na1,nn1,AUC2,na2,nn2,r){
>> se1<-seROC(AUC1,na1,nn1)
>> se2<-seROC(AUC2,na2,nn2)
>>
>> sed<-sqrt(se1^2+se2^2-2*r*se1*se2)
>> zad<-(AUC1-AUC2)/sed
>> p<-dnorm(zad)
>> a<-list(zad,p)
>> a
> 
> Could somebody kindly suggest me how to calculate the value of
"r" or
> some ways to calculate the statistical significance measure for the
> differences of auc for a part of
> the curve like AUC0.1.
> 
> Thank You
The ROC area is not a sensitive enough measure for comparing two 
competing predictors.  Its power is too low.  See for example the 
following papers.  Note that Pencina et al's approach is now in the 
Hmisc package (function improveProb; documentation to be coming soon). 
Likelihood ratio tests are even more powerful.

@Article{pen08eva,
   author = 		 {Pencina, Michael J. and {D'Agostino Sr}, Ralph B. and 
{D'Agostino Jr}, Ralph B. and Vasan, Ramachandran S.},
   title = 		 {Evaluating the added predictive ability of a new marker: 
{From} area under the {ROC} curve to reclassification and beyond},
   journal = 	 Stat in Med,
   year = 		 2008,
   volume =		 27,
   pages =        {157-172},
   annote =		 {discrimination;model performance;AUC;C-index;risk 
prediction;biomarker;small differences in ROC area can still be very 
meaningful;example of insignificant test for difference in ROC areas 
with very significant results from new method;Yates' discrimination 
slope;reclassification table;limiting version of this based on whether 
and amount by which probabilities rise for events and lower for 
non-events when compare new model to old;comparing two models}
}

@Article{coo07use,
   author = 		 {Cook, Nancy R.},
   title = 		 {Use and misues of the receiver operating characteristic 
curve in risk prediction},
   journal = 	 {Circulation},
   year = 		 2007,
   volume =		 115,
   pages =		 {928-935},
   annote =		 {reclassification table;problems with c index;problems 
with ROC area;example of large change in predicted risk in 
cardiovascular disease with tiny change in ROC area;possible limits to c 
index when calibration is perfect;importance of calibration accuracy and 
changes in predicted risk when new variables are added}
}


-- 
Frank E Harrell Jr   Professor and Chair           School of Medicine
                      Department of Biostatistics   Vanderbilt University