R help - Jul 2009 - ROCR - confidence interval for Sens and Spec

Dear List,
 
I am new to ROC analysis and the package ROCR. I want to compute the confidence
intervals of sensitivity and specificity for a given cutoff value. I have used
the following to calculate sensitivity and specificity:
 
data(ROCR.simple)
pred <- prediction(ROCR.simple$predictions, ROCR.simple$labels)
 
se.sp <- function (cutoff, performance) { 
    sens <- performance(pred,"sens") 
    spec <- performance(pred,"spec") 
    num.cutoff <- which.min(abs(sens@x.values[[1]] - cutoff))     
return(list(Cutoff=sens@x.values[[1]][num.cutoff],
Sensitivity=sens@y.values[[1]][num.cutoff], Specificity=spec@y.values
[[1]][num.cutoff]))
} 
 
se.sp(cutoff=0.5, pred)

 
I would be grateful if someone could let me know how to get 95% CIs for these
sensitivity and specificity?
 
Thanks vermuchch in advance.
Rasanga
 
 
 


      
	[[alternative HTML version deleted]]

Rasanga Ruwanthi wrote:
> Dear List,
>  
> I am new to ROC analysis and the package ROCR. I want to compute the
confidence intervals of sensitivity and specificity for a given cutoff value. I
have used the following to calculate sensitivity and specificity:
>  
> data(ROCR.simple)
> pred <- prediction(ROCR.simple$predictions, ROCR.simple$labels)
>  
> se.sp <- function (cutoff, performance) { 
>     sens <- performance(pred,"sens") 
>     spec <- performance(pred,"spec") 
>     num.cutoff <- which.min(abs(sens at x.values[[1]] - cutoff))     
> return(list(Cutoff=sens at x.values[[1]][num.cutoff], Sensitivity=sens at
y.values[[1]][num.cutoff], Specificity=spec at y.values [[1]][num.cutoff]))
> } 
>  
> se.sp(cutoff=0.5, pred)
> 
>  
> I would be grateful if someone could let me know how to get 95% CIs for
these sensitivity and specificity?
>  
> Thanks vermuchch in advance.
> Rasanga
The confidence interval will be meaningless unless you pre-specified a 
single cutoff without looking at the dataset.  You will probably have to 
use the bootstrap to penalize for the (severe) instability caused by 
examining multiple cutpoints.  Then you have to ask yourself why a 
cutoff is needed anyway.

Frank
>  
>  
>  
> 
> 
>       
> 	[[alternative HTML version deleted]]
> 
> 
> 
> ------------------------------------------------------------------------
> 
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.

-- 
Frank E Harrell Jr   Professor and Chair           School of Medicine
                      Department of Biostatistics   Vanderbilt University