R help - Aug 2009 - Best performance measure?

Hello,

I working on a model to predict probabilities.

I don't really care about binary prediction accuracy.

I do really care about the accuracy of my probability predictions.

Frank was nice enough to point me to the val.prob function from the 
Design library.  It looks very promising for my needs.

I've put together some tests and run the val.prob analysis.  It produces 
some very informative graphs along with a bunch of performance measures.

Unfortunately, I'm not sure which measure, if any, is the "best"
one.
I'm comparing hundreds of different models/parameter combinations/etc.  
So Ideally I'd like a single value or two as the "performance
measure"
for each one.  That way I can pick the "best"  model from all my 
experiments.

As mentioned above, I'm mainly interested in the accuracy of my 
probability predictions.

Does anyone have an opinion about which measure I should look at??
(I see Dxy, C, R2, D, U, Briar, Emax, Eavg, etc.)

Thanks!!

-N

Noah Silverman wrote:
> Hello,
> 
> I working on a model to predict probabilities.
> 
> I don't really care about binary prediction accuracy.
> 
> I do really care about the accuracy of my probability predictions.
> 
> Frank was nice enough to point me to the val.prob function from the 
> Design library.  It looks very promising for my needs.
> 
> I've put together some tests and run the val.prob analysis.  It
produces
> some very informative graphs along with a bunch of performance measures.
> 
> Unfortunately, I'm not sure which measure, if any, is the
"best" one.
> I'm comparing hundreds of different models/parameter combinations/etc.
> So Ideally I'd like a single value or two as the "performance
measure"
> for each one.  That way I can pick the "best"  model from all my 
> experiments.
> 
> As mentioned above, I'm mainly interested in the accuracy of my 
> probability predictions.
> 
> Does anyone have an opinion about which measure I should look at??
> (I see Dxy, C, R2, D, U, Briar, Emax, Eavg, etc.)
> 
> Thanks!!
> 
> -N
It all depends on the goal, i.e., the relative value you place on 
absolute accuracy vs. discrimination ability. The Brier score combines 
both and other than interpretability has many advantages.

Frank
> 
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide 
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
> 

-- 
Frank E Harrell Jr   Professor and Chair           School of Medicine
                      Department of Biostatistics   Vanderbilt University