R help - Jul 2009 - Hmisc improveProb() function

Dear list,

I am trying to work out how the improveProb() function works and how to
interpret the results, and I have a few questions. I would be grateful if anyone
could shed some light on these.

in the Net Reclassification Improvement section of the output, is the 2P column
the two-sided p-value for the differences in classification? So if a limit is
set at 0.05, then lower values indicate significant differences in the
classification of both models?

is the index value the actual percentage of improvement? Could this be negative
if there was actually no improvement?

This one might be more related to the Pencina et al article and to my
non-statistics background, but how are the # of events moving and nonevents
moving up and down defined? does one have to specify a probability value cutoff
for classification?

Thanks for your help,


Dave



_________________________________________________________________


	[[alternative HTML version deleted]]

David A.G wrote:
> Dear list,
> 
> I am trying to work out how the improveProb() function works and how to
interpret the results, and I have a few questions. I would be grateful if anyone
could shed some light on these.
> 
> in the Net Reclassification Improvement section of the output, is the 2P
column the two-sided p-value for the differences in classification? So if a
limit is set at 0.05, then lower values indicate significant differences in the
classification of both models?
Correct.  But why use an arbitrary level for 'significance'?
> 
> is the index value the actual percentage of improvement? Could this be
negative if there was actually no improvement?
NRI is the difference in two proportions.  You may want to emphasize the 
more continuous measure IDI, and a scatterplot relating the two sets of 
predicted probabilities.  The calculations are clear from typing the 
following at the command prompt:

improveProb
print.improveProb

Frank
> 
> This one might be more related to the Pencina et al article and to my
non-statistics background, but how are the # of events moving and nonevents
moving up and down defined? does one have to specify a probability value cutoff
for classification?
> 
> Thanks for your help,
> 
> 
> Dave
> 
> 

-- 
Frank E Harrell Jr   Professor and Chair           School of Medicine
                      Department of Biostatistics   Vanderbilt University