R help - Oct 2008 - Fw: Logistic regresion - Interpreting (SENS) and (SPEC)

Dear Mr Peter Dalgaard and Mr Dieter Menne,

I sincerely thank you for helping me out with my problem. The thing is taht I
already have calculated SENS = Gg / (Gg + Bg) = 89.97%
and SPEC = Bb / (Bb + Gb) = 74.38%.

Now I have values of SENS and SPEC, which are absolute in nature. My question
was how do I interpret these absolue values. How does these values help me to
find out wheher my model is good.

With regards

Ms Maithili Shiva

________________________________________________________________________





> Subject: [R] Logistic regresion - Interpreting (SENS) and (SPEC)
> To: r-help at r-project.org
> Date: Friday, October 10, 2008, 5:54 AM
> Hi
> 
> Hi I am working on credit scoring model using logistic
> regression. I havd main sample of 42500 clentes and based on
> their status as regards to defaulted / non - defaulted, I
> have genereted the probability of default.
> 
> I have a hold out sample of 5000 clients. I have calculated
> (1) No of correctly classified goods Gg, (2) No of correcly
> classified Bads Bg and also (3) number of wrongly classified
> bads (Gb) and (4) number of wrongly classified goods (Bg).
> 
> My prolem is how to interpret these results? What I have
> arrived at are the absolute figures.
> 
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained,
> reproducible code.

Hi Maithili,

There are two good papers that illustrate how to compare classifiers
using Sensitivity and Specificity and their extensions (e.g., likelihood
ratios, young index, KL distance, etc).

See: 
1) Biggerstaff, Brad, 2000, "Comparing diagnostic tests: a simple
graphic using likelihood ratios," Statistics in Medicine, 19:649-663.

2) Lee, Wen-Chung, 1999, "Selecting diagnostic tests for ruling out or
ruling in disease: the use of the Kllback-Leibler distance,"
International Epidemiological Association, 28:521-525.

Please let me know if have problems finding the aforementioned papers.

Kind Regards,

Pedro


-----Original Message-----
From: r-help-bounces at r-project.org [mailto:r-help-bounces at r-project.org]
On Behalf Of Maithili Shiva
Sent: Monday, October 13, 2008 3:28 AM
To: r-help at r-project.org
Cc: dieter.menne at menne-biomed.de; p.dalgaard at biostat.ku.dk
Subject: [R] Fw: Logistic regresion - Interpreting (SENS) and (SPEC)


Dear Mr Peter Dalgaard and Mr Dieter Menne,

I sincerely thank you for helping me out with my problem. The thing is
taht I already have calculated SENS = Gg / (Gg + Bg) = 89.97%
and SPEC = Bb / (Bb + Gb) = 74.38%.

Now I have values of SENS and SPEC, which are absolute in nature. My
question was how do I interpret these absolue values. How does these
values help me to find out wheher my model is good.

With regards

Ms Maithili Shiva

________________________________________________________________________





> Subject: [R] Logistic regresion - Interpreting (SENS) and (SPEC)
> To: r-help at r-project.org
> Date: Friday, October 10, 2008, 5:54 AM
> Hi
> 
> Hi I am working on credit scoring model using logistic
> regression. I havd main sample of 42500 clentes and based on
> their status as regards to defaulted / non - defaulted, I
> have genereted the probability of default.
> 
> I have a hold out sample of 5000 clients. I have calculated
> (1) No of correctly classified goods Gg, (2) No of correcly
> classified Bads Bg and also (3) number of wrongly classified
> bads (Gb) and (4) number of wrongly classified goods (Bg).
> 
> My prolem is how to interpret these results? What I have
> arrived at are the absolute figures.
> 
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained,
> reproducible code.
______________________________________________
R-help at r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.

<Pedro.Rodriguez <at> sungard.com> writes:
> There are two good papers that illustrate how to compare classifiers
> using Sensitivity and Specificity and their extensions (e.g., likelihood
> ratios, young index, KL distance, etc).
> 
> See: 
> 1) Biggerstaff, Brad, 2000, "Comparing diagnostic tests: a simple
> graphic using likelihood ratios," Statistics in Medicine, 19:649-663.
> 
> 2) Lee, Wen-Chung, 1999, "Selecting diagnostic tests for ruling out or
> ruling in disease: the use of the Kllback-Leibler distance,"
> International Epidemiological Association, 28:521-525.
> 
Both papers refer to medical applications, and even the most basic books on
medical statistics explain the concepts in the context of incidence and
prevalance of a disease. Interpreting sensitivity and specificity is much more a
problem of the context than one of R and statistics: note that her application
was in econometrics.

Dieter

Maithili Shiva wrote:
> Dear Mr Peter Dalgaard and Mr Dieter Menne,
> 
> I sincerely thank you for helping me out with my problem. The thing is taht
I already have calculated SENS = Gg / (Gg + Bg) = 89.97%
> and SPEC = Bb / (Bb + Gb) = 74.38%.
> 
> Now I have values of SENS and SPEC, which are absolute in nature. My
question was how do I interpret these absolue values. How does these values help
me to find out wheher my model is good.
> 
> With regards
> 
> Ms Maithili Shiva
I can't understand why you are interested in probabilities that are in 
backwards time order.

Frank
> 
> ________________________________________________________________________
> 
> 
> 
> 
> 
> 
>> Subject: [R] Logistic regresion - Interpreting (SENS) and (SPEC)
>> To: r-help at r-project.org
>> Date: Friday, October 10, 2008, 5:54 AM
>> Hi
>>
>> Hi I am working on credit scoring model using logistic
>> regression. I havd main sample of 42500 clentes and based on
>> their status as regards to defaulted / non - defaulted, I
>> have genereted the probability of default.
>>
>> I have a hold out sample of 5000 clients. I have calculated
>> (1) No of correctly classified goods Gg, (2) No of correcly
>> classified Bads Bg and also (3) number of wrongly classified
>> bads (Gb) and (4) number of wrongly classified goods (Bg).
>>
>> My prolem is how to interpret these results? What I have
>> arrived at are the absolute figures.
>>
>> ______________________________________________
>> R-help at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide
>> http://www.R-project.org/posting-guide.html
>> and provide commented, minimal, self-contained,
>> reproducible code.
> 
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
> 

-- 
Frank E Harrell Jr   Professor and Chair           School of Medicine
                      Department of Biostatistics   Vanderbilt University

I recall a concept of Snout:  sensitivity that is high enough to essentially
rule out the presence of disease.  And Spin:  specificity that is high enough to
essentially rule in the presence of disease.

So perhaps the below is backwards?  The higher the sensitivity, the greater the
NPV?  And the higher the specificity, the greater the PPV?

http://www.musc.edu/dc/icrebm/diagnostictests.html

--Chris Ryan

---- Original message ----
>Date: Mon, 13 Oct 2008 18:14:39 -0400
>From: "John Sorkin" <jsorkin at grecc.umaryland.edu>  
>Subject: Re: [R] Fw:  Logistic regresion - Interpreting	(SENS)	and	(SPEC)  
>To: "Ph.D. Robert W. Baer" <rbaer at atsu.edu>, "Frank
E Harrell Jr" <f.harrell at vanderbilt.edu>
>Cc: r-help at r-project.org, dieter.menne at menne-biomed.de, p.dalgaard at
biostat.ku.dk
. . . . .
>Further, PPV is a function of sensitivity (for a given specificity in a
population with a given disease prevalence), the higher the sensitivity almost
always the greater the PPV (it can by unchanged, but I don't believe it can
be lower) and as
>              NPV is a function of specificity (for a given sensitivity in a
population with a given disease prevelance), the higher the specificity almost
always the greater the NPV (it can by unchanged, but I don't believe it can
be lower) . . . .