Hi! I am working on scorecard model and I have arrived at the regression equation. I have used logistic regression using R. My question is how do I validate this model? I do have hold out sample of 5000 customers. Please guide me. Problem is I had never used Logistic regression earlier neither I am used to credit scoring models. Thanks in advance Maithili
Maithili Shiva wrote:> Hi! > > I am working on scorecard model and I have arrived at the regression equation. I have used logistic regression using R. > > My question is how do I validate this model? I do have hold out sample of 5000 customers. > > Please guide me. Problem is I had never used Logistic regression earlier neither I am used to credit scoring models. > > Thanks in advance > > MaithiliThe holdout sample of 5000 may be too small. You didn't specify the size of the training sample. I assume it is 50,000 or larger. 50-fold repeats of 10-fold cross-validation is better. With any resampling technique you need to repeat all exploratory and modeling steps inside the loop. Frank> > ______________________________________________ > R-help at r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. >-- Frank E Harrell Jr Professor and Chair School of Medicine Department of Biostatistics Vanderbilt University
Usually one validates scorecards with the ROC curve, Pietra Index, KS test, etc. You may be interested in the WP 14 from BIS (www.bis.org). Regards, Pedro -----Original Message----- From: r-help-bounces at r-project.org [mailto:r-help-bounces at r-project.org] On Behalf Of Maithili Shiva Sent: Tuesday, October 07, 2008 8:22 AM To: r-help at r-project.org Subject: [R] How to validate model? Hi! I am working on scorecard model and I have arrived at the regression equation. I have used logistic regression using R. My question is how do I validate this model? I do have hold out sample of 5000 customers. Please guide me. Problem is I had never used Logistic regression earlier neither I am used to credit scoring models. Thanks in advance Maithili ______________________________________________ R-help at r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
generally speaking, there are 2 types of validation, out of sample and out of time. it is hard to tell if 5000 is enough or not. how many 1s and 0s in 5000? i am just curious. how could you be asked to do something that you have no idea? On Tue, Oct 7, 2008 at 8:22 AM, Maithili Shiva <maithili_shiva@yahoo.com>wrote:> Hi! > > I am working on scorecard model and I have arrived at the regression > equation. I have used logistic regression using R. > > My question is how do I validate this model? I do have hold out sample of > 5000 customers. > > Please guide me. Problem is I had never used Logistic regression earlier > neither I am used to credit scoring models. > > Thanks in advance > > Maithili > > ______________________________________________ > R-help@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide > http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. >-- ==============================WenSui Liu Acquisition Risk, Chase Email : wensui.x.liu@chase.com Blog : statcompute.spaces.live.com ============================== [[alternative HTML version deleted]]