R help - Nov 2002 - binomial glm for relevant feature selection?

As suggested in my earlier message, I have a large population of 
independent variables and a binary dependent outcome.  It is expected 
that only a few of the independent variables actually contribute to the 
outcome, and I'd like to find those.

If it wasn't already obvious, I am *not* a statistician.  Not even 
close.  :-)  Statistician colleagues have suggested that I use logistic 
regression for this problem.  My understanding is that logistic 
regression is available in R as glm(..., family=binomial).

When I use this solver on fictitious data, though, the answers I expect 
are not the answers I see.  Consider the following fictitious data, 
where "z" is the dependent binary outcome, "y" is irrelevant
noise, and
"x" is actually relevant to predicting the outcome:

	  x y z
	1 8 7 1
	2 8 3 1
	3 0 5 0
	4 0 9 0
	5 8 1 1

If I feed this data to glm(z ~ x + y) using the default gaussian family, 
the results make some sense to me.  The estimated coefficient for x is 
positive and the corresponding "Pr(>|t|)" value is tiny
(<2e-16), which
I take to imply a high degree of confidence that larger values of x 
correlate with increased likelihood of z.  Conversely, the estimated 
coefficient for y has a "Pr(>|t|)" value of 0.552, which I take to
imply
that there is no strong correlation between y and z.  Good.

However, I've been told that I want to use family=binomial for a 
logistic regression problem with a binary dependent outcome like this. 
If I give this data to glm(z ~ x + y, family=binomial), the results 
become quite mysterious.  I receive a warning that "Algorithm did not 
converge".  The "Pr(>|t|)" values for x and y are 0.916 and
1.000
respectively, which would seem to indicate that neither one correlates 
with the outcome.

I realize that this is not a problem with R.  It is a problem with my 
understanding of what R is doing.  But you all have been so helpful thus 
far, perhaps I can impose on you to give me one more clue?  What am I 
doing wrong here?  What should I be looking at that I'm not?

Thank you, once again!

-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !)  To: r-help-request at
stat.math.ethz.ch
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._

On Sun, 10 Nov 2002, Ben Liblit wrote:
> As suggested in my earlier message, I have a large population of
> independent variables and a binary dependent outcome.  It is expected
> that only a few of the independent variables actually contribute to the
> outcome, and I'd like to find those.
>
> If it wasn't already obvious, I am *not* a statistician.  Not even
> close.  :-)  Statistician colleagues have suggested that I use logistic
> regression for this problem.  My understanding is that logistic
> regression is available in R as glm(..., family=binomial).
>
> When I use this solver on fictitious data, though, the answers I expect
> are not the answers I see.  Consider the following fictitious data,
> where "z" is the dependent binary outcome, "y" is
irrelevant noise, and
> "x" is actually relevant to predicting the outcome:
>
> 	  x y z
> 	1 8 7 1
> 	2 8 3 1
> 	3 0 5 0
> 	4 0 9 0
> 	5 8 1 1
>
> If I feed this data to glm(z ~ x + y) using the default gaussian family,
> the results make some sense to me.  The estimated coefficient for x is
> positive and the corresponding "Pr(>|t|)" value is tiny
(<2e-16), which
> I take to imply a high degree of confidence that larger values of x
> correlate with increased likelihood of z.  Conversely, the estimated
> coefficient for y has a "Pr(>|t|)" value of 0.552, which I
take to imply
> that there is no strong correlation between y and z.  Good.
>
> However, I've been told that I want to use family=binomial for a
> logistic regression problem with a binary dependent outcome like this.
> If I give this data to glm(z ~ x + y, family=binomial), the results
> become quite mysterious.  I receive a warning that "Algorithm did not
> converge".  The "Pr(>|t|)" values for x and y are 0.916
and 1.000
> respectively, which would seem to indicate that neither one correlates
> with the outcome.
>
> I realize that this is not a problem with R.  It is a problem with my
> understanding of what R is doing.  But you all have been so helpful thus
> far, perhaps I can impose on you to give me one more clue?  What am I
> doing wrong here?  What should I be looking at that I'm not?
Your problem is linearly separable, and you are seeing the Hauck-Donner
effect.  This is rare (but by no means unknown) in real problems, and
means the Wald test as used by the t values is unreliable.

More details in Venables & Ripley (1999, 2002), look Hauck-Donner up in
the index.  It's a technical point and the explanation is technical, but
there is also a practical summary there.

-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272860 (secr)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595

-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !)  To: r-help-request at
stat.math.ethz.ch
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._