Disclaimer : Short of having local statistical expertise at hand, I'm using
this list because I use R for variable selection in the context of linear
multiple regression but the questions I have relate more to basic statistics
than to R per se. Please redirect me to another appropriate list if such a
list exists.
I have the very common problem of identifying which (subset of) variables
are important in a multiple linear regression problem. Googling and
browsing around, I ended up with four methods I could easily access through
R packages. My aim was to use those methods with continuous variables in a
bid to find out which had an effect on my dependent variable, but more so I
thought this would lead me to identify variables amenable to categorization.
Example : I have population as a continuous variable so I figured if some
variable selection algorithm flagged population as "significant", then
I
could try and apply a priori knowledge of the classes of population ranges I
suspect could show different behaviours in the form of statistically
different means as per ANOVA analysis. Coming back to my original variable
selection need, here's what I did.
First : regular lm
Second : step
Third : all-subsets (regsubsets, package leaps)
Fourth : lasso (l1ce, package lasso2)
Fifth : (I meant to use the lars package, but it does not allow for
formulas; I know I could cast my dataset as matrices, but I didn't find an
easy way of doing this and I figured I had enough options already)
I'm trying to make sense of the information that is sent back at me from the
summary calls. I'm looking for what variables are identified by each
method, hoping to find comparable/complementary results. Given I'm a
transient user of statistics, I find many of the "Details" sections on
the
help files lack specific instructions as to exactly how to interpret the
results. Here we go
*lm*
>
anova(tonlm)
Analysis of Variance Table
Response: Cout.ton
Df Sum Sq Mean Sq F value Pr(>F)
Tonnage 1 9720 9720 1.3497 0.2470437
Popul 1 112361 112361 15.6014 0.0001164 ***
DensiteOcc 1 173350 173350 24.0699 2.245e-06 ***
NbmRues.hab 1 280 280 0.0389 0.8438903
RFU.hab 1 183 183 0.0254 0.8734816
UO.MAMR.Precis 1 67161 67161 9.3254 0.0026428 **
t.CS.t.déchets.MAMR 1 24188 24188 3.3586 0.0686925 .
Pct.Pot.CS.hab 1 78764 78764 10.9365 0.0011614 **
NbCentresTriDs100km 1 218725 218725 30.3702 1.380e-07 ***
DistMarche 1 54114 54114 7.5137 0.0068110 **
Residuals 162 1166717 7202
*ANOVA applied on step(lm) *
Response: Cout.ton
Df Sum Sq Mean Sq F value Pr(>F)
Popul 1 2917 2917 0.4038 0.525990
DensiteOcc 1 163741 163741 22.6730 4.179e-06 ***
RFU.hab 1 83 83 0.0115 0.914899
UO.MAMR.Precis 1 168407 168407 23.3191 3.112e-06 ***
Pct.Pot.CS.hab 1 122685 122685 16.9880 5.943e-05 ***
NbCentresTriDs100km 1 190659 190659 26.4002 7.761e-07 ***
DistMarche 1 65467 65467 9.0652 0.003015 **
Residuals 165 1191606 7222
Questions compared to lm above : What tells me which variable was selected
first in the stepwise process ? Do I sort Pr(>F), the lowest value of which
corresponds to the first variable ? Popul has a 3 star rating in lm and
nothing in step. How do I interpret that ?
* all-subsets regression *
> summary(tonall)$cp
[1] 37.338138 31.452375 25.300272 17.965950 13.751043 10.021910 8.455827
8.810498 9.273599 11.000000> summary(tonall)$adjr2
[1] 0.2155976 0.2411378 0.2680048 0.2997661 0.3197652 0.3381030 0.3481410
0.3506880 0.3528339 0.3499369
> summary(tonall)$which[1,][summary(tonall)$which[1,]]
(Intercept) DistMarche > summary(tonall)$which[2,][summary(tonall)$which[2,]]
(Intercept), NbCentresTriDs100km, DistMarche > summary(tonall)$which[3,][summary(tonall)$which[3,]]
(Intercept), Tonnage, UO.MAMR.Precis, NbCentresTriDs100km
> summary(tonall)$which[4,][summary(tonall)$which[4,]]
(Intercept), Tonnage, DensiteOcc, UO.MAMR.Precis, NbCentresTriDs100km
> summary(tonall)$which[5,][summary(tonall)$which[5,]]
(Intercept), Tonnage, DensiteOcc, UO.MAMR.Precis NbCentresTriDs100km,
DistMarche
omitting the remaining values up to which[10,]
Questions w/r to lm and step : all-subsets says DistMarche is the single
most important. That makes some sense as that variable had a two-star
rating in both lm and step. But shouldn't the 3-star ratings in step be
close to those in all-subsets ? For example, the best 3-variable model
shows Tonnage popping up. Tonnage has no rating in lm and doesn't even show
in step. Is it a matter of step being initialized with a variable such that
Tonnage will never be considered whereas it is in an exhaustive all-subsets
regression ? I'm puzzled.
* lasso *
> summary(tonlasso)
...
Coefficients:
Value Std. Error Z score Pr(>|Z|)
(Intercept) 234.5910 36.0120 6.5142 0.0000
Tonnage -0.0155 0.0065 -2.3736 0.0176
Popul -0.0001 0.0008 -0.1639 0.8698
DensiteOcc -0.0719 0.0201 -3.5732 0.0004
NbmRues.hab -0.0130 0.0110 -1.1820 0.2372
RFU.hab 0.0005 0.0002 2.6586 0.0078
UO.MAMR.Precis 0.0025 0.0013 1.9025 0.0571
t.CS.t.déchets.MAMR -25.9682 51.2912 -0.5063 0.6127
Pct.Pot.CS.hab -69.0914 35.9212 -1.9234 0.0544
NbCentresTriDs100km -5.9783 2.3811 -2.5108 0.0120
DistMarche 0.2210 0.0805 2.7461 0.0060
I read that LASSO effectively allows for variable selection in the form of
coefficients being set to 0. With the figures above, can I fix Values < 0.1
or 0.01 as 0, which would eliminate a number of variates ? Does a
decreasing order of (absolute) coefficient values amount to determining the
order of selection of variables according to LASSO ?
Thank you for your patience and for pointers.
Yves Moisan
--
View this message in context:
http://www.nabble.com/Variable-selection-in-R-tf4556775.html#a13004728
Sent from the R help mailing list archive at Nabble.com.
[[alternative HTML version deleted]]
