R devel - Oct 2005 - R crashes for large formulas in lm() (PR#8180)

Full_Name: Hallgeir Grinde
Version: 2.1.1
OS: Windows XP
Submission from: (NULL) (144.127.1.1)


While using lm(y~(x*z*c*...*v)^2) R crashes/closes if the numbers of variables
are at least 8.

On Wed, 5 Oct 2005 hallgeir.grinde at elkem.no wrote:
> Full_Name: Hallgeir Grinde
> Version: 2.1.1
> OS: Windows XP
> Submission from: (NULL) (144.127.1.1)
>
>
> While using lm(y~(x*z*c*...*v)^2) R crashes/closes if the numbers of
variables
> are at least 8.
OK, let's try to reproduce that:
> x1 <- runif(1000)
> x2 <- runif(1000)
> x3 <- runif(1000)
> x4 <- runif(1000)
> x5 <- runif(1000)
> x6 <- runif(1000)
> x7 <- runif(1000)
> x8 <- runif(1000)
> y <- rnorm(1000)
> fit <- lm(y~(x1*x2*x3*x4*x5*x6*x7*x8)^2)
No crash, a quite reasonable fit.

Can we please have a reproducible example, as we do ask?

-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595

Prof Brian Ripley wrote:
> OK, let's try to reproduce that:
> 
> 
>>x1 <- runif(1000)
...
>>y <- rnorm(1000)
>>fit <- lm(y~(x1*x2*x3*x4*x5*x6*x7*x8)^2)
> 
> 
> No crash, a quite reasonable fit.
  Add one more:

  > x9 <- runif(1000)

  works with 8:

  > fit <- lm(y~(x1*x2*x3*x4*x5*x6*x7*x8)^2)

  but go 'one over the eight' and:

  > fit <- lm(y~(x1*x2*x3*x4*x5*x6*x7*x8*x9)^2)
  Segmentation fault


platform i686-pc-linux-gnu
arch     i686
os       linux-gnu
system   i686, linux-gnu
status
major    2
minor    1.1
year     2005
month    06
day      20
language R

Baz

Prof Brian Ripley wrote:
> On Wed, 5 Oct 2005 hallgeir.grinde at elkem.no wrote:
> 
> 
>>Full_Name: Hallgeir Grinde
>>Version: 2.1.1
>>OS: Windows XP
>>Submission from: (NULL) (144.127.1.1)
>>
>>
>>While using lm(y~(x*z*c*...*v)^2) R crashes/closes if the numbers of
variables
>>are at least 8.
> 
> 
> OK, let's try to reproduce that:
> 
> 
>>x1 <- runif(1000)
>>x2 <- runif(1000)
>>x3 <- runif(1000)
>>x4 <- runif(1000)
>>x5 <- runif(1000)
>>x6 <- runif(1000)
>>x7 <- runif(1000)
>>x8 <- runif(1000)
>>y <- rnorm(1000)
>>fit <- lm(y~(x1*x2*x3*x4*x5*x6*x7*x8)^2)
> 
> 
> No crash, a quite reasonable fit.
> 
> Can we please have a reproducible example, as we do ask?
> 
Hmm, crashes for me as well with R-2.1.1 and R-2.2.0 beta (2005-09-27 
r35682M) on WinNT 4.0, SP6.


Let's make it reproducible:

set.seed(123)
x1 <- runif(1000)
x2 <- runif(1000)
x3 <- runif(1000)
x4 <- runif(1000)
x5 <- runif(1000)
x6 <- runif(1000)
x7 <- runif(1000)
x8 <- runif(1000)
y <- rnorm(1000)
fit <- lm(y~(x1*x2*x3*x4*x5*x6*x7*x8)^2)


Uwe Ligges

> set.seed(123)
> x1 <- runif(1000)
> x2 <- runif(1000)
> x3 <- runif(1000)
> x4 <- runif(1000)
> x5 <- runif(1000)
> x6 <- runif(1000)
> x7 <- runif(1000)
> x8 <- runif(1000)
> y <- rnorm(1000)
> fit <- lm(y~(x1*x2*x3*x4*x5*x6*x7*x8)^2)
  For me (with 9 variables) this crashes in model.matrix:

  > f1=y~(x1*x2*x3*x4*x5*x6*x7*x8*x9)^2
  > model.matrix(f1)
  Segmentation fault

  I guess someone with a debug-compiled R can get a bit further...

Baz

On Wed, 2005-10-05 at 14:12 +0200, ligges at statistik.uni-dortmund.de
wrote:
> Prof Brian Ripley wrote:
> 
> > On Wed, 5 Oct 2005 hallgeir.grinde at elkem.no wrote:
> > 
> > 
> >>Full_Name: Hallgeir Grinde
> >>Version: 2.1.1
> >>OS: Windows XP
> >>Submission from: (NULL) (144.127.1.1)
> >>
> >>
> >>While using lm(y~(x*z*c*...*v)^2) R crashes/closes if the numbers
of variables
> >>are at least 8.
> > 
> > 
> > OK, let's try to reproduce that:
> > 
> > 
> >>x1 <- runif(1000)
> >>x2 <- runif(1000)
> >>x3 <- runif(1000)
> >>x4 <- runif(1000)
> >>x5 <- runif(1000)
> >>x6 <- runif(1000)
> >>x7 <- runif(1000)
> >>x8 <- runif(1000)
> >>y <- rnorm(1000)
> >>fit <- lm(y~(x1*x2*x3*x4*x5*x6*x7*x8)^2)
> > 
> > 
> > No crash, a quite reasonable fit.
> > 
> > Can we please have a reproducible example, as we do ask?
> > 
> 
> Hmm, crashes for me as well with R-2.1.1 and R-2.2.0 beta (2005-09-27 
> r35682M) on WinNT 4.0, SP6.
> 
> 
> Let's make it reproducible:
> 
> set.seed(123)
> x1 <- runif(1000)
> x2 <- runif(1000)
> x3 <- runif(1000)
> x4 <- runif(1000)
> x5 <- runif(1000)
> x6 <- runif(1000)
> x7 <- runif(1000)
> x8 <- runif(1000)
> y <- rnorm(1000)
> fit <- lm(y~(x1*x2*x3*x4*x5*x6*x7*x8)^2)

No problems on FC4 using Version 2.1.1 Patched (2005-10-04):
> summary(fit)
Call:
lm(formula = y ~ (x1 * x2 * x3 * x4 * x5 * x6 * x7 * x8)^2)

Residuals:
      Min        1Q    Median        3Q       Max
-3.315028 -0.532338  0.005368  0.548012  2.957057

Coefficients:
                          Estimate Std. Error t value Pr(>|t|)
(Intercept)                16.2460    15.7375   1.032   0.3023
x1                        -10.0310    29.7557  -0.337   0.7361
x2                        -26.2678    24.0473  -1.092   0.2750
x3                        -42.5713    32.9481  -1.292   0.1967

...

x1:x2:x4:x5:x6:x7:x8     -339.8099  1024.8652  -0.332   0.7403
x1:x3:x4:x5:x6:x7:x8     -833.5375  1095.1153  -0.761   0.4468
x2:x3:x4:x5:x6:x7:x8     -546.0684  1050.8648  -0.520   0.6035
x1:x2:x3:x4:x5:x6:x7:x8   533.8390  1861.5862   0.287   0.7744
---
Signif. codes:  0 ???***??? 0.001 ???**??? 0.01 ???*??? 0.05 ???.??? 0.1 ??? ???
1

Residual standard error: 1.001 on 744 degrees of freedom
Multiple R-Squared: 0.2541,     Adjusted R-squared: -0.001499
F-statistic: 0.9941 on 255 and 744 DF,  p-value: 0.5161


Is this perhaps a RAM issue? I have 2 Gb.

Marc

Have you considered that 8 variables will give you
sum(choose(8,1:8))
[1] 255
terms.
That's a suspicious number.  Have you checked whether there is a limit in
the code?

-------------------------
Full_Name: Hallgeir Grinde
Version: 2.1.1
OS: Windows XP
Submission from: (NULL) (144.127.1.1)


While using lm(y~(x*z*c*...*v)^2) R crashes/closes if the numbers of
variables
are at least 8.


-- 
Ken Knoblauch
Inserm U371
Cerveau et Vision
Dept. of Cognitive Neuroscience
18 avenue du Doyen L?pine
69500 Bron
France
tel: +33 (0)4 72 91 34 77
fax: +33 (0)4 72 91 34 61
portable: +33 (0)6 84 10 64 10
http://www.lyon.inserm.fr/371/

Oops, sorry.  I didn't notice the ^2...
Nevermind.


________________
Have you considered that 8 variables will give you
sum(choose(8,1:8))
[1] 255
terms.
That's a suspicious number.  Have you checked whether there is a limit in
the code?

-------------------------
Full_Name: Hallgeir Grinde
Version: 2.1.1
OS: Windows XP
Submission from: (NULL) (144.127.1.1)


While using lm(y~(x*z*c*...*v)^2) R crashes/closes if the numbers of
variables
are at least 8.
-- 
Ken Knoblauch
Inserm U371
Cerveau et Vision
Dept. of Cognitive Neuroscience
18 avenue du Doyen L?pine
69500 Bron
France
tel: +33 (0)4 72 91 34 77
fax: +33 (0)4 72 91 34 61
portable: +33 (0)6 84 10 64 10
http://www.lyon.inserm.fr/371/

On Thu, 6 Oct 2005, Ei-ji Nakama wrote:
> Hi,
>
> Wed, 5 Oct 2005 13:38:12 +0100 (BST)
> Prof Brian Ripley <ripley at stats.ox.ac.uk> wrote.
>> This is almost certainly C stack overflow, which will be _very_
>> OS-specific. I can run this 9-var example in Linux with a stacksize of
>> 30Mb, but not the default 10Mb. I tried it with the default stack under
>> valgrind, and it works.
>>
>> Windows versions of R have I believe a stack of 8Mb, and so I suspect
are
>> running close to that with 8-var example.
>>
>> The problem appears to be recursion in StripTerm.
>
> default of stack of ld for Mingw seems to be 2M.
Yes thanks I have since found that (from ld --help)
> see. binutils-2.16.91-20050827-1/ld/emultempl/pe.em
>   289    D(SizeOfStackReserve,"__size_of_stack_reserve__",
0x200000),
>
> How about adding the following to MkRules?
> MINGW32LDFLAG=-Wl,--stack=33554432
I have been trying that but I cannot make it work (checking the header).
It seems the way that does work is -Wl,--stack=0xA00000.  That does allow 
this example (with 8 vars) to always run on WinXP.

-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595