R help - Jan 2010 - Once again: Error: cannot allocate vector of size

Hi,

I have browsed the help list and looked at the FAQ but I don't find
conclusive evidence if this is normal or I am doing something wrong.
I am running a lm() on a data.frame with 27136 observations of 6 variables (3
num and 3 factor).
After a while R throws this:
> lm(log(y) ~ log(a) + log(b) + c + d + e, data=reg.data ,
na.action=na.exclude)
Error: cannot allocate vector of size 203.7 MB

This is a Windows XP 32 bit machine with 4 GB in it so that theoretically, R
should be able to claim close to 2 GB.
This is the gc() after the regression:
          used (Mb) gc trigger  (Mb)  max used   (Mb)
Ncells  272299  7.3     875833  23.4   1368491   36.6
Vcells 4526037 34.6  116536251 889.2 145524997 1110.3
> memory.size(max=T)
[1] 1230.25
> memory.size(max=F)
[1] 47.89
> 
Looking at memory.size, R should be easily able to allocate that space,
shouldn't it?

Many thanks for any hints!

--Werner

__________________________________________________
Do You Yahoo!?
Sie si
n Massenmails. 
http://mail.yahoo.com

Please re-read the posting guide e.g. you didn't provide an example data set
or a way to generate one,  or any R version information.

"Werner W." <pensterfuzzer at yahoo.de> wrote in message 
news:646146.32238.qm at web23002.mail.ird.yahoo.com...
> Hi,
>
> I have browsed the help list and looked at the FAQ but I don't find 
> conclusive evidence if this is normal or I am doing something wrong.
> I am running a lm() on a data.frame with 27136 observations of 6 variables 
> (3 num and 3 factor).
> After a while R throws this:
>
>> lm(log(y) ~ log(a) + log(b) + c + d + e, data=reg.data , 
>> na.action=na.exclude)
> Error: cannot allocate vector of size 203.7 MB
>
> This is a Windows XP 32 bit machine with 4 GB in it so that theoretically, 
> R should be able to claim close to 2 GB.
> This is the gc() after the regression:
>          used (Mb) gc trigger  (Mb)  max used   (Mb)
> Ncells  272299  7.3     875833  23.4   1368491   36.6
> Vcells 4526037 34.6  116536251 889.2 145524997 1110.3
>
>> memory.size(max=T)
> [1] 1230.25
>> memory.size(max=F)
> [1] 47.89
>>
>
> Looking at memory.size, R should be easily able to allocate that space, 
> shouldn't it?
>
> Many thanks for any hints!
>
> --Werner
>
> __________________________________________________
> Do You Yahoo!?
> Sie si
> n Massenmails.
> http://mail.yahoo.com
>

Hi Werner,

On Thu, Jan 21, 2010 at 9:23 PM, Werner W. <pensterfuzzer at yahoo.de>
wrote:
> Hi,
>
> I have browsed the help list and looked at the FAQ but I don't find
conclusive evidence if this is normal or I am doing something wrong.
> I am running a lm() on a data.frame with 27136 observations of 6 variables
(3 num and 3 factor).
> After a while R throws this:
>
>> lm(log(y) ~ log(a) + log(b) + c + d + e, data=reg.data ,
na.action=na.exclude)
> Error: cannot allocate vector of size 203.7 MB
>
> This is a Windows XP 32 bit machine with 4 GB in it so that theoretically,
R should be able to claim close to 2 GB.
> This is the gc() after the regression:
> ? ? ? ? ?used (Mb) gc trigger ?(Mb) ?max used ? (Mb)
> Ncells ?272299 ?7.3 ? ? 875833 ?23.4 ? 1368491 ? 36.6
> Vcells 4526037 34.6 ?116536251 889.2 145524997 1110.3
>
>> memory.size(max=T)
> [1] 1230.25
>> memory.size(max=F)
> [1] 47.89
>>
>
> Looking at memory.size, R should be easily able to allocate that space,
shouldn't it?
AFAIK, the error message "Error: cannot allocate vector of size 203.7
MB" doesn't mean that your entire data set is 203.7MB and it can't
load it ... but rather R just tried grab another chunk of RAM that
size, and the OS has denied it that favor ... so, I'm guessing it's
getting "half way" through loading/running on your data, but can't
complete the task because it has already chewed through all the RAM
that at has access to.

You can look at your RAM usage (through the task manager? sorry,
haven't used windows in a long time) while R is doing its thing to
verify until a windows guru chimes in.

-steve

-- 
Steve Lianoglou
Graduate Student: Computational Systems Biology
 | Memorial Sloan-Kettering Cancer Center
 | Weill Medical College of Cornell University
Contact Info: http://cbio.mskcc.org/~lianos/contact

Thanks Matthew, you are absolutely right.

I am working on Windows XP SP2 32bit with R versions 2.9.1.

Here is an example:
  d <- as.data.frame(matrix(trunc(rnorm(6*27136, 10000, 100)),ncol=6))
  d[,4:5] <- trunc(100*runif(2*27136, 0, 1))
  d[,6] <- trunc(1000*runif(27136, 0, 1))
  for (i in 4:6) d[,i] <- as.factor(d[,i])
  lm(V1 ~ log(V2) + log(V3) + V4 + V5 + V6, data=d)
  memory.size(max=F)
  memory.size(max=T)

I managed to get it run through after setting the 3GB switch for Windows and
with a clean R session.
I also noticed later, that after removing na.action=na.exclude more
regressions run through.

But before and after the lm() it seems there should be enough memory which
means that lm() builds up some quite large objects during its computations?
-- 
View this message in context:
http://n4.nabble.com/Once-again-Error-cannot-allocate-vector-of-size-tp1083506p1100164.html
Sent from the R help mailing list archive at Nabble.com.