R help - Dec 2008 - cannot allocate vector of size... restructuring suggestion please...

Dear R Users,

I was running some data analysis scripts and ran into this error:

Error: cannot allocate vector of size 27.6 Mb

Doing a "memory.size(max=TRUE)" will give me:
[1] 1506.812

The current situation is:
I'm working on a Windows Vista 32bit laptop with 4GB RAM (effectively
3GB I assume...)
I have a data file of 450Mb loaded into R and have around 1500
data.frames floating in the global space as my data source.
The way I run this analysis:
I call a patch processing & procedure script
>> it retrieves 4 lists of info (each around 400x100) from an index
data.frame, and then it calls another script to retrieve info from the
corresponding data.frames on the 4 lists in the global space
>> through calling another script, about 1000x3 will be retrieved by
another script
>> the 1000x3 will be passed to a third script expanded to 20001x3, and
only 20001x1 will be used
>> 20001x1 will accumulate into a matrix of up to 20001x1500 (number of
data frames / trials), say I have to divide the trials into 2 groups and do a
comparison, then that's processing of 2 matrices of size 20001x750
But the allocation error stopped the script after script has processed
around 280 data frames, i.e. made the first matrix up to 20001x280...

I know running the analysis should possibly be achieved by
restructuring my script a little bit, but I have no idea where to
start with to try...
Also, I have no idea about Garbage Collection ability or memory
recycle / reuse ability in R and I think some memory may have been
lost in the middle of the process, and it may be possible to put them
back to the system for R to make use of...
Please advise me to let me to find out the most efficient way of
eliminating the error...
Thanks so much!

Regards,


      John

Also,each data.frame of the 1500 working as data sources floating in
the global environment is of a size ranging from 2000x36 to 9000x36
Please help...! THANKS!!!

- John

On Mon, Dec 15, 2008 at 1:12 PM, tsunhin wong <thjwong at gmail.com>
wrote:
> Dear R Users,
>
> I was running some data analysis scripts and ran into this error:
>
> Error: cannot allocate vector of size 27.6 Mb
>
> Doing a "memory.size(max=TRUE)" will give me:
> [1] 1506.812
>
> The current situation is:
> I'm working on a Windows Vista 32bit laptop with 4GB RAM (effectively
> 3GB I assume...)
> I have a data file of 450Mb loaded into R and have around 1500
> data.frames floating in the global space as my data source.
> The way I run this analysis:
> I call a patch processing & procedure script
>>> it retrieves 4 lists of info (each around 400x100) from an index
data.frame, and then it calls another script to retrieve info from the
corresponding data.frames on the 4 lists in the global space
>>> through calling another script, about 1000x3 will be retrieved by
another script
>>> the 1000x3 will be passed to a third script expanded to 20001x3,
and only 20001x1 will be used
>>> 20001x1 will accumulate into a matrix of up to 20001x1500 (number
of data frames / trials), say I have to divide the trials into 2 groups and do a
comparison, then that's processing of 2 matrices of size 20001x750
> But the allocation error stopped the script after script has processed
> around 280 data frames, i.e. made the first matrix up to 20001x280...
>
> I know running the analysis should possibly be achieved by
> restructuring my script a little bit, but I have no idea where to
> start with to try...
> Also, I have no idea about Garbage Collection ability or memory
> recycle / reuse ability in R and I think some memory may have been
> lost in the middle of the process, and it may be possible to put them
> back to the system for R to make use of...
> Please advise me to let me to find out the most efficient way of
> eliminating the error...
> Thanks so much!
>
> Regards,
>
>
>      John
>

On Mon, 15 Dec 2008, tsunhin wong wrote:
> Dear R Users,
>
> I was running some data analysis scripts and ran into this error:
>
> Error: cannot allocate vector of size 27.6 Mb
>
> Doing a "memory.size(max=TRUE)" will give me:
> [1] 1506.812
>
> The current situation is:
> I'm working on a Windows Vista 32bit laptop with 4GB RAM (effectively
> 3GB I assume...)
Why do you assume?  Reading the rw-FAQ should make clear that a 2GB 
address limit is a lot more plausible given the data you presented.
> I have a data file of 450Mb loaded into R and have around 1500
> data.frames floating in the global space as my data source.
> The way I run this analysis:
> I call a patch processing & procedure script
'batch'?
>>> it retrieves 4 lists of info (each around 400x100) from an index
data.frame, and then it calls another script to retrieve info from the
corresponding data.frames on the 4 lists in the global space
>>> through calling another script, about 1000x3 will be retrieved by
another script
>>> the 1000x3 will be passed to a third script expanded to 20001x3,
and only 20001x1 will be used
>>> 20001x1 will accumulate into a matrix of up to 20001x1500 (number
of data frames / trials), say I have to divide the trials into 2 groups and do a
comparison, then that's processing of 2 matrices of size 20001x750
> But the allocation error stopped the script after script has processed
> around 280 data frames, i.e. made the first matrix up to 20001x280...
>
> I know running the analysis should possibly be achieved by
> restructuring my script a little bit, but I have no idea where to
> start with to try...
Run the script on each 100 data frames in turn.  You can even use R to set 
up scripts for Rscript to run.
> Also, I have no idea about Garbage Collection ability or memory
> recycle / reuse ability in R and I think some memory may have been
> lost in the middle of the process, and it may be possible to put them
> back to the system for R to make use of...
Well, do read the help page ?"Memory-limits' as well as the appropriate
FAQ.  (And also the parts of the posting guide about doing your homework 
before posting.  Since this list is a public record, we do need to correct 
false claims for the sake of future enquirers doing their homework before 
posting.)
> Please advise me to let me to find out the most efficient way of
> eliminating the error...
> Thanks so much!
>
> Regards,
>
>
>      John
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>
-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595