R help - Oct 2005 - Memory problems with large dataset in rpart

Dear helpers,

I am a Dutch student from the Erasmus University. For my Bachelor thesis I
have written a script in R using boosting by means of classification and
regression trees. This script uses the function the predefined function
rpart. My input file consists of about 4000 vectors each having 2210
dimensions. In the third iteration R complains of a lack of memory,
although in each iteration every variable is removed from the memory. Thus
the first two iterations run without any problems.

My computer runs on Windows XP and has 1 gigabye of internal memory.
I tried R using more memory by refiguring the swap files as memtioned in
the FAQ (/3gb), but I didn't succeed in making this work.
The command round(memory.limit()/1048576.0, 2) gives 1023.48

If such an increase of memory can not succeed, perhaps the size of the
rpart object could be reduced by not storing unnecessary information.
The rpart function call is (the calls of FALSE is to try to reduce the
size of the fit object):
fit <- rpart(price ~ ., data = trainingset,
control=rpart.control(maxdepth=2,cp=0.001),model=FALSE,x=FALSE,y=FALSE)

This fit object is later called in 2 predict functions, for example:
predict(fit,newdata=sample)

Can anybody please help me by letting R use more memory (for example swap)
or can anybody help me reducing the size of the fit object?

Kind regards
Dennis van Heijst
Student Informatics & Economics
Erasmus University Rotterdam
The Netherlands

Looks like you have missed the section in the rw-FAQ entitled

     2.9 There seems to be a limit on the memory it uses!

and not set --max-mem-size (which defaults to 1Gb on your system).

However, it looks like your problem is memory fragmentation, and trying to 
run 1Gb tasks in a 2Gb address space is intrinsically a problem to which 
the only solution is a 64-bit version of R.

BTW, /3GB is nothing whatsoever to do with `swap files': if both OS and 
application are configured correctly it increases the user address space 
*for that process* to /3GB (whereas swap space is shared between 
processes).

On Tue, 18 Oct 2005 vheijst at few.eur.nl wrote:
> Dear helpers,
>
> I am a Dutch student from the Erasmus University. For my Bachelor thesis I
> have written a script in R using boosting by means of classification and
> regression trees. This script uses the function the predefined function
> rpart. My input file consists of about 4000 vectors each having 2210
> dimensions. In the third iteration R complains of a lack of memory,
> although in each iteration every variable is removed from the memory. Thus
> the first two iterations run without any problems.
>
> My computer runs on Windows XP and has 1 gigabye of internal memory.
> I tried R using more memory by refiguring the swap files as memtioned in
> the FAQ (/3gb), but I didn't succeed in making this work.
> The command round(memory.limit()/1048576.0, 2) gives 1023.48
>
> If such an increase of memory can not succeed, perhaps the size of the
> rpart object could be reduced by not storing unnecessary information.
> The rpart function call is (the calls of FALSE is to try to reduce the
> size of the fit object):
> fit <- rpart(price ~ ., data = trainingset,
> control=rpart.control(maxdepth=2,cp=0.001),model=FALSE,x=FALSE,y=FALSE)
>
> This fit object is later called in 2 predict functions, for example:
> predict(fit,newdata=sample)
>
> Can anybody please help me by letting R use more memory (for example swap)
> or can anybody help me reducing the size of the fit object?
-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595