R help - Nov 2003 - RandomForest & memory demand

Hi,

is it correct that i need  ~ 2GB RAM that it's
possible to work with the default setting 
ntree=500 and a data.frame with 100.000 rows 
and max. 10 columns for training and testing?

P.S.
It's possible calculate approximate the
memory demand for different settings with RF?

Many thanks & regards,
Christian

> Hi,
>
> is it correct that i need  ~ 2GB RAM that it's
> possible to work with the default setting
> ntree=500 and a data.frame with 100.000 rows
> and max. 10 columns for training and testing?
>
no. You may parallelize the computations: perform 5 runs of RF with `ntree
= 100' (or less) and save the resulting RF-objects into files.

For prediction, calculate the prediction of each of the 5 objects and
aggregate them. This requires some simple lines of code but will help to
circumvent RAM restrictions.

Best,

Torsten
> P.S.
> It's possible calculate approximate the
> memory demand for different settings with RF?
>
> Many thanks & regards,
> Christian
>
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://www.stat.math.ethz.ch/mailman/listinfo/r-help
>
>

> From: Christian Schulz
> 
> Hi,
> 
> is it correct that i need  ~ 2GB RAM that it's
> possible to work with the default setting 
> ntree=500 and a data.frame with 100.000 rows 
> and max. 10 columns for training and testing?
If you have the test set, and don't need the forest for predicting other
data, you can give both training data and test data to randomForest() at the
same time (if that fits in memory).  This way there will only be one tree
kept in memory.  E.g., you would do something like:

my.result <- randomForest(x, y, xtest)

Then my.result$test will contain a list of results on the test set.  If you
also give ytest, there will be a bit more output.

If you follow Torsten's suggestion, you can use the combine() function to
merge the five forests into one.
 
> P.S.
> It's possible calculate approximate the
> memory demand for different settings with RF?
The current implementation of the code requires (assuming classification, no
test data, and proximity=FALSE) approximately:

At R level:
- One copy of the training data.
- 6*(2n+1)*ntree integers for storing the forest.

At C level (dynamically allocated):
- (2n + 37)*nclass + 9*n + p*(2+nclass) doubles.
- 5 + (3*p + 22)*n + 5*(p + nclass) integers.

(nclass is the number of classes, n the number of cases in training data, p
the number of variables.)

HTH,
Andy
 
> Many thanks & regards,
> Christian