[R] Memory problem on a linux cluster using a large data set

Iris Kolder
Mon, 18 Dec 2006 10:19:48 -0000

Hello,

I have a large data set 320.000 rows and 1000 columns. All the data has the
values 0,1,2.
I wrote a script to remove all the rows with more than 46 missing values. This
works perfect on a smaller dataset. But the problem arises when I try to run it
on the larger data set I get an error “cannot allocate vector size 1240 kb”.
I’ve searched through previous posts and found out that it might be because i’m
running it on a linux cluster with R version R 2.1.0.  which operates on a 32
bit processor. But I could not find a solution for this problem. The cluster is
a really fast one and should be able to cope with these large amounts of data
the systems configuration are Speed: 3.4 GHz, memory 4GByte. Is there a way to
change the settings or processor under R? I want to run the function Random
Forest on my large data set it should be able to cope with that amount. Perhaps
someone has tried this before in R or is Fortram a better choice? I added my R
script down below.

Best regards,

Iris Kolder

SNP <- read.table("file.txt", header=FALSE, sep="")    #
read in data file
SNP[SNP==9]<-NA                                   # change missing values
from a 9 to a NA
SNP$total.NAs = rowSums(is.na(SN         # calculate the number of NA per row
and adds a colum with total Na's
SNP = SNP[ SNP$total.NAs < 46,  ]         # create a subset with no more than
5%(46) NA's
SNP$total.NAs=NULL                              # remove added column with sum
of NA's
SNP  = t(as.matrix(SNP))                          # transpose rows and columns
set.seed(1)
snp.na<-SNP
snp.roughfix<-na.roughfix(snp.na)
fSNP<-factor(snp.roughfix[, 1])                # Asigns factor to case
control status

snp.narf<- randomForest(snp.roughfix[,-1], fSNP, na.action=na.roughfix,
ntree=500, mtry=10, importance=TRUE, keep.forest=FALSE, do.trace=100)

print(snp.narf)

__________________________________________________



	[[alternative HTML version deleted]]