R help - Jan 2011 - Memory issues

Hi,

I have read several threads about memory issues in R and I can't seem to
find a solution to my problem.

I am running a sort of LASSO regression on several subsets of a big dataset.
For some subsets it works well, and for some bigger subsets it does not
work, with errors of type "cannot allocate vector of size 1.6Gb". The
error
occurs at this line of the code:

   example <- cv.glmnet(x=bigmatrix, y=price, nfolds=3)


It also depends on the number of variables that were included in
"bigmatrix".


I tried on R and R64 for both Mac and R for PC but recently went onto a
faster virtual machine on Linux thinking I would avoid any memory issues. It
was better but still had some limits, even though memory.limit indicates
"Inf".


Is there anyway to make this work or do I have to cut a few variables in the
matrix or take a smaller subset of data ?

I have read that R is looking for some contiguous bits of memory and that
maybe I should pre-allocate the matrix ? Any idea ?


Many thanks

Emmanuel

	[[alternative HTML version deleted]]

Hi Emmanuel,

Try the following:

1) removing unnecessary programs from memory, this might give u a larger
contiguous memory block for R
2) remove unnecessary data from R's memory, so many of the preceding data
sets U no longer need can be removed. use the rm() command. U might need
to run gc() after this to insure the new memory is available
3) make sure U've assigned as much memory to R as possible using
memory.size()




And make sure u have r's

Chris Howden
Founding Partner
Tricky Solutions
Tricky Solutions 4 Tricky Problems
Evidence Based Strategic Development, IP Commercialisation and Innovation,
Data Analysis, Modelling, and Training
(mobile) 0410 689 945
chris at trickysolutions.com.au

-----Original Message-----
From: r-help-bounces at r-project.org [mailto:r-help-bounces at r-project.org]
On Behalf Of Emmanuel Bellity
Sent: Monday, 17 January 2011 4:53 AM
To: r-help at r-project.org
Subject: [R] Memory issues

Hi,

I have read several threads about memory issues in R and I can't seem to
find a solution to my problem.

I am running a sort of LASSO regression on several subsets of a big
dataset.
For some subsets it works well, and for some bigger subsets it does not
work, with errors of type "cannot allocate vector of size 1.6Gb". The
error
occurs at this line of the code:

   example <- cv.glmnet(x=bigmatrix, y=price, nfolds=3)


It also depends on the number of variables that were included in
"bigmatrix".


I tried on R and R64 for both Mac and R for PC but recently went onto a
faster virtual machine on Linux thinking I would avoid any memory issues.
It
was better but still had some limits, even though memory.limit indicates
"Inf".


Is there anyway to make this work or do I have to cut a few variables in
the
matrix or take a smaller subset of data ?

I have read that R is looking for some contiguous bits of memory and that
maybe I should pre-allocate the matrix ? Any idea ?


Many thanks

Emmanuel

	[[alternative HTML version deleted]]

______________________________________________
R-help at r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.