R help - Jul 2012 - using save() to work with objects that exceed memory capacity

Context:  I'm relatively new to R and am working with very large datasets.

General problem:  If working on a dataset requires that I produce more than
two objects of roughly the size of the dataset, R quickly uses up its
available memory and slows to a virtual halt.

My tentative solution:  To save and remove objects as they're created, and
load them when I need them.  To do this I'm trying to automatically
generate file names derived from these objects, and use these in save().

My specific question to the list:  How do I capture the string that names
an object I want to save, in such a way that I can use it in a function
that calls save()?

For example, suppose I create a matrix and then save it
follows:
> mat<-matrix(1:9,3,3)
> save(mat, file="matfile")
Then I get a file of the kind I'd like: the command
'load("matfile")'
retrieves the correct matrix, with the original name 'mat'.

Further, if I instead save it this way:
> objectname<-"mat"
> save(list=ls(pattern=objectname), file="matfile")
then I get the same positive result.

But now suppose I create a function
> saveobj <- function(objectname,objectfile)
+   {
+     save(list=ls(pattern=objectname),file=objectfile);
+     return()};
Then if I now try to save 'mat' by
> matname<-"mat"
> saveobj(matname,"matfile")
I do not get the same result; namely, the command
'load("mat")' retrieves
no objects.  Why is this?


I'd be grateful for any help on either my specific questions, or
suggestions of a better ways to address the issue of limited memory.

Thanks,
David Romano

	[[alternative HTML version deleted]]

On Sat, Jul 28, 2012 at 10:48 AM, David Romano <romanod at grinnell.edu>
wrote:
> Context:  I'm relatively new to R and am working with very large
datasets.
>
> General problem:  If working on a dataset requires that I produce more than
> two objects of roughly the size of the dataset, R quickly uses up its
> available memory and slows to a virtual halt.
>
> My tentative solution:  To save and remove objects as they're created,
and
> load them when I need them.  To do this I'm trying to automatically
> generate file names derived from these objects, and use these in save().
>
> My specific question to the list:  How do I capture the string that names
> an object I want to save, in such a way that I can use it in a function
> that calls save()?
>
> For example, suppose I create a matrix and then save it follows:
>> mat<-matrix(1:9,3,3)
>> save(mat, file="matfile")
> Then I get a file of the kind I'd like: the command
'load("matfile")'
> retrieves the correct matrix, with the original name 'mat'.
>
> Further, if I instead save it this way:
>> objectname<-"mat"
>> save(list=ls(pattern=objectname), file="matfile")
> then I get the same positive result.
>
> But now suppose I create a function
>> saveobj <- function(objectname,objectfile)
> +   {
> +     save(list=ls(pattern=objectname),file=objectfile);
> +     return()};
> Then if I now try to save 'mat' by
>> matname<-"mat"
>> saveobj(matname,"matfile")
> I do not get the same result; namely, the command
'load("mat")' retrieves
> no objects.  Why is this?
load("matfile") no?

It seems to work for me:

R> x <- matrix(1:9, ncol = 3)
R> saveobj <- function(obj, file){
+ save(list = obj, file = file)
+ }
R> exists("x")
[1] FALSE
R> saveobj("x", "amatrix.rdat")
R> rm(x)
R> load("amatrix.rdat")
R> x
     [,1] [,2] [,3]
[1,]    1    4    7
[2,]    2    5    8
[3,]    3    6    9

Cheers,
Michael
>
>
> I'd be grateful for any help on either my specific questions, or
> suggestions of a better ways to address the issue of limited memory.
>
> Thanks,
> David Romano
>
>         [[alternative HTML version deleted]]
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.