R help - Nov 2011 - how "big" (in RAM and/or disk storage) is each of these objects in a list?

Greetings, friends (and others :) )

We generated a bunch of results and saved them in an RData file. We
can open, use, all is well, except that the size of the saved file is
quite a bit larger than we expected.  I suspect there's something
floating about in there that one of the packages we are using puts in,
such as a spare copy of a data frame that is saved in some subtle way
that has escaped my attention.

Consider a list of objects. Are there ways to do these things:

1. ask R how much memory is used by the things inside the list?

2.   Does "as.expression(anObject)" print everything in there? Or, is
there a better way to convert each thing to text or some other format
that you can actually read line by line to see what is in there, to
"see" everything?

If there's no giant hidden data frame floating about, I figure I'll
have to convert symmetric matrices to lower triangles or such to save
space.  Unless R already is automatically saving a matrix in that way
but just showing me the full matrix, which I suppose is possible. If
you have other ideas about general ways to make saved objects smaller,
I'm open for suggestions.

-- 
Paul E. Johnson
Professor, Political Science
1541 Lilac Lane, Room 504
University of Kansas

On 11-11-26 1:41 PM, Paul Johnson wrote:
 > We generated a bunch of results and saved them in an RData file. We
 > can open, use, all is well, except that the size of the saved file is
 > quite a bit larger than we expected.  I suspect there's something
 > floating about in there that one of the packages we are using puts in,
 > such as a spare copy of a data frame that is saved in some subtle way
 > that has escaped my attention.
 >
 > Consider a list of objects. Are there ways to do these things:
 >
 > 1. ask R how much memory is used by the things inside the list?

You can use object.size, but read the man page:  it is not a completely 
well-defined question.


 >
 > 2.   Does "as.expression(anObject)" print everything in there?
Or, is
 > there a better way to convert each thing to text or some other format
 > that you can actually read line by line to see what is in there, to
 > "see" everything?

No, as.expression won't necessarily work.  save(..., ascii=TRUE) will 
show you everything, but it's not designed to be readable.  Probably the 
most useful function is str().


 > If there's no giant hidden data frame floating about, I figure
I'll
 > have to convert symmetric matrices to lower triangles or such to save
 > space.  Unless R already is automatically saving a matrix in that way
 > but just showing me the full matrix, which I suppose is possible. If
 > you have other ideas about general ways to make saved objects smaller,
 > I'm open for suggestions.

You could try different compression methods (see ?save), but probably 
the best idea is to identify the things that you didn't mean to include, 
and don't include those.  A common way this happens is objects like 
functions or formulas that carry their environment with them.

Duncan Murdoch

On Sat, 26 Nov 2011 12:41:08 -0600
Paul Johnson <pauljohn32 at gmail.com> wrote:
> Greetings, friends (and others :) )
> 
> We generated a bunch of results and saved them in an RData file. We
> can open, use, all is well, except that the size of the saved file is
> quite a bit larger than we expected.  I suspect there's something
> floating about in there that one of the packages we are using puts in,
> such as a spare copy of a data frame that is saved in some subtle way
> that has escaped my attention.
> 
> Consider a list of objects. Are there ways to do these things:
> 
> 1. ask R how much memory is used by the things inside the list?
> 
> 2.   Does "as.expression(anObject)" print everything in there?
Or, is
> there a better way to convert each thing to text or some other format
> that you can actually read line by line to see what is in there, to
> "see" everything?
> 
> If there's no giant hidden data frame floating about, I figure I'll
> have to convert symmetric matrices to lower triangles or such to save
> space.  Unless R already is automatically saving a matrix in that way
> but just showing me the full matrix, which I suppose is possible. If
> you have other ideas about general ways to make saved objects smaller,
> I'm open for suggestions.
> 
As an initial step, what is the result of running ls() with your RData
file loaded?  You should get a list of what is in memory.  Using RData
files can be as space-efficient or costly as the user's habits.  Did you
use save() or the save.image() command to produce the file? The
save.image() command stashes what is in memory and if you've run a
number of experimental procedures that did not pan out and you did not
discard with the results with rm(), they were saved to the rdata file
along with the information you did want, a procedure rather like filing
away all your work in a file drawer and then emptying the waste basket
into the drawer as well. If you save the data with ascii = TRUE as an
option, you can troll through the file and read what you saved.

JWD