[R] Memory issue

Prof Brian Ripley
Wed, 05 May 2010 11:39:36 -0000

On Wed, 5 May 2010, Alex van der Spek wrote:

> Reading a flat text file 138 Mbyte large into R with a combination of scan
> (to get the header) and read.table. After conversion of text time stamps to
> POSIXct and conversion of integer codes to factors I convert everything
into
> one data frame and release the old structures containing the data by using
> rm().
>
> Strangely, the rm() does not appear to reduce the used memory. I checked
> using memory.size(). Worse still, the amount of memory required grows. When
I
> save an image the .RData image file is only 23 Mbyte, yet at some point in
to
> the program, after having done nothing particularly difficult (two and
three
> way frequency tables and some lattice graphs) the amount of memory in use
is
> over 1 Gbyte.
>
> Not yet a problem, but it will become a problem. This is using R2.10.0 on
> Windows Vista.
>
> Does anybody know how to release memory as rm(dat) does not appear to do
this
> properly.

Rather, you do not appear to understand 'properly'.

First, you need to garbage-collect to find how much memory is
available for re-use.  R does that internally as needed, but you can
force it with gc().

Second, there is simply no reason for R not to use 'over 1 Gbyte' if
it is available (and it was).  Using lots of memory is faster, but the
garbage collector will clean up when needed.  The likely bottleneck
for you is not the amount of memory used but fragmentation of the
limited address space on 32-bit Windows.  See the documentation ....

Third, the .RData file is (by default) compressed.

And fourth, 'releasing memory' usually means giving it back to the OS.
That is an implementation detail and C runtime memory managers on many
builds of R either never do so or do so tardily.  This is again not an
issue unless your system is short of virtual memory and given how
cheap disc space is, there is no reason to be so.

>
> Regards,
> Alex van der Spek
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>

--
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595