R help - Jun 2000 - Garbage Collecting

Dear all,

Me and the Garbage Collector doesn't understand each other... 
I'm on R-1.0.1 (I have requested an upgrade to the right local people), an
osf1 system with an alphaev6 CPU. I have fair amounts of memory to use,
but it seems to be used up pretty fast nevertheless. 
I'm now trying to see what the garbage collector does, I have inserted a
gc(v=T)
call at a spot in my code.
I also discovered gctorture(), but I don't claim to be an R hacker for
that matter. :-)

I do a call 
> l4 <- t(apply(cbind(rep(1000000, 4)), 1, lineprofile, 1, 80, 
                smap01n$histogram))
and this will create a 101 * 4 element matrix. My first line in
'lineprofile' is gc(v=T), so it reports while running
Garbage collection [nr. 362602]...
629255 cons cells free (78%)
155163 Kbytes of heap free (76%)
Garbage collection [nr. 362603]...
627739 cons cells free (78%)
149224 Kbytes of heap free (73%)
Garbage collection [nr. 362604]...
626223 cons cells free (78%)
143286 Kbytes of heap free (70%)
Garbage collection [nr. 362605]...
624707 cons cells free (78%)
137347 Kbytes of heap free (67%)

So, it seems like close to 6 MB is used for time lineprofile is run. I
thought that the garbage collector would come and remove everything that
wasn't needed, so that the difference in free heap size would be
corresponding to the size of the array produced by lineprofile, that is,
202 bytes. And that when the call returns with the expected object, the gc
would free memory, so that what's in the memory would correspond to the
size of the objects, which again would correspond to what I have in my
.RData file. Currently, my .RData file (saved shortly before the previous
call) is slightly less than 450 kb, but 
> gc()
           free    total  (Mb)
Ncells   629305   800000  30.6
Vcells 19860721 26214400 200.0

Last night (before this call), I did a run with gctorture(T) but on a 
slightly more complex piece of my code (yes, it is _slow_), and since it
was started just after starting R, it starts with almost all the heap free
Garbage collection [nr. 2111]...
641457 cons cells free (80%)
203427 Kbytes of heap free (99%)
Garbage collection [nr. 43375]...
639941 cons cells free (79%)
197488 Kbytes of heap free (96%)
[...]
Garbage collection [nr. 303218]...
630828 cons cells free (78%)
161854 Kbytes of heap free (79%)
Garbage collection [nr. 356743]...
629295 cons cells free (78%)
155912 Kbytes of heap free (76%)

At this point, I halted it with BREAK, and did 
> gc()
           free    total  (Mb)
Ncells   629306   800000  30.6
Vcells 19860723 26214400 200.0

I would expect it to get rid of everything that went on in the last call,
but 19860723/26214400 is 76%, so that didn't happen. 

Now, this isn't too much to go on, I realize, but I want to hear if my
understanding of what the garbage collector is supposed to be doing is
totally wrong before I get deeper into this. If I'm not totally
wrong, and anybody cares to have a look, I have saved the objects that is
sufficient to reproduce the problem on my system in
<URL:http://www.astro.uio.no/~kjetikj/tmp/kjetil.RData>

I had a core dump (warning, 291MB), after killing R with wildly giving all
kinds of signals... (BREAK, QUIT, STOP, KILL) :-)
<URL:http://www.astro.uio.no/~kjetikj/tmp/Rcore1>, I don't know if it
is
of much use. 

Yours Confusedly,

Kjetil
-- 
Kjetil Kjernsmo
Graduate astronomy-student                    Problems worthy of attack
University of Oslo, Norway            Prove their worth by hitting back
E-mail: kjetikj at astro.uio.no                                - Piet Hein
Homepage <URL:http://www.astro.uio.no/~kjetikj/>
Webmaster at skepsis.no 


-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !)  To: r-help-request at
stat.math.ethz.ch
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._

Kjetil Kjernsmo <kjetil.kjernsmo at astro.uio.no> writes:
> I do a call 
> > l4 <- t(apply(cbind(rep(1000000, 4)), 1, lineprofile, 1, 80, 
>                 smap01n$histogram))
> and this will create a 101 * 4 element matrix. My first line in
> 'lineprofile' is gc(v=T), so it reports while running
> Garbage collection [nr. 362602]...
> 629255 cons cells free (78%)
> 155163 Kbytes of heap free (76%)
> Garbage collection [nr. 362603]...
> 627739 cons cells free (78%)
> 149224 Kbytes of heap free (73%)
> Garbage collection [nr. 362604]...
> 626223 cons cells free (78%)
> 143286 Kbytes of heap free (70%)
> Garbage collection [nr. 362605]...
> 624707 cons cells free (78%)
> 137347 Kbytes of heap free (67%)
> 
> So, it seems like close to 6 MB is used for time lineprofile is run. I
> thought that the garbage collector would come and remove everything that
> wasn't needed, so that the difference in free heap size would be
> corresponding to the size of the array produced by lineprofile, that is,
> 202 bytes. 
Hm. I think I would do that as

do.call("rbind", lapply(rep(1000000,4), lineprofile, ....) 

(provided I grasped the intention correctly). Still looks strange,
though, and as usual I wouldn't rule out that there might be a problem
with 32/64 bit assumptions somewhere since we have so few people on 64
bit platforms. 

In general, the garbage collector should indeed remove anything that
is not used by an active R object or is otherwise protected. So if
lineprofile doesn't have any side effects, then only the return value
should be left after the call.
> Now, this isn't too much to go on, I realize, but I want to hear if my
> understanding of what the garbage collector is supposed to be doing is
> totally wrong before I get deeper into this. If I'm not totally
> wrong, and anybody cares to have a look, I have saved the objects that is
> sufficient to reproduce the problem on my system in
> <URL:http://www.astro.uio.no/~kjetikj/tmp/kjetil.RData>
Perhaps you could cut the size of the problem down a bit? Some of us
are sitting at machines with 64MB RAM and are not too thrilled about R
processes with 200+ MB of memory footprint...

If you're feeling adventurous, Luke Tierney has just opened up an
experimental branch with a substantially modified garbage collector. I
believe there's a way to get it by anonymous CVS, the branch label is
R-GenGC. However, that one should be *less* strict about cleaning out
every last bit of unused data than the current one (in the hope of
getting better speed in return).

-- 
   O__  ---- Peter Dalgaard             Blegdamsvej 3  
  c/ /'_ --- Dept. of Biostatistics     2200 Cph. N   
 (*) \(*) -- University of Copenhagen   Denmark      Ph: (+45) 35327918
~~~~~~~~~~ - (p.dalgaard at biostat.ku.dk)             FAX: (+45) 35327907
-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
r-help mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !)  To: r-help-request at
stat.math.ethz.ch
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._