R devel - Sep 2000 - Calloc() & Free() in different .C(.) calls --> not usable ?

[sending this to R-devel, since it may be of general interest and 
        could possibly lead to better instructions in the "R-exts"
manual]

Is this another case where only .Call() or .External() [or a fudge factor]
solve my problem? 

Following situation: 
 I use dyn.loaded C code to compute a recursive tree structure from my data.
 Beforehand I don't know how big the tree will be, but I want the tree
 (as integer array in my case) be passed back to R.

 Current solution [simplified!]:

  
 ## First I fit the tree to my Data, all in C, but only return the *size*
 ## I'll need to store that tree in R

	 size <-      .C("fittree", data, ..., res = integer(1))$res

 ## Now that I know its size, I can get it into R :

	 tree.in.R <- .C("gettree", size, res = integer(size))$res

 where in the C code, I have a
 global 
	static *tree_struct my_tree;
 and in C's fittree(), I use 
	my_tree = Calloc(1, tree_struct); 
 for building the tree, which is 
 then accessed by gettree(), which ends in
	Free(my_tree);

All well, BUT: 
    problem is that memory just builds up and is not freed at all.
>From a (more private) E-mail of Prof Brian Ripley, I read (maybe wrongly?)
that this seems to be ``common'':

    BDR> and the way that most mallocs work is to use sbrk to raise or lower
    BDR> the brk, that is the size of the data space. 
    BDR> They only reduce it when the data space is sufficiently
unfragmented.
	 --------------------------------------------------------------------
which may be my case : "Free(my_tree)" doesn't really free the
memory [according to my tests, looking at Unix' "ps .."].

Solutions (yes, one of them is  ``use .Call() !'') ??

Martin Maechler <maechler@stat.math.ethz.ch>
http://stat.ethz.ch/~maechler/
-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
r-devel mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !)  To:
r-devel-request@stat.math.ethz.ch
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._

> From: Martin Maechler <maechler@stat.math.ethz.ch>
> Date: Thu, 7 Sep 2000 18:32:59 +0200 (CEST)
> To: R-devel@stat.math.ethz.ch
> Subject: [Rd] Calloc() & Free() in different .C(.) calls --> not
usable ?
> 
>        [sending this to R-devel, since it may be of general interest and 
>         could possibly lead to better instructions in the
"R-exts" manual]
> 
> Is this another case where only .Call() or .External() [or a fudge factor]
> solve my problem? 
> 
> Following situation: 
>  I use dyn.loaded C code to compute a recursive tree structure from my
data.
>  Beforehand I don't know how big the tree will be, but I want the tree
>  (as integer array in my case) be passed back to R.
> 
>  Current solution [simplified!]:
> 
>   
>  ## First I fit the tree to my Data, all in C, but only return the *size*
>  ## I'll need to store that tree in R
> 
> 	 size <-      .C("fittree", data, ..., res = integer(1))$res
> 
>  ## Now that I know its size, I can get it into R :
> 
> 	 tree.in.R <- .C("gettree", size, res = integer(size))$res
> 
>  where in the C code, I have a
>  global 
> 	static *tree_struct my_tree;
>  and in C's fittree(), I use 
> 	my_tree = Calloc(1, tree_struct); 
>  for building the tree, which is 
>  then accessed by gettree(), which ends in
> 	Free(my_tree);
> 
> All well, BUT: 
>     problem is that memory just builds up and is not freed at all.
My nnet code does that, as does the rpart code.  I have not seen any
problems with memory-build up with either in R, and have run both
repetitively, e.g. within two-level cross-validations running for
days.  Calloc and Free are just error-checking wrappers for calloc and
free, so what happens is down to those on your OS.
> >From a (more private) E-mail of Prof Brian Ripley, I read (maybe
wrongly?)
> that this seems to be ``common'':
> 
>     BDR> and the way that most mallocs work is to use sbrk to raise or
lower
>     BDR> the brk, that is the size of the data space. 
>     BDR> They only reduce it when the data space is sufficiently
unfragmented.
> 	 --------------------------------------------------------------------
> which may be my case : "Free(my_tree)" doesn't really free
the
> memory [according to my tests, looking at Unix' "ps .."].
It frees the memory to malloc. There is no guarantee that malloc
returns it to the OS, but malloc should do so or re-use it.  I've just
monitored Solaris running R 1.1.1, and there is no evidence from ps/top that
ever returns any memory to the OS during these runs.  But it does not
build up more than 2Mb or so above R's usage at its first GC call, even
though it is being asked to allocate a few hundred Kb and deallocate it
many times.

-- 
Brian D. Ripley,                  ripley@stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272860 (secr)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595

-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
r-devel mailing list -- Read http://www.ci.tuwien.ac.at/~hornik/R/R-FAQ.html
Send "info", "help", or "[un]subscribe"
(in the "body", not the subject !)  To:
r-devel-request@stat.math.ethz.ch
_._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._._