R help - Sep 2016 - Memory not release when an environment is created

Dear,

When I use big data for a temporary use it seems that the memory is not
released when a function/environement is created nearby.
Here the reproducible exemple:

test<-function(){
x=matrix(0,50000,10000)
y=function(nb) nb^2
return(y)
}
xx=test() # 3 Go of Ram is used
gc() # Memory is not released !! even if x has been destroyed [look into
software mem used]
format(object.size(xx),units="auto") # 1.4 KiB => R is worng on the
size of
the object
rm(xx)
gc() # Memory is released

## Classic
test2<-function(){
x=matrix(0,50000,10000)
y=1
return(y)
}
xx=test2() # Memory is used
gc() # => Memory is released

How can I release the data in test without destroying the xx object ? As x
which is big object is destroyed, I though I could get my memory back but
it seems that the function y is keeping the x object.

Best

	[[alternative HTML version deleted]]

> On 22 Sep 2016, at 18:41, Olivier Merle <oliviermerle35 at gmail.com>
wrote:
> 
> Dear,
> 
> When I use big data for a temporary use it seems that the memory is not
> released when a function/environement is created nearby.
> Here the reproducible exemple:
> 
> test<-function(){
> x=matrix(0,50000,10000)
> y=function(nb) nb^2
> return(y)
> }
> xx=test() # 3 Go of Ram is used
> gc() # Memory is not released !! even if x has been destroyed [look into
> software mem used]
Because y is a function and returns with its own environment.

ls(environment(xx)) # x and y objects are still there
> How can I release the data in test without destroying the xx object ? As x
> which is big object is destroyed, I though I could get my memory back but
> it seems that the function y is keeping the x object.
if you do not need the x object in y function then remove it in it?s own
environment as follows;
> test<-function(){
> x=matrix(0,50000,10000)
   rm(x)
> y=function(nb) nb^2
> return(y)
> }
or if you need to remove it out of the function;

rm("x", envir = environment(xx))
ls(environment(xx)) # x has gone

 If y function uses x somehow, then you will need to live with a big object.

> On Sep 22, 2016, at 8:41 AM, Olivier Merle <oliviermerle35 at
gmail.com> wrote:
> 
> Dear,
> 
> When I use big data for a temporary use it seems that the memory is not
> released when a function/environement is created nearby.
> Here the reproducible exemple:
> 
> test<-function(){
> x=matrix(0,50000,10000)
> y=function(nb) nb^2
> return(y)
> }
> xx=test() # 3 Go of Ram is used
> gc() # Memory is not released !! even if x has been destroyed [look into
> software mem used]
Looking at this I would imagine that the unreleased memory is allocated within
the `xx` objects environment (since functions in R are actually closures that
carry along their environments of creation.

> format(object.size(xx),units="auto") # 1.4 KiB => R is worng
on the size of
> the object
> rm(xx)
> gc() # Memory is released
> 
> ## Classic
> test2<-function(){
> x=matrix(0,50000,10000)
> y=1
> return(y)
> }
> xx=test2() # Memory is used
> gc() # => Memory is released
> 
> How can I release the data in test without destroying the xx object ? As x
> which is big object is destroyed, I though I could get my memory back but
> it seems that the function y is keeping the x object.
That's how I understand the semantics of R. You would need to replace the
environment that is attached to `xx`. I madesomwaht smaller object:

test<-function(
x=matrix(0,500,
y=function(nb) 
return(y)
}
xx=test() 
> environment(xx)
<environment: 0x7fda21ba35f0>
> object.size(xx)
1384 bytes
> ls(x, envir=environment(xx) )
[1] "x" "y"
> ?get
> object.size( get("x", envir=environment(xx) ) )
400200 bytes

Even with that perspective in mind it still took me two tries to get rid of the
xx object, since my abilities to "program on the language" are still
fairly modest.:
> eval( x <- NULL, envir=environment(xx) )
NULL
> object.size( get("x", envir=environment(xx) ) )
400200 bytes
> eval( quote(x <- NULL), envir=environment(xx) )
> object.size( get("x", envir=environment(xx) ) )
0 bytes

So now I have learned that object.size does not include measurements of the size
of function environments, a fact about which I was not aware.

Best;
David.
> 
> Best
> 
> 	[[alternative HTML version deleted]]
> 
> ______________________________________________
> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
David Winsemius
Alameda, CA, USA

> On Sep 22, 2016, at 9:45 AM, Ismail SEZEN <sezenismail at gmail.com>
wrote:
> 
> 
>> On 22 Sep 2016, at 18:41, Olivier Merle <oliviermerle35 at
gmail.com> wrote:
>> 
>> Dear,
>> 
>> When I use big data for a temporary use it seems that the memory is not
>> released when a function/environement is created nearby.
>> Here the reproducible exemple:
>> 
>> test<-function(){
>> x=matrix(0,50000,10000)
>> y=function(nb) nb^2
>> return(y)
>> }
>> xx=test() # 3 Go of Ram is used
>> gc() # Memory is not released !! even if x has been destroyed [look
into
>> software mem used]
> 
> Because y is a function and returns with its own environment.
> 
> ls(environment(xx)) # x and y objects are still there
> 
>> How can I release the data in test without destroying the xx object ?
As x
>> which is big object is destroyed, I though I could get my memory back
but
>> it seems that the function y is keeping the x object.
> 
> if you do not need the x object in y function then remove it in it?s own
environment as follows;
> 
>> test<-function(){
>> x=matrix(0,50000,10000)
>   rm(x)
>> y=function(nb) nb^2
>> return(y)
>> }
> 
> or if you need to remove it out of the function;
> 
> rm("x", envir = environment(xx))
> ls(environment(xx)) # x has gone
That's much clearer than my `eval(quote(...` approach. I had forgotten that
`rm` had an 'environment' parameter. I had considered trying:

environment(xx)$x <- NULL   # but after looking at `environment`'s help
page I was pretty sure it would have failed

# But there again I was wrong:

test<-function(){
x=matrix(0,500,100)
y=function(nb) nb^2
return(y)
}
xx=test()

environment(xx)$x <- NULL

> object.size( get("x", envir=environment(xx) ) )
0 bytes

I still think rm() is the way to go but this offers another illustration of
available methods of environment mangling, er, manipulation.

Best;
David.

> 
> If y function uses x somehow, then you will need to live with a big object.
> ______________________________________________
> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
David Winsemius
Alameda, CA, USA

I like to have my function-returning functions use new.env(parent=XXX)
to make an environment for the returned function and put into it only
the objects needed by the function.  The 'XXX' should be a an
environment
which will hang around anyway.  It could be globalenv(), but if your
function
is in a package, as.environment(paste0("package:", .packageName))
would work well.  The later ensures the your returned function has access
to all the other functions in that package.

E.g.,
> makeFunc1 <- function(x) {
    envir <- new.env(parent = environment(sys.function()))
    envir$xmax <- max(x)
    envir$xmin <- min(x)
    with(envir, function(y) (y - xmin) / (xmax - xmin))
}
> f <- makeFunc1(1:1e8)
> ls.str(all=TRUE, environment(f))
xmax :  int 100000000
xmin :  int 1
> parent.env(environment(f))
<environment: R_GlobalEnv>
> f(c(1234567, 2345678))
[1] 0.01234566 0.02345677



Bill Dunlap
TIBCO Software
wdunlap tibco.com

On Thu, Sep 22, 2016 at 8:41 AM, Olivier Merle <oliviermerle35 at
gmail.com>
wrote:
> Dear,
>
> When I use big data for a temporary use it seems that the memory is not
> released when a function/environement is created nearby.
> Here the reproducible exemple:
>
> test<-function(){
> x=matrix(0,50000,10000)
> y=function(nb) nb^2
> return(y)
> }
> xx=test() # 3 Go of Ram is used
> gc() # Memory is not released !! even if x has been destroyed [look into
> software mem used]
> format(object.size(xx),units="auto") # 1.4 KiB => R is worng
on the size
> of
> the object
> rm(xx)
> gc() # Memory is released
>
> ## Classic
> test2<-function(){
> x=matrix(0,50000,10000)
> y=1
> return(y)
> }
> xx=test2() # Memory is used
> gc() # => Memory is released
>
> How can I release the data in test without destroying the xx object ? As x
> which is big object is destroyed, I though I could get my memory back but
> it seems that the function y is keeping the x object.
>
> Best
>
>         [[alternative HTML version deleted]]
>
> ______________________________________________
> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/
> posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>
	[[alternative HTML version deleted]]

My preference is to use a top level function in the package or global
env that takes as arguments just the variables I want in the parent
frame. That avoids the explicit environment manipulations. Here that
would be
> makeFunc0 <- function(xmin, xmax)
       function(y) (y - xmin) / (xmax - xmin)
> makeFunc1 <- function(x)
       makeFunc0(min(x), max(x))
> f <- makeFunc1(1:1e8)
> ls.str(all=TRUE, environment(f))
xmax :  int 100000000
xmin :  int 1
> parent.env(environment(f))
<environment: R_GlobalEnv>
> f(c(1234567, 2345678))
[1] 0.01234566 0.02345677

Best,

luke


On Thu, 22 Sep 2016, William Dunlap via R-help wrote:
> I like to have my function-returning functions use new.env(parent=XXX)
> to make an environment for the returned function and put into it only
> the objects needed by the function.  The 'XXX' should be a an
environment
> which will hang around anyway.  It could be globalenv(), but if your
> function
> is in a package, as.environment(paste0("package:", .packageName))
> would work well.  The later ensures the your returned function has access
> to all the other functions in that package.
>
> E.g.,
>> makeFunc1 <- function(x) {
>    envir <- new.env(parent = environment(sys.function()))
>    envir$xmax <- max(x)
>    envir$xmin <- min(x)
>    with(envir, function(y) (y - xmin) / (xmax - xmin))
> }
>> f <- makeFunc1(1:1e8)
>> ls.str(all=TRUE, environment(f))
> xmax :  int 100000000
> xmin :  int 1
>> parent.env(environment(f))
> <environment: R_GlobalEnv>
>> f(c(1234567, 2345678))
> [1] 0.01234566 0.02345677
>
>
>
> Bill Dunlap
> TIBCO Software
> wdunlap tibco.com
>
> On Thu, Sep 22, 2016 at 8:41 AM, Olivier Merle <oliviermerle35 at
gmail.com>
> wrote:
>
>> Dear,
>>
>> When I use big data for a temporary use it seems that the memory is not
>> released when a function/environement is created nearby.
>> Here the reproducible exemple:
>>
>> test<-function(){
>> x=matrix(0,50000,10000)
>> y=function(nb) nb^2
>> return(y)
>> }
>> xx=test() # 3 Go of Ram is used
>> gc() # Memory is not released !! even if x has been destroyed [look
into
>> software mem used]
>> format(object.size(xx),units="auto") # 1.4 KiB => R is
worng on the size
>> of
>> the object
>> rm(xx)
>> gc() # Memory is released
>>
>> ## Classic
>> test2<-function(){
>> x=matrix(0,50000,10000)
>> y=1
>> return(y)
>> }
>> xx=test2() # Memory is used
>> gc() # => Memory is released
>>
>> How can I release the data in test without destroying the xx object ?
As x
>> which is big object is destroyed, I though I could get my memory back
but
>> it seems that the function y is keeping the x object.
>>
>> Best
>>
>>         [[alternative HTML version deleted]]
>>
>> ______________________________________________
>> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
>> https://stat.ethz.ch/mailman/listinfo/r-help
>> PLEASE do read the posting guide http://www.R-project.org/
>> posting-guide.html
>> and provide commented, minimal, self-contained, reproducible code.
>>
>
> 	[[alternative HTML version deleted]]
>
> ______________________________________________
> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>
-- 
Luke Tierney
Ralph E. Wareham Professor of Mathematical Sciences
University of Iowa                  Phone:             319-335-3386
Department of Statistics and        Fax:               319-335-3017
    Actuarial Science
241 Schaeffer Hall                  email:   luke-tierney at uiowa.edu
Iowa City, IA 52242                 WWW:  http://www.stat.uiowa.edu