luke-tierney at uiowa.edu
2016-Sep-22 19:01 UTC
[R] Memory not release when an environment is created
My preference is to use a top level function in the package or global env that takes as arguments just the variables I want in the parent frame. That avoids the explicit environment manipulations. Here that would be> makeFunc0 <- function(xmin, xmax)function(y) (y - xmin) / (xmax - xmin)> makeFunc1 <- function(x)makeFunc0(min(x), max(x))> f <- makeFunc1(1:1e8) > ls.str(all=TRUE, environment(f))xmax : int 100000000 xmin : int 1> parent.env(environment(f))<environment: R_GlobalEnv>> f(c(1234567, 2345678))[1] 0.01234566 0.02345677 Best, luke On Thu, 22 Sep 2016, William Dunlap via R-help wrote:> I like to have my function-returning functions use new.env(parent=XXX) > to make an environment for the returned function and put into it only > the objects needed by the function. The 'XXX' should be a an environment > which will hang around anyway. It could be globalenv(), but if your > function > is in a package, as.environment(paste0("package:", .packageName)) > would work well. The later ensures the your returned function has access > to all the other functions in that package. > > E.g., >> makeFunc1 <- function(x) { > envir <- new.env(parent = environment(sys.function())) > envir$xmax <- max(x) > envir$xmin <- min(x) > with(envir, function(y) (y - xmin) / (xmax - xmin)) > } >> f <- makeFunc1(1:1e8) >> ls.str(all=TRUE, environment(f)) > xmax : int 100000000 > xmin : int 1 >> parent.env(environment(f)) > <environment: R_GlobalEnv> >> f(c(1234567, 2345678)) > [1] 0.01234566 0.02345677 > > > > Bill Dunlap > TIBCO Software > wdunlap tibco.com > > On Thu, Sep 22, 2016 at 8:41 AM, Olivier Merle <oliviermerle35 at gmail.com> > wrote: > >> Dear, >> >> When I use big data for a temporary use it seems that the memory is not >> released when a function/environement is created nearby. >> Here the reproducible exemple: >> >> test<-function(){ >> x=matrix(0,50000,10000) >> y=function(nb) nb^2 >> return(y) >> } >> xx=test() # 3 Go of Ram is used >> gc() # Memory is not released !! even if x has been destroyed [look into >> software mem used] >> format(object.size(xx),units="auto") # 1.4 KiB => R is worng on the size >> of >> the object >> rm(xx) >> gc() # Memory is released >> >> ## Classic >> test2<-function(){ >> x=matrix(0,50000,10000) >> y=1 >> return(y) >> } >> xx=test2() # Memory is used >> gc() # => Memory is released >> >> How can I release the data in test without destroying the xx object ? As x >> which is big object is destroyed, I though I could get my memory back but >> it seems that the function y is keeping the x object. >> >> Best >> >> [[alternative HTML version deleted]] >> >> ______________________________________________ >> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see >> https://stat.ethz.ch/mailman/listinfo/r-help >> PLEASE do read the posting guide http://www.R-project.org/ >> posting-guide.html >> and provide commented, minimal, self-contained, reproducible code. >> > > [[alternative HTML version deleted]] > > ______________________________________________ > R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code. >-- Luke Tierney Ralph E. Wareham Professor of Mathematical Sciences University of Iowa Phone: 319-335-3386 Department of Statistics and Fax: 319-335-3017 Actuarial Science 241 Schaeffer Hall email: luke-tierney at uiowa.edu Iowa City, IA 52242 WWW: http://www.stat.uiowa.edu
luke-tierney at uiowa.edu
2016-Sep-22 19:24 UTC
[R] Memory not release when an environment is created
On Thu, 22 Sep 2016, luke-tierney at uiowa.edu wrote:> My preference is to use a top level function in the package or global > env that takes as arguments just the variables I want in the parent > frame. That avoids the explicit environment manipulations. Here that > would be > >> makeFunc0 <- function(xmin, xmax) > function(y) (y - xmin) / (xmax - xmin) >But I do keep forgetting the need to force the parameters if you don't want a big value to stick around until the returned function is used the first time, so a better definition of makeFUnc0 is makeFunc0 <- function(xmin, xmax) { force(xmin) force(xmax) function(y) (y - xmin) / (xmax - xmin) } Best, luke>> makeFunc1 <- function(x) > makeFunc0(min(x), max(x)) > >> f <- makeFunc1(1:1e8) >> ls.str(all=TRUE, environment(f)) > xmax : int 100000000 > xmin : int 1 >> parent.env(environment(f)) > <environment: R_GlobalEnv> >> f(c(1234567, 2345678)) > [1] 0.01234566 0.02345677 > > Best, > > luke > > > On Thu, 22 Sep 2016, William Dunlap via R-help wrote: > >> I like to have my function-returning functions use new.env(parent=XXX) >> to make an environment for the returned function and put into it only >> the objects needed by the function. The 'XXX' should be a an environment >> which will hang around anyway. It could be globalenv(), but if your >> function >> is in a package, as.environment(paste0("package:", .packageName)) >> would work well. The later ensures the your returned function has access >> to all the other functions in that package. >> >> E.g., >>> makeFunc1 <- function(x) { >> envir <- new.env(parent = environment(sys.function())) >> envir$xmax <- max(x) >> envir$xmin <- min(x) >> with(envir, function(y) (y - xmin) / (xmax - xmin)) >> } >>> f <- makeFunc1(1:1e8) >>> ls.str(all=TRUE, environment(f)) >> xmax : int 100000000 >> xmin : int 1 >>> parent.env(environment(f)) >> <environment: R_GlobalEnv> >>> f(c(1234567, 2345678)) >> [1] 0.01234566 0.02345677 >> >> >> >> Bill Dunlap >> TIBCO Software >> wdunlap tibco.com >> >> On Thu, Sep 22, 2016 at 8:41 AM, Olivier Merle <oliviermerle35 at gmail.com> >> wrote: >> >>> Dear, >>> >>> When I use big data for a temporary use it seems that the memory is not >>> released when a function/environement is created nearby. >>> Here the reproducible exemple: >>> >>> test<-function(){ >>> x=matrix(0,50000,10000) >>> y=function(nb) nb^2 >>> return(y) >>> } >>> xx=test() # 3 Go of Ram is used >>> gc() # Memory is not released !! even if x has been destroyed [look into >>> software mem used] >>> format(object.size(xx),units="auto") # 1.4 KiB => R is worng on the size >>> of >>> the object >>> rm(xx) >>> gc() # Memory is released >>> >>> ## Classic >>> test2<-function(){ >>> x=matrix(0,50000,10000) >>> y=1 >>> return(y) >>> } >>> xx=test2() # Memory is used >>> gc() # => Memory is released >>> >>> How can I release the data in test without destroying the xx object ? As x >>> which is big object is destroyed, I though I could get my memory back but >>> it seems that the function y is keeping the x object. >>> >>> Best >>> >>> [[alternative HTML version deleted]] >>> >>> ______________________________________________ >>> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see >>> https://stat.ethz.ch/mailman/listinfo/r-help >>> PLEASE do read the posting guide http://www.R-project.org/ >>> posting-guide.html >>> and provide commented, minimal, self-contained, reproducible code. >>> >> >> [[alternative HTML version deleted]] >> >> ______________________________________________ >> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see >> https://stat.ethz.ch/mailman/listinfo/r-help >> PLEASE do read the posting guide >> http://www.R-project.org/posting-guide.html >> and provide commented, minimal, self-contained, reproducible code. >> > >-- Luke Tierney Ralph E. Wareham Professor of Mathematical Sciences University of Iowa Phone: 319-335-3386 Department of Statistics and Fax: 319-335-3017 Actuarial Science 241 Schaeffer Hall email: luke-tierney at uiowa.edu Iowa City, IA 52242 WWW: http://www.stat.uiowa.edu
peter dalgaard
2016-Sep-22 23:52 UTC
[R] Memory not release when an environment is created
> On 22 Sep 2016, at 21:24 , luke-tierney at uiowa.edu wrote: > > On Thu, 22 Sep 2016, luke-tierney at uiowa.edu wrote: > >> My preference is to use a top level function in the package or global >> env that takes as arguments just the variables I want in the parent >> frame. That avoids the explicit environment manipulations. Here that >> would be >> >>> makeFunc0 <- function(xmin, xmax) >> function(y) (y - xmin) / (xmax - xmin) >> > > But I do keep forgetting the need to force the parameters if you don't > want a big value to stick around until the returned function is used > the first time,...not to mention the pains that arise if the makeFunc0 gets called with argument expressions that involve items that may change before the first call of the returned function.> so a better definition of makeFUnc0 is > > makeFunc0 <- function(xmin, xmax) { > force(xmin) > force(xmax) > function(y) (y - xmin) / (xmax - xmin) > } > > Best, > > luke > >>> makeFunc1 <- function(x) >> makeFunc0(min(x), max(x)) >> >>> f <- makeFunc1(1:1e8) >>> ls.str(all=TRUE, environment(f)) >> xmax : int 100000000 >> xmin : int 1 >>> parent.env(environment(f)) >> <environment: R_GlobalEnv> >>> f(c(1234567, 2345678)) >> [1] 0.01234566 0.02345677 >> >> Best, >> >> luke >> >> >> On Thu, 22 Sep 2016, William Dunlap via R-help wrote: >> >>> I like to have my function-returning functions use new.env(parent=XXX) >>> to make an environment for the returned function and put into it only >>> the objects needed by the function. The 'XXX' should be a an environment >>> which will hang around anyway. It could be globalenv(), but if your >>> function >>> is in a package, as.environment(paste0("package:", .packageName)) >>> would work well. The later ensures the your returned function has access >>> to all the other functions in that package. >>> E.g., >>>> makeFunc1 <- function(x) { >>> envir <- new.env(parent = environment(sys.function())) >>> envir$xmax <- max(x) >>> envir$xmin <- min(x) >>> with(envir, function(y) (y - xmin) / (xmax - xmin)) >>> } >>>> f <- makeFunc1(1:1e8) >>>> ls.str(all=TRUE, environment(f)) >>> xmax : int 100000000 >>> xmin : int 1 >>>> parent.env(environment(f)) >>> <environment: R_GlobalEnv> >>>> f(c(1234567, 2345678)) >>> [1] 0.01234566 0.02345677 >>> Bill Dunlap >>> TIBCO Software >>> wdunlap tibco.com >>> On Thu, Sep 22, 2016 at 8:41 AM, Olivier Merle <oliviermerle35 at gmail.com> >>> wrote: >>>> Dear, >>>> When I use big data for a temporary use it seems that the memory is not >>>> released when a function/environement is created nearby. >>>> Here the reproducible exemple: >>>> test<-function(){ >>>> x=matrix(0,50000,10000) >>>> y=function(nb) nb^2 >>>> return(y) >>>> } >>>> xx=test() # 3 Go of Ram is used >>>> gc() # Memory is not released !! even if x has been destroyed [look into >>>> software mem used] >>>> format(object.size(xx),units="auto") # 1.4 KiB => R is worng on the size >>>> of >>>> the object >>>> rm(xx) >>>> gc() # Memory is released >>>> ## Classic >>>> test2<-function(){ >>>> x=matrix(0,50000,10000) >>>> y=1 >>>> return(y) >>>> } >>>> xx=test2() # Memory is used >>>> gc() # => Memory is released >>>> How can I release the data in test without destroying the xx object ? As x >>>> which is big object is destroyed, I though I could get my memory back but >>>> it seems that the function y is keeping the x object. >>>> Best >>>> >>>> [[alternative HTML version deleted]] >>>> ______________________________________________ >>>> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see >>>> https://stat.ethz.ch/mailman/listinfo/r-help >>>> PLEASE do read the posting guide http://www.R-project.org/ >>>> posting-guide.html >>>> and provide commented, minimal, self-contained, reproducible code. >>> >>> [[alternative HTML version deleted]] >>> ______________________________________________ >>> R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see >>> https://stat.ethz.ch/mailman/listinfo/r-help >>> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html >>> and provide commented, minimal, self-contained, reproducible code. >> >> > > -- > Luke Tierney > Ralph E. Wareham Professor of Mathematical Sciences > University of Iowa Phone: 319-335-3386 > Department of Statistics and Fax: 319-335-3017 > Actuarial Science > 241 Schaeffer Hall email: luke-tierney at uiowa.edu > Iowa City, IA 52242 WWW: http://www.stat.uiowa.edu > > ______________________________________________ > R-help at r-project.org mailing list -- To UNSUBSCRIBE and more, see > https://stat.ethz.ch/mailman/listinfo/r-help > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html > and provide commented, minimal, self-contained, reproducible code.-- Peter Dalgaard, Professor, Center for Statistics, Copenhagen Business School Solbjerg Plads 3, 2000 Frederiksberg, Denmark Phone: (+45)38153501 Office: A 4.23 Email: pd.mes at cbs.dk Priv: PDalgd at gmail.com