Strange because in my all my experiments calling malloc.trim always helped - memory reported by top decreased to the level it supposed to be. Do you have in mind case when calling malloc.trim won't do anything? Also shouldn't MALLOC_TRIM_THRESHOLD_ env variable has impact on malloc.trim calls? At the moment seems any value is ignored... 12 ???. 2017 ?. 6:09 ?? ???????????? "Simon Urbanek" < simon.urbanek at r-project.org> ???????:> > > On Aug 11, 2017, at 12:57 PM, I?aki ?car <i.ucar86 at gmail.com> wrote: > > > > 2017-08-11 16:00 GMT+02:00 Martin Maechler <maechler at stat.math.ethz.ch>: > >>>>>>> Dmitriy Selivanov <selivanov.dmitriy at gmail.com> > >>>>>>> on Fri, 11 Aug 2017 17:33:31 +0400 writes: > >> > >>> Hi mailing list and R-core. Could someone from R-core please help me to > >>> create account in bugzilla? I would like to submit issue related to > gc() to > >>> wishlist. > >> > >> I will create one. > >> > >> Your previous e-mails left me pretty clueless about what the > >> problem is that you want to solve ... but maybe others > >> understand better what you mean. > >> > >> Note that in the case of such a relatively sophisticated wish > >> without a clear sign of a problem (in my view) > >> chances are not high that anything will change, unless someone > >> provides a (small footprint) patch towards the (R-devel aka > >> "trunk") sources *and* reproducible R code that depicts the > >> problem. > > > > How to reproduce it: > > > > a <- replicate(2e6, new.env()) # ~ 1.4 GB of memory > > rm(a) > > gc() # the R process still has the memory assigned > > > > Right, but that's unavoidable because of the way Linux allocates memory - > see FAQ 7.42 > The memory is free, Linux just keeps it for future allocations. > > Running malloc.trim doesn't help, because the issue is fragmentation due > to the linear design of the pool - you likely will have another object on > top so in most practical cases malloc.trim() doesn't actually do anything. > You can always call malloc.trim() yourself is you think it helps, but it > doesn't in the general case. The only way to address that would be to move > allocated objects from top of the pool down, but that's not something R can > allow, because it cannot know which code still has SEXP pointers referring > to that object. > > Cheers, > Simon > > >[[alternative HTML version deleted]]
On Sat, 12 Aug 2017, Dmitriy Selivanov wrote:> Strange because in my all my experiments calling malloc.trim always helped > - memory reported by top decreased to the level it supposed to be. Do you > have in mind case when calling malloc.trim won't do anything? Also > shouldn't MALLOC_TRIM_THRESHOLD_ env variable has impact on malloc.trim > calls? At the moment seems any value is ignored...This is a question for glibc developers. For that matter, this entire thread is really about tuning of the malloc in glibc and should ideally be addressed upstream. There has been some discussion of this in other contexts, e.g. Python at https://nuald.blogspot.com/2013/06/memory-reclaiming-in-python.html and emacs at http://notes.secretsauce.net/notes/2016/04/08_glibc-malloc-inefficiency.html As the Python posts poitns out, it is possible to use alternate malloc implementations, either rebuilding R to use them or using LD_PRELOAD. On Ubuntu for example, you can have R use jemalloc with sudo apt-get install libjemalloc1 env LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1 R This does not seem to hold onto memory to the same degree, but I don't know about any other aspect of its performance. The emacs post suggests that calling malloc_trim may have more of an effect in some cases: the post describes callign it via gdb on all running processes and sees a substantial memory footprint drop for emacs, Xorg and opera. I tried the same experiment on my system with emacs, Xorg and firefox and didn't see what that post saw -- maybe 1 few percent recovery. At this point I'm not sure we have enough data to show that adding malloc_trim calls at the end of each GC, say, would warrant the nusance of having to add configure checks. It would also be necessary to make sure that adding this doesn't significantly ompact malloc performance. I don't know if this issue exists on Windows as well; it might as the basic malloc we use there is the same as used in glibc (Dog Lea's malloc). Best, luke> > 12 ???. 2017 ?. 6:09 ?? ???????????? "Simon Urbanek" < > simon.urbanek at r-project.org> ???????: > >> >> > On Aug 11, 2017, at 12:57 PM, I?aki ?car <i.ucar86 at gmail.com> wrote: >> > >> > 2017-08-11 16:00 GMT+02:00 Martin Maechler <maechler at stat.math.ethz.ch>: >> >>>>>>> Dmitriy Selivanov <selivanov.dmitriy at gmail.com> >> >>>>>>> on Fri, 11 Aug 2017 17:33:31 +0400 writes: >> >> >> >>> Hi mailing list and R-core. Could someone from R-core please help me to >> >>> create account in bugzilla? I would like to submit issue related to >> gc() to >> >>> wishlist. >> >> >> >> I will create one. >> >> >> >> Your previous e-mails left me pretty clueless about what the >> >> problem is that you want to solve ... but maybe others >> >> understand better what you mean. >> >> >> >> Note that in the case of such a relatively sophisticated wish >> >> without a clear sign of a problem (in my view) >> >> chances are not high that anything will change, unless someone >> >> provides a (small footprint) patch towards the (R-devel aka >> >> "trunk") sources *and* reproducible R code that depicts the >> >> problem. >> > >> > How to reproduce it: >> > >> > a <- replicate(2e6, new.env()) # ~ 1.4 GB of memory >> > rm(a) >> > gc() # the R process still has the memory assigned >> > >> >> Right, but that's unavoidable because of the way Linux allocates memory - >> see FAQ 7.42 >> The memory is free, Linux just keeps it for future allocations. >> >> Running malloc.trim doesn't help, because the issue is fragmentation due >> to the linear design of the pool - you likely will have another object on >> top so in most practical cases malloc.trim() doesn't actually do anything. >> You can always call malloc.trim() yourself is you think it helps, but it >> doesn't in the general case. The only way to address that would be to move >> allocated objects from top of the pool down, but that's not something R can >> allow, because it cannot know which code still has SEXP pointers referring >> to that object. >> >> Cheers, >> Simon >> >> >> > > [[alternative HTML version deleted]] > > ______________________________________________ > R-devel at r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-devel-- Luke Tierney Ralph E. Wareham Professor of Mathematical Sciences University of Iowa Phone: 319-335-3386 Department of Statistics and Fax: 319-335-3017 Actuarial Science 241 Schaeffer Hall email: luke-tierney at uiowa.edu Iowa City, IA 52242 WWW: http://www.stat.uiowa.edu
On 12 August 2017 at 15:10, luke-tierney at uiowa.edu wrote: | As the Python posts poitns out, it is possible to use alternate malloc | implementations, either rebuilding R to use them or using LD_PRELOAD. | On Ubuntu for example, you can have R use jemalloc with | | sudo apt-get install libjemalloc1 | env LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1 R | | This does not seem to hold onto memory to the same degree, but I don't | know about any other aspect of its performance. Interesting. I don't really know anything about malloc versus jemalloc internals but I can affirm that redis -- an in-memory database written in single-threaded C for high performance -- in its Debian builds has been using jemalloc for years, presumably by choice of the maintainer. (We are very happy users of [a gently patched] redis at work; lots of writes; very good uptime.) Having the ability to switch to jemalloc, we could design a test bench and compare what the impact is. Similarly, if someone cared, I could (presumably) alter the default R build for Debian and Ubunto to also switch to jemalloc. Anybody feel like doing some empirics? Dirk -- http://dirk.eddelbuettel.com | @eddelbuettel | edd at debian.org
FWIW if we are talking about alternative allocators, tcmalloc is another candidate that we are using for our projects where we care about allocations and performance (another upshot is that it's very flexible so you can do a lot of cool things if you care). However I didn't try it with R - I'll have a look if it can address the issue we're talking about.> On Aug 12, 2017, at 4:10 PM, luke-tierney at uiowa.edu wrote: > > On Sat, 12 Aug 2017, Dmitriy Selivanov wrote: > >> Strange because in my all my experiments calling malloc.trim always helped >> - memory reported by top decreased to the level it supposed to be. Do you >> have in mind case when calling malloc.trim won't do anything? Also >> shouldn't MALLOC_TRIM_THRESHOLD_ env variable has impact on malloc.trim >> calls? At the moment seems any value is ignored... > > This is a question for glibc developers. For that matter, this entire > thread is really about tuning of the malloc in glibc and should > ideally be addressed upstream. > > There has been some discussion of this in other contexts, e.g. Python at > > https://nuald.blogspot.com/2013/06/memory-reclaiming-in-python.html > > and emacs at > > http://notes.secretsauce.net/notes/2016/04/08_glibc-malloc-inefficiency.html > > As the Python posts poitns out, it is possible to use alternate malloc > implementations, either rebuilding R to use them or using LD_PRELOAD. > On Ubuntu for example, you can have R use jemalloc with > > sudo apt-get install libjemalloc1 > env LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1 R > > This does not seem to hold onto memory to the same degree, but I don't > know about any other aspect of its performance. > > The emacs post suggests that calling malloc_trim may have more of an > effect in some cases: the post describes callign it via gdb on all > running processes and sees a substantial memory footprint drop for > emacs, Xorg and opera. I tried the same experiment on my system with > emacs, Xorg and firefox and didn't see what that post saw -- maybe 1 > few percent recovery. > > At this point I'm not sure we have enough data to show that adding > malloc_trim calls at the end of each GC, say, would warrant the > nusance of having to add configure checks. It would also be necessary > to make sure that adding this doesn't significantly ompact malloc > performance. > > I don't know if this issue exists on Windows as well; it might as the > basic malloc we use there is the same as used in glibc (Dog Lea's > malloc). > > Best, > > luke > > >> >> 12 ???. 2017 ?. 6:09 ?? ???????????? "Simon Urbanek" < >> simon.urbanek at r-project.org> ???????: >> >>> >>> > On Aug 11, 2017, at 12:57 PM, I?aki ?car <i.ucar86 at gmail.com> wrote: >>> > >>> > 2017-08-11 16:00 GMT+02:00 Martin Maechler <maechler at stat.math.ethz.ch>: >>> >>>>>>> Dmitriy Selivanov <selivanov.dmitriy at gmail.com> >>> >>>>>>> on Fri, 11 Aug 2017 17:33:31 +0400 writes: >>> >> >>> >>> Hi mailing list and R-core. Could someone from R-core please help me to >>> >>> create account in bugzilla? I would like to submit issue related to >>> gc() to >>> >>> wishlist. >>> >> >>> >> I will create one. >>> >> >>> >> Your previous e-mails left me pretty clueless about what the >>> >> problem is that you want to solve ... but maybe others >>> >> understand better what you mean. >>> >> >>> >> Note that in the case of such a relatively sophisticated wish >>> >> without a clear sign of a problem (in my view) >>> >> chances are not high that anything will change, unless someone >>> >> provides a (small footprint) patch towards the (R-devel aka >>> >> "trunk") sources *and* reproducible R code that depicts the >>> >> problem. >>> > >>> > How to reproduce it: >>> > >>> > a <- replicate(2e6, new.env()) # ~ 1.4 GB of memory >>> > rm(a) >>> > gc() # the R process still has the memory assigned >>> > >>> >>> Right, but that's unavoidable because of the way Linux allocates memory - >>> see FAQ 7.42 >>> The memory is free, Linux just keeps it for future allocations. >>> >>> Running malloc.trim doesn't help, because the issue is fragmentation due >>> to the linear design of the pool - you likely will have another object on >>> top so in most practical cases malloc.trim() doesn't actually do anything. >>> You can always call malloc.trim() yourself is you think it helps, but it >>> doesn't in the general case. The only way to address that would be to move >>> allocated objects from top of the pool down, but that's not something R can >>> allow, because it cannot know which code still has SEXP pointers referring >>> to that object. >>> >>> Cheers, >>> Simon >>> >>> >>> >> >> [[alternative HTML version deleted]] >> >> ______________________________________________ >> R-devel at r-project.org mailing list >> https://stat.ethz.ch/mailman/listinfo/r-devel > > -- > Luke Tierney > Ralph E. Wareham Professor of Mathematical Sciences > University of Iowa Phone: 319-335-3386 > Department of Statistics and Fax: 319-335-3017 > Actuarial Science > 241 Schaeffer Hall email: luke-tierney at uiowa.edu > Iowa City, IA 52242 WWW: http://www.stat.uiowa.edu