Jonathan Greenberg
2013-Jun-20 14:45 UTC
Determining the maximum memory usage of a function
Folks: I apologize for the cross-posting between r-help and r-sig-hpc, but I figured this question was relevant to both lists. I''m writing a function to be applied to an input dataset that will be broken up into chunks for memory management reasons and for parallel execution. I am trying to determine, for a given function, what the *maximum* memory usage during its execution is (which may not be the beginning or the end of the function, but somewhere in the middle), so I can "plan" for the chunk size (e.g. have a table of chunk size vs. max memory usage). Is there a trick for determining this? --j -- Jonathan A. Greenberg, PhD Assistant Professor Global Environmental Analysis and Remote Sensing (GEARS) Laboratory Department of Geography and Geographic Information Science University of Illinois at Urbana-Champaign 607 South Mathews Avenue, MC 150 Urbana, IL 61801 Phone: 217-300-1924 http://www.geog.illinois.edu/~jgrn/ AIM: jgrn307, MSN: jgrn307@hotmail.com, Gchat: jgrn307, Skype: jgrn3007
Jonathan Greenberg
2013-Jun-20 15:36 UTC
[R] Determining the maximum memory usage of a function
Jim: Thanks, but I'm looking for something that can be used somewhat automatically -- the function in question would be user-provided and passed to my "chunking" algorithm, so in this case it would be the end-user (not me) who would have to embed these -- would Rprof(memory.profiling=TRUE) # my function Rprof(NULL) ... and then taking the max of the tseries output be a reasonable approach? If so, which of the three outputs (vsize.small vsize.large nodes) would be best compared against the available memory? Cheers! --j On Thu, Jun 20, 2013 at 10:07 AM, jim holtman <jholtman at gmail.com> wrote:> What I would do is to use "memory.size()" to get the amount of memory being > used. Do a call at the beginning of the function to determine the base, and > then at other points in the code to see what the difference from the base is > and keep track of the maximum difference. I am not sure if just getting the > memory usage at the end would be sufficient since there may be some garbage > collection in between, or you might be creating some large objects and then > deleting/reusing them. So keep track after large chunks of code to see what > is happening. > > > On Thu, Jun 20, 2013 at 10:45 AM, Jonathan Greenberg <jgrn at illinois.edu> > wrote: >> >> Folks: >> >> I apologize for the cross-posting between r-help and r-sig-hpc, but I >> figured this question was relevant to both lists. I'm writing a >> function to be applied to an input dataset that will be broken up into >> chunks for memory management reasons and for parallel execution. I am >> trying to determine, for a given function, what the *maximum* memory >> usage during its execution is (which may not be the beginning or the >> end of the function, but somewhere in the middle), so I can "plan" for >> the chunk size (e.g. have a table of chunk size vs. max memory usage). >> >> Is there a trick for determining this? >> >> --j >> >> -- >> Jonathan A. Greenberg, PhD >> Assistant Professor >> Global Environmental Analysis and Remote Sensing (GEARS) Laboratory >> Department of Geography and Geographic Information Science >> University of Illinois at Urbana-Champaign >> 607 South Mathews Avenue, MC 150 >> Urbana, IL 61801 >> Phone: 217-300-1924 >> http://www.geog.illinois.edu/~jgrn/ >> AIM: jgrn307, MSN: jgrn307 at hotmail.com, Gchat: jgrn307, Skype: jgrn3007 >> >> ______________________________________________ >> R-help at r-project.org mailing list >> https://stat.ethz.ch/mailman/listinfo/r-help >> PLEASE do read the posting guide >> http://www.R-project.org/posting-guide.html >> and provide commented, minimal, self-contained, reproducible code. > > > > > -- > Jim Holtman > Data Munger Guru > > What is the problem that you are trying to solve? > Tell me what you want to do, not how you want to do it.-- Jonathan A. Greenberg, PhD Assistant Professor Global Environmental Analysis and Remote Sensing (GEARS) Laboratory Department of Geography and Geographic Information Science University of Illinois at Urbana-Champaign 607 South Mathews Avenue, MC 150 Urbana, IL 61801 Phone: 217-300-1924 http://www.geog.illinois.edu/~jgrn/ AIM: jgrn307, MSN: jgrn307 at hotmail.com, Gchat: jgrn307, Skype: jgrn3007