I would like to profile heap usage on a per thread basis in a large application process. To do this I am tracking calls to malloc and free with the attached script. Everything seems to look OK with some simple test programmes, however, when I track the live system, the results suggest that one thread has grown by approximately 1Gb and I would be surprised if this was true because I ran pmap -x on the same process immediately before starting the DTrace script and immediately afterwards. Comparing the two pmap results no heap memory growth (I summed the individual mappings), a reduction in the RSS total and no change in the virtual memory. Is it valid to assume that all memory allocated with malloc will be released with a call to free or are there alternatives to free which I have not taken account of? My test programmes also made use of new and delete (c++) and it appears that there are underlying calls to malloc and free on the implementation of these. Is this always the case? This message posted from opensolaris.org -------------- next part -------------- A non-text attachment was scrubbed... Name: watch_malloc.d Type: application/octet-stream Size: 1809 bytes Desc: not available URL: <http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20060926/307b2dfe/attachment.obj>
On Tue, Sep 26, 2006 at 07:51:37AM -0700, J.J. Shore wrote:> I would like to profile heap usage on a per thread basis in a large > application process.<SNIP!> Couldn''t libumem better solve this problem? Check out the umem_debug(3MALLOC) page for details.> Is it valid to assume that all memory allocated with malloc will be > released with a call to free or are there alternatives to free which I have > not taken account of?There''s realloc(), but that may actually call free() in its implementation.> My test programmes also made use of new and delete (c++) and it appears > that there are underlying calls to malloc and free on the implementation of > these. Is this always the case?I''m pretty sure new and delete from c++ call malloc, free, and friends all the time. Dan
> I''m pretty sure new and delete from c++ call malloc, free, and friends> all the time. That seems like an implementation artifact -- and as I recall, you can overload or replace many flavors of operator new -- including the "global" operator new -- to do whatever you want. -- meem
Thanks for your suggestions. I am now fairly satisfied that new and delete are
indeed implemented using malloc and free. However, realloc does not appear to be
implemented using free. So a program such as
int main ()
{
void * ptr = malloc (1000);
realloc (ptr, 0);
}
will catch my DTrace script out. I am in the process of testing an updated
script.
This message posted from opensolaris.org
J.J., If you''re only interested in the heap size, and which threads are changing it, wouldn''t monitoring "syscall::brk:entry" be better, since this is the system call that actually changes the heap. Chip J.J. Shore wrote:> Thanks for your suggestions. I am now fairly satisfied that new and delete are indeed implemented using malloc and free. However, realloc does not appear to be implemented using free. So a program such as > > int main () > { > void * ptr = malloc (1000); > realloc (ptr, 0); > } > > will catch my DTrace script out. I am in the process of testing an updated script. > > > This message posted from opensolaris.org > _______________________________________________ > dtrace-discuss mailing list > dtrace-discuss at opensolaris.org >
There is interesting heap profiling technology from the Sun
Studio tools (compilers). Check collect(1)/analyzer(1).
-r
____________________________________________________________________________________
Performance, Availability & Architecture Engineering
Roch Bourbonnais Sun Microsystems, Icnc-Grenoble
Senior Performance Analyst 180, Avenue De L''Europe, 38330,
Montbonnot Saint Martin, France
http://icncweb.france/~rbourbon http://blogs.sun.com/roller/page/roch
Roch.Bourbonnais at Sun.Com (+33).4.76.18.83.20
Chip Bennett writes:
> J.J.,
>
> If you''re only interested in the heap size, and which threads are
> changing it, wouldn''t monitoring "syscall::brk:entry"
be better, since
> this is the system call that actually changes the heap.
>
> Chip
>
> J.J. Shore wrote:
> > Thanks for your suggestions. I am now fairly satisfied that new and
delete are indeed implemented using malloc and free. However, realloc does not
appear to be implemented using free. So a program such as
> >
> > int main ()
> > {
> > void * ptr = malloc (1000);
> > realloc (ptr, 0);
> > }
> >
> > will catch my DTrace script out. I am in the process of testing an
updated script.
> >
> >
> > This message posted from opensolaris.org
> > _______________________________________________
> > dtrace-discuss mailing list
> > dtrace-discuss at opensolaris.org
> >
>
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org
Chip, That''s an interesting proposition. We have used a monitor on brk to get a rough idea of where the application is growing. However, I believe that the picture can be a bit missleading and little cause grained. Suppose, there are 10Kb of free memory and thread t1 makes a malloc for 8Kb, then thread t2 mallocs 3Kb, and finally t1 mallocs another 8Kb. Thread t1 would be holding 16Kb and t2 3Kb, but the brk (for more memory) would occur in the context of t2. I suspect that if t2 makes a lot of mallocs and frees it would appear that it is using a lot more memory than it actually is because its chances of calling malloc when the heap is low on memory are increased by the number of times it makes the call not by the amount of memory is uses. Do you think that this unlikely to happen in practise? Thanks for the suggestion. I might re-vist this approach. J.J. Shore This message posted from opensolaris.org
It looks like the are a few points that need to be taken account of such as the fact that memory which is allocated by malloc may be freed by other functions such as realloc and that memalign makes a private call to malloc. I am now testing a more reliable version of my script. Hopefully it will also take account of memory managed with new a delete as well. Thanks for all the suggestions. This message posted from opensolaris.org