dtrace discuss - Jan 2008 - Getting total memory allocated so far

Hey all, I''ve been trying to track memory usage in an application.
I''ve got probes into malloc() and free(), but I''d like to just
get a
total allocated amount at any given time (as malloc & free only give
me deltas, and buffer filling can make my running count worthless).

Is there any probe for that?  This is on Solaris 10, and I can
recompile the app as needed (but there are too many calls to malloc()
to directly instrument them or what not).  Would libumem help with
this?

Thanks in advance!

Btw, dtrace is a godsend.  Like a year less working on my PhD godsend.

-- 
H. Lally Singh
Ph.D. Candidate, Computer Science
Virginia Tech

Can you be more specific about "track memory usage"???
There''s a broad range of things that could fall into tracking memory
usage.
For example, if you''re just interested in tracking total physical
and/or
virtual
address space sizes, you can use prstat(1). If you''re interested in the
various
segments, you could take some pmap(1) samples from the target process
periodically.

What, precisely, would you like to track?

Thanks,
/jim


Lally Singh wrote:
> Hey all, I''ve been trying to track memory usage in an application.
> I''ve got probes into malloc() and free(), but I''d like to
just get a
> total allocated amount at any given time (as malloc & free only give
> me deltas, and buffer filling can make my running count worthless).
>
> Is there any probe for that?  This is on Solaris 10, and I can
> recompile the app as needed (but there are too many calls to malloc()
> to directly instrument them or what not).  Would libumem help with
> this?
>
> Thanks in advance!
>
> Btw, dtrace is a godsend.  Like a year less working on my PhD godsend.
>
>

I''d like to track to total amount of memory the application''s
actually
using right now.   More specifically, I guess these rules would do it:

1. start at M=0
2. for each call to malloc(): set M += size of block returned by malloc
3. for each call to free(): set M -= size of block freed.

I''m using the term ''size of block'' to account for
malloc rounding the
amount requested up to its nearest internal block size.

Pretending that malloc() rounds up to 16 byte increments: if the app
called a=malloc(28), b=malloc(60), free(a), then M=64 (from the
still-used malloc of b, which was rounded up to 64).

So really, just how much memory the app''s using via malloc/free right
now.

Thanks,


On Jan 2, 2008 10:44 AM, Jim Mauro <James.Mauro at sun.com>
wrote:
>
> Can you be more specific about "track memory usage"???
> There''s a broad range of things that could fall into tracking
memory usage.
> For example, if you''re just interested in tracking total physical
and/or
> virtual
> address space sizes, you can use prstat(1). If you''re interested
in the
> various
> segments, you could take some pmap(1) samples from the target process
> periodically.
>
> What, precisely, would you like to track?
>
> Thanks,
> /jim
>
>
>
> Lally Singh wrote:
> > Hey all, I''ve been trying to track memory usage in an
application.
> > I''ve got probes into malloc() and free(), but I''d
like to just get a
> > total allocated amount at any given time (as malloc & free only
give
> > me deltas, and buffer filling can make my running count worthless).
> >
> > Is there any probe for that?  This is on Solaris 10, and I can
> > recompile the app as needed (but there are too many calls to malloc()
> > to directly instrument them or what not).  Would libumem help with
> > this?
> >
> > Thanks in advance!
> >
> > Btw, dtrace is a godsend.  Like a year less working on my PhD godsend.
> >
> >
>


-- 
H. Lally Singh
Ph.D. Candidate, Computer Science
Virginia Tech

Lally Singh writes:
> I''d like to track to total amount of memory the
application''s actually
> using right now.   More specifically, I guess these rules would do it:
> 
> 1. start at M=0
> 2. for each call to malloc(): set M += size of block returned by malloc
> 3. for each call to free(): set M -= size of block freed.
Using "env LD_PRELOAD=libumem.so myapp" with mdb''s umem
support should
get you that (plus quite a bit more).

-- 
James Carlson, Solaris Networking              <james.d.carlson at
sun.com>
Sun Microsystems / 35 Network Drive        71.232W   Vox +1 781 442 2084
MS UBUR02-212 / Burlington MA 01803-2757   42.496N   Fax +1 781 442 1677

Lally Singh wrote:
> So really, just how much memory the app''s using via malloc/free
right now.
You should bear in mind that Unices traditionally support malloc/free
library calls with a "brk" or "sbrk" syscall and, for
efficiency,
the tracking is not exact.  In fact it is common to *never* release
memory to the kernel via the syscall - only by the process exit or exec.

So while tracking what you ask is possible, it might not be what you want.

Cheers,
   Jeremy

@Jeremy: yeah, I''m tracking how much memory it''s actually
using right
now.. sbrk makes it complicated, but I figure that more advanced
memory management may help, once I''ve measured how it''s
getting used
today.  Via mmap() for large transient allocations and/or some
smartness about allocating smartly to keep the working set of pages
reasonable.

@James: I saw that, but could I use it on a running program without
stopping it?  The dtrace scripts don''t do much damage, and the program
still gets all its work done at ~60 Hz for its clients.  Single
threaded, no less.

Some more details: I was trying to keep this all a simple question,
but with memory there rarely are simple questions.  My research is on
the scalability of massively multiplayer online games.  So, I''ve
written a load simulator which logs in from remote computers and
starts playing the game.  I''ve got a bunch of dtrace scripts that
measure the network, memory, and cpu usage as I add more simulated
clients.  The scripts haven''t shown themselves to interfere
significantly with performance (yet), but in exchange they put out
data once a second, which another script reads & shoves into a db for
me.

I''d like to monitor memory usage without interfering with my other
measurements.  Hence, I''d love to know something close to a global var
(which''d be great, malloc() already has to do some locking (for normal
malloc, haven''t seen umem), so why not?) or some way to quickly
assemble a running total.  Right now all I''ve got is a pid provider
probe for malloc & free that reads the headers on the memory blocks
they use.

Thanks in advance everyone!

On Jan 2, 2008 1:25 PM, James Carlson <james.d.carlson at sun.com>
wrote:
> Lally Singh writes:
> > I''d like to track to total amount of memory the
application''s actually
> > using right now.   More specifically, I guess these rules would do it:
> >
> > 1. start at M=0
> > 2. for each call to malloc(): set M += size of block returned by
malloc
> > 3. for each call to free(): set M -= size of block freed.
>
> Using "env LD_PRELOAD=libumem.so myapp" with mdb''s umem
support should
> get you that (plus quite a bit more).
>
> --
> James Carlson, Solaris Networking              <james.d.carlson at
sun.com>
> Sun Microsystems / 35 Network Drive        71.232W   Vox +1 781 442 2084
> MS UBUR02-212 / Burlington MA 01803-2757   42.496N   Fax +1 781 442 1677
>


-- 
H. Lally Singh
Ph.D. Candidate, Computer Science
Virginia Tech

Lally Singh writes:
> @James: I saw that, but could I use it on a running program without
> stopping it?
You can attach to the process using "mdb -p <pid>" or use
"gcore" to
get a core-file snapshot of the running process.

Something like:

	echo ::umastat | mdb -p `pgrep myapp`
> I''d like to monitor memory usage without interfering with my other
> measurements.  Hence, I''d love to know something close to a global
var
> (which''d be great, malloc() already has to do some locking (for
normal
> malloc, haven''t seen umem), so why not?) or some way to quickly
> assemble a running total.  Right now all I''ve got is a pid
provider
> probe for malloc & free that reads the headers on the memory blocks
> they use.
If all you want is a "quick" and non-intrusive answer, then
I''d think
that ps(1) or vmstat(1M) would be a good tool.  Using
mallinfo(3MALLOC) functions would provide more details, at the expense
of getting the application involved.

Dtrace can do it, but you''d be tracing every single malloc, calloc,
free, memalign, realloc, and valloc call to get the numbers, and then
keeping associative arrays to monitor allocations.  It sounds somewhat
expensive to me.

Still another approach would be to write your own interposition
library that keeps the statistics you want.  I''m not sure why
instantaneous memory allocation would be an interesting value, but it
sounds like it''d be doable that way.

Yet another would be to modify malloc/free in libc.  You have the
source ...

-- 
James Carlson, Solaris Networking              <james.d.carlson at
sun.com>
Sun Microsystems / 35 Network Drive        71.232W   Vox +1 781 442 2084
MS UBUR02-212 / Burlington MA 01803-2757   42.496N   Fax +1 781 442 1677

If you are wanting heap size or rss then.
man -s 4 proc
pstatus_t structure, status  file
           size_t pr_brksize;       /* size of the process heap, in bytes */
and
psinfo_t,  psinfo file
          size_t pr_rssize;        /* resident set size in Kbytes */

Contact me offline if you would like sample code.
But pmap -x gives you all that already.
rick

On Thu, Jan 03, 2008 at 02:15:48AM -0500, Lally Singh
wrote:
> Date: Thu, 03 Jan 2008 02:15:48 -0500
> From: Lally Singh <lally.singh at gmail.com>
> Subject: Re: [dtrace-discuss] Getting total memory allocated so far
> In-reply-to: <18299.55046.472427.139710 at gargle.gargle.HOWL>
> To: James Carlson <James.D.Carlson at Sun.COM>, jgh at wizmail.org
> Cc: dtrace-discuss at opensolaris.org, Jim Mauro <James.Mauro at
Sun.COM>
> Errors-to: dtrace-discuss-bounces at opensolaris.org
> Precedence: list
> X-BeenThere: dtrace-discuss at opensolaris.org
> Delivered-to: dtrace-discuss at opensolaris.org
> X-PMX-Version: 5.2.0.264296
> X-Brightmail-Tracker: AAAAAA=> X-Spam-Checker-Version: SpamAssassin
3.2.3 (2007-08-08) on
> 	oss-mail1.opensolaris.org
> X-Original-To: dtrace-discuss at opensolaris.org
> X-Antispam: No, score=-0.4/5.0, scanned in 0.056sec at (localhost
[127.0.0.1])
> 	by smf-spamd v1.3.1 - http://smfs.sf.net/
> X-Mailman-Version: 2.1.9
> List-Post: <mailto:dtrace-discuss at opensolaris.org>
> List-Subscribe:
<http://mail.opensolaris.org/mailman/listinfo/dtrace-discuss>,
> 	<mailto:dtrace-discuss-request at opensolaris.org?subject=subscribe>
> List-Unsubscribe:
>  <http://mail.opensolaris.org/mailman/listinfo/dtrace-discuss>,
> 	<mailto:dtrace-discuss-request at
opensolaris.org?subject=unsubscribe>
> List-Archive: <http://mail.opensolaris.org/pipermail/dtrace-discuss>
> List-Help: <mailto:dtrace-discuss-request at
opensolaris.org?subject=help>
> List-Id: DTrace General Discussion <dtrace-discuss.opensolaris.org>
> X-Spam-Status: No, score=0.0 required=5.0 tests=AWL autolearn=unavailable
> 	version=3.2.3
> X-Spam-Level:
> 
> @Jeremy: yeah, I''m tracking how much memory it''s actually
using right
> now.. sbrk makes it complicated, but I figure that more advanced
> memory management may help, once I''ve measured how it''s
getting used
> today.  Via mmap() for large transient allocations and/or some
> smartness about allocating smartly to keep the working set of pages
> reasonable.
> 
> @James: I saw that, but could I use it on a running program without
> stopping it?  The dtrace scripts don''t do much damage, and the
program
> still gets all its work done at ~60 Hz for its clients.  Single
> threaded, no less.
> 
> Some more details: I was trying to keep this all a simple question,
> but with memory there rarely are simple questions.  My research is on
> the scalability of massively multiplayer online games.  So, I''ve
> written a load simulator which logs in from remote computers and
> starts playing the game.  I''ve got a bunch of dtrace scripts that
> measure the network, memory, and cpu usage as I add more simulated
> clients.  The scripts haven''t shown themselves to interfere
> significantly with performance (yet), but in exchange they put out
> data once a second, which another script reads & shoves into a db for
> me.
> 
> I''d like to monitor memory usage without interfering with my other
> measurements.  Hence, I''d love to know something close to a global
var
> (which''d be great, malloc() already has to do some locking (for
normal
> malloc, haven''t seen umem), so why not?) or some way to quickly
> assemble a running total.  Right now all I''ve got is a pid
provider
> probe for malloc & free that reads the headers on the memory blocks
> they use.
> 
> Thanks in advance everyone!
> 
> On Jan 2, 2008 1:25 PM, James Carlson <james.d.carlson at sun.com>
wrote:
> > Lally Singh writes:
> > > I''d like to track to total amount of memory the
application''s actually
> > > using right now.   More specifically, I guess these rules would
do it:
> > >
> > > 1. start at M=0
> > > 2. for each call to malloc(): set M += size of block returned by
malloc
> > > 3. for each call to free(): set M -= size of block freed.
> >
> > Using "env LD_PRELOAD=libumem.so myapp" with mdb''s
umem support should
> > get you that (plus quite a bit more).
> >
> > --
> > James Carlson, Solaris Networking              <james.d.carlson at
sun.com>
> > Sun Microsystems / 35 Network Drive        71.232W   Vox +1 781 442
2084
> > MS UBUR02-212 / Burlington MA 01803-2757   42.496N   Fax +1 781 442
1677
> >
> 
> 
> 
> -- 
> H. Lally Singh
> Ph.D. Candidate, Computer Science
> Virginia Tech
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org
-- 

Rickey C. Weisner 
Software Development and Performance Specialist 
Sun Microsystems, INC
cell phone: 615-308-1147
email: rick.weisner at sun.com

I hate to beat this to death, but it really sounds like you can make your
life simple and use prstat and pmap to track the information you want.

Heap memory usage tracking is tricky, since libc malloc may or may not
call into the kernel (sbrk) to satisfy a memory request. Also, tracking
malloc and keeping a running total in dtrace is dead easy, but tracking
free and correlating to how much space was freed is tricky, since
free takes an address, not a size. You could build an array of addresses
in a pid<PID:libc:malloc:return probe clause, but making use of it to
track free''s and sizes is tricky...

It really all gets down to the level of granularity and accuracy that you
require. prstat and pmap will provide the information you need with
reasonable accuracy - pmap will allow you to track heap segment sizes
(pmap -x <PID> | grep heap) - you could easily script something with
pmap and generate a summary report.

Finally, as Rickey pointed out, you could also write some C code that
uses procfs and grabs the desired data.

Thanks,
/jim


Lally Singh wrote:
> @Jeremy: yeah, I''m tracking how much memory it''s actually
using right
> now.. sbrk makes it complicated, but I figure that more advanced
> memory management may help, once I''ve measured how it''s
getting used
> today.  Via mmap() for large transient allocations and/or some
> smartness about allocating smartly to keep the working set of pages
> reasonable.
>
> @James: I saw that, but could I use it on a running program without
> stopping it?  The dtrace scripts don''t do much damage, and the
program
> still gets all its work done at ~60 Hz for its clients.  Single
> threaded, no less.
>
> Some more details: I was trying to keep this all a simple question,
> but with memory there rarely are simple questions.  My research is on
> the scalability of massively multiplayer online games.  So, I''ve
> written a load simulator which logs in from remote computers and
> starts playing the game.  I''ve got a bunch of dtrace scripts that
> measure the network, memory, and cpu usage as I add more simulated
> clients.  The scripts haven''t shown themselves to interfere
> significantly with performance (yet), but in exchange they put out
> data once a second, which another script reads & shoves into a db for
> me.
>
> I''d like to monitor memory usage without interfering with my other
> measurements.  Hence, I''d love to know something close to a global
var
> (which''d be great, malloc() already has to do some locking (for
normal
> malloc, haven''t seen umem), so why not?) or some way to quickly
> assemble a running total.  Right now all I''ve got is a pid
provider
> probe for malloc & free that reads the headers on the memory blocks
> they use.
>
> Thanks in advance everyone!
>
> On Jan 2, 2008 1:25 PM, James Carlson <james.d.carlson at sun.com>
wrote:
>   
>> Lally Singh writes:
>>     
>>> I''d like to track to total amount of memory the
application''s actually
>>> using right now.   More specifically, I guess these rules would do
it:
>>>
>>> 1. start at M=0
>>> 2. for each call to malloc(): set M += size of block returned by
malloc
>>> 3. for each call to free(): set M -= size of block freed.
>>>       
>> Using "env LD_PRELOAD=libumem.so myapp" with mdb''s
umem support should
>> get you that (plus quite a bit more).
>>
>> --
>> James Carlson, Solaris Networking              <james.d.carlson at
sun.com>
>> Sun Microsystems / 35 Network Drive        71.232W   Vox +1 781 442
2084
>> MS UBUR02-212 / Burlington MA 01803-2757   42.496N   Fax +1 781 442
1677
>>
>>     
>
>
>
>

Yeah, I know it''d be a lot easier.  But there are two issues:

1. The program tends to allocate a lot at the beginning (when a client
logs in, to transfer the game level over), and trickle-free it after
that.  So a lot of the memory it initially allocates is unused for
quite some time (as the client acknowledges receipt of the data
packets, the data is freed).

2. I''m trying to track the memory allocations/frees relative to
specific events in the code (e.g. someone getting hit by a projectile,
etc).  The granularity for this stuff is pretty tight.  Some state
variables linked to the place in the code where the program is (single
threaded) when the malloc/free occurs tells me a lot.  Makes it easy
to tell which portions of the program are responsible for most of the
memory allocations.

I''ve already implemented a dtrace script that properly tracks malloc
&
free sizes (reading in the block size from copyin(pointer-8,4)).  If
anyone''s interested in that script, I''ll happily post it.

But, combining that with the mdb trick above may do it for me.  Thanks everyone!


On Jan 3, 2008 11:27 AM, Jim Mauro <James.Mauro at sun.com>
wrote:
>
> I hate to beat this to death, but it really sounds like you can make your
> life simple and use prstat and pmap to track the information you want.
>
> Heap memory usage tracking is tricky, since libc malloc may or may not
> call into the kernel (sbrk) to satisfy a memory request. Also, tracking
> malloc and keeping a running total in dtrace is dead easy, but tracking
> free and correlating to how much space was freed is tricky, since
> free takes an address, not a size. You could build an array of addresses
> in a pid<PID:libc:malloc:return probe clause, but making use of it to
> track free''s and sizes is tricky...
>
> It really all gets down to the level of granularity and accuracy that you
> require. prstat and pmap will provide the information you need with
> reasonable accuracy - pmap will allow you to track heap segment sizes
> (pmap -x <PID> | grep heap) - you could easily script something with
> pmap and generate a summary report.
>
> Finally, as Rickey pointed out, you could also write some C code that
> uses procfs and grabs the desired data.
>
> Thanks,
> /jim
>
>
> Lally Singh wrote:
>
> > @Jeremy: yeah, I''m tracking how much memory it''s
actually using right
> > now.. sbrk makes it complicated, but I figure that more advanced
> > memory management may help, once I''ve measured how
it''s getting used
> > today.  Via mmap() for large transient allocations and/or some
> > smartness about allocating smartly to keep the working set of pages
> > reasonable.
> >
> > @James: I saw that, but could I use it on a running program without
> > stopping it?  The dtrace scripts don''t do much damage, and
the program
> > still gets all its work done at ~60 Hz for its clients.  Single
> > threaded, no less.
> >
> > Some more details: I was trying to keep this all a simple question,
> > but with memory there rarely are simple questions.  My research is on
> > the scalability of massively multiplayer online games.  So,
I''ve
> > written a load simulator which logs in from remote computers and
> > starts playing the game.  I''ve got a bunch of dtrace scripts
that
> > measure the network, memory, and cpu usage as I add more simulated
> > clients.  The scripts haven''t shown themselves to interfere
> > significantly with performance (yet), but in exchange they put out
> > data once a second, which another script reads & shoves into a db
for
> > me.
> >
> > I''d like to monitor memory usage without interfering with my
other
> > measurements.  Hence, I''d love to know something close to a
global var
> > (which''d be great, malloc() already has to do some locking
(for normal
> > malloc, haven''t seen umem), so why not?) or some way to
quickly
> > assemble a running total.  Right now all I''ve got is a pid
provider
> > probe for malloc & free that reads the headers on the memory
blocks
> > they use.
> >
> > Thanks in advance everyone!
> >
> > On Jan 2, 2008 1:25 PM, James Carlson <james.d.carlson at
sun.com> wrote:
> >
> >> Lally Singh writes:
> >>
> >>> I''d like to track to total amount of memory the
application''s actually
> >>> using right now.   More specifically, I guess these rules
would do it:
> >>>
> >>> 1. start at M=0
> >>> 2. for each call to malloc(): set M += size of block returned
by malloc
> >>> 3. for each call to free(): set M -= size of block freed.
> >>>
> >> Using "env LD_PRELOAD=libumem.so myapp" with
mdb''s umem support should
> >> get you that (plus quite a bit more).
> >>
> >> --
> >> James Carlson, Solaris Networking              <james.d.carlson
at sun.com>
> >> Sun Microsystems / 35 Network Drive        71.232W   Vox +1 781
442 2084
> >> MS UBUR02-212 / Burlington MA 01803-2757   42.496N   Fax +1 781
442 1677
> >>
> >>
> >
> >
> >
> >
>


-- 
H. Lally Singh
Ph.D. Candidate, Computer Science
Virginia Tech

Thanks for the update. You probably already know this,
but...
>
> 2. I''m trying to track the memory allocations/frees relative to
> specific events in the code (e.g. someone getting hit by a projectile,
> etc).  The granularity for this stuff is pretty tight.  Some state
> variables linked to the place in the code where the program is (single
> threaded) when the malloc/free occurs tells me a lot.  Makes it easy
> to tell which portions of the program are responsible for most of the
> memory allocations.
Grabbing a ustack() in the pid<PID>:libc:malloc:entry and 
pid<PID>:libc:free:entry
probes is your friend here...

Thanks,
/jim