dtrace discuss - Feb 2008 - How to know exact how much memory app is using currently ?

Hi,
  In performance test, my application (multi-thread programme) expriences a
memory consumption peak because there are at most e.g. 100 clients communicate
with it.  My question is, why after all clients exit, when application has
noting to do, according to result of tool "pstack" or
"pmap", my application still occupy a big amount of memory ?

My application use libmtmalloc.so.1 for memory allocation.

I found description "After free() is executed, this space is made available
for further  allocation  by  the application, though not returned to the system.
Memory is returned to the system only upon  termination of  the 
application"
in the second paragraph of "malloc"  man page;    and description
"After free() is performed this space is available for  further 
allocation" in the thrid paragraph of "mtmalloc" man page.

According to the descriptions, it seems "mtmalloc" is exactly
different from "malloc", and if using mtmalloc, after
"free()" the consumed memory is returned to system immediately, since
in man page of "mtmalloc" it doesn''t emphasize the memory is
not returned to system after "free()" is called until application
termination.  Right ?

But in our performance test we found "mtmalloc" has the same behaviour
with "malloc".

BR/LiJian


--
This message posted from opensolaris.org

LiJian,

On Mon, Feb 25, 2008 at 2:33 PM, LiJIan <lijianwhq at 163.com>
wrote:
> Hi,
>   In performance test, my application (multi-thread programme) expriences a
memory consumption peak because there are at most e.g. 100 clients communicate
with it.  My question is, why after all clients exit, when application has
noting to do, according to result of tool "pstack" or
"pmap", my application still occupy a big amount of memory ?
>
>  My application use libmtmalloc.so.1 for memory allocation.
>
>  I found description "After free() is executed, this space is made
available for further  allocation  by  the application, though not returned to
the system. Memory is returned to the system only upon  termination of  the 
application"
>  in the second paragraph of "malloc"  man page;    and
description "After free() is performed this space is available for  further
allocation" in the thrid paragraph of "mtmalloc" man page.
>
>  According to the descriptions, it seems "mtmalloc" is exactly
different from "malloc", and if using mtmalloc, after
"free()" the consumed memory is returned to system immediately, since
in man page of "mtmalloc" it doesn''t emphasize the memory is
not returned to system after "free()" is called until application
termination.  Right ?
>
>  But in our performance test we found "mtmalloc" has the same
behaviour with "malloc".
>
>  BR/LiJian
I believe mtmalloc behaves exactly the same as the default malloc
subsystem in this regard: in both cases, the free() call will not
release the physical memory back to OS - otherwise the (mt)malloc/free
call will be very expensive (i.e. slow).

Tao

There are smarter people than me that can go into better detail, but
I''ll give it a shot.  All the memory allocators you''re usually
going
to see grow your address space using brk(1)/sbrk(1).  They generally
don''t shrink the address space back down.

If you really need the address space to shrink (the common solution is
just to let those pages get pushed out to the VM), try mmap() on a
temporary file (or /dev/zero, but I donno if that works on solaris)
and allocate memory out of that.  When you''re done with that, just
unmap the file and the address space is freed.  GNU malloc does this
in certain situations.


On Mon, Feb 25, 2008 at 1:33 AM, LiJIan <lijianwhq at 163.com>
wrote:
> Hi,
>   In performance test, my application (multi-thread programme) expriences a
memory consumption peak because there are at most e.g. 100 clients communicate
with it.  My question is, why after all clients exit, when application has
noting to do, according to result of tool "pstack" or
"pmap", my application still occupy a big amount of memory ?
>
>  My application use libmtmalloc.so.1 for memory allocation.
>
>  I found description "After free() is executed, this space is made
available for further  allocation  by  the application, though not returned to
the system. Memory is returned to the system only upon  termination of  the 
application"
>  in the second paragraph of "malloc"  man page;    and
description "After free() is performed this space is available for  further
allocation" in the thrid paragraph of "mtmalloc" man page.
>
>  According to the descriptions, it seems "mtmalloc" is exactly
different from "malloc", and if using mtmalloc, after
"free()" the consumed memory is returned to system immediately, since
in man page of "mtmalloc" it doesn''t emphasize the memory is
not returned to system after "free()" is called until application
termination.  Right ?
>
>  But in our performance test we found "mtmalloc" has the same
behaviour with "malloc".
>
>  BR/LiJian
>
>
>  --
>  This message posted from opensolaris.org
>  _______________________________________________
>  dtrace-discuss mailing list
>  dtrace-discuss at opensolaris.org
>


-- 
H. Lally Singh
Ph.D. Candidate, Computer Science
Virginia Tech

> There are smarter people than me that can go into
> better detail, but
> I''ll give it a shot.  All the memory allocators
> you''re usually going
> to see grow your address space using brk(1)/sbrk(1).
>  They generally
> on''t shrink the address space back down.
> 
> If you really need the address space to shrink (the
> common solution is
> just to let those pages get pushed out to the VM),
> try mmap() on a
> temporary file (or /dev/zero, but I donno if that
> works on solaris)
> and allocate memory out of that.  When you''re done
> with that, just
> unmap the file and the address space is freed.  GNU
> malloc does this
> in certain situations.
It does work on Solaris; see
http://docs.sun.com/app/docs/doc/819-2254/zero-7d?a=view

where it says
> Mapping a zero special file creates a zero-initialized unnamed memory
> object of a length equal to the length of the mapping and rounded up to
> the nearest page size as returned by sysconf. Multiple processes can share
> such a zero special file object provided a common ancestor mapped the
> object MAP_SHARED.

--
This message posted from opensolaris.org

> > There are smarter people than me that can go into
> > better detail, but
> > I''ll give it a shot.  All the memory allocators
> > you''re usually going
> > to see grow your address space using
> brk(1)/sbrk(1).
> >  They generally
> > on''t shrink the address space back down.
> > 
> > If you really need the address space to shrink
> (the
> > common solution is
> > just to let those pages get pushed out to the VM),
> > try mmap() on a
> > temporary file (or /dev/zero, but I donno if that
> > works on solaris)
> > and allocate memory out of that.  When you''re done
> > with that, just
> > unmap the file and the address space is freed.
>  GNU
>  malloc does this
>  in certain situations.
> 
> It does work on Solaris; see
> http://docs.sun.com/app/docs/doc/819-2254/zero-7d?a=vi
> ew
> 
> where it says
> 
> > Mapping a zero special file creates a
> zero-initialized unnamed memory
> > object of a length equal to the length of the
> mapping and rounded up to
> > the nearest page size as returned by sysconf.
> Multiple processes can share
> > such a zero special file object provided a common
> ancestor mapped the
> > object MAP_SHARED.
P.S. back in Solaris 8 or so, munmap(2) of large files (or in 64-bit
programs?) used to be very slow, not something one wanted to do
frequently.  But that''s supposed to have been fixed at some point in
Solaris 9 (and in some Solaris 8 patch, I think).
>From Solaris 8 onward, there''s a shortcut for anonymous memory:
rather than opening /dev/zero and applying mmap(2) to the resulting
file descriptor, one can pass to mmap(2) a file descriptor of -1 and
include the flag MAP_ANON in those one ORs together, and get the same
result (but with one system call instead of two, and without the file
descriptor lying around needing a third system call to close).

Most (but not necessarily all) other OSs also support MAP_ANON, although
the Single Unix Specification does not mention it (only mentions
MAP_SHARED, MAP_PRIVATE, and MAP_FIXED).  Pre-2.4, Linux may
not have supported MAP_ANON|MAP_SHARED*.  Other OSs may support
additional mmap() flags that Solaris doesn''t know about.

* I don''t know this from personal experience, just stumbled across
mention
of it trying to get a general feel for how widespread MAP_ANON was.


--
This message posted from opensolaris.org