dtrace discuss - May 2008 - newbie question about tracing malloc/free

Hello,

This is my first post for trace. To be honest I''m total newbie
regarding dtrace :)
I need to trace number of frees and mallocs in my application. Therefore I wrote
my first d script.
---------------------------------------------------------------
BEGIN{
  printf("malloc/free tracking%s\n", execname);
}
 
pid$target:libc:malloc:entry
{
   @table[probefunc] = count();
}
 
 
pid$target:libc:free:entry
{
    @table[probefunc] = count();
}
 
END{
   printa(@table);
}
---------------------------------------------------------------

I invoke script  with following command

[b]dtrace -s mallocfree.d -c ./testapp[/b]

The problem is that I''m not sure if script is ok, since there is
discrepancy between number fo mallocs and frees.
For simple C++ application which have only

int main()
{
 return 0;
}
 
Scripts shows 42 mallocs and 58 frees. The problem is in 16 extra frees - where
do they come from ?
Is it buggy script or there is some other explanation? 
I would also like to know what code is calling mallocs and frees - is there easy
way to trace those info with dtrace?

Many thanks,

Mirko


--
This message posted from opensolaris.org

man malloc
     malloc, calloc, free, memalign, realloc,  valloc,  alloca  -
     memory allocator
So you need to match up your frees with more than just malloc.
calloc, memalign, etc.
rick

On Mon, May 12, 2008 at 06:36:57AM -0700, Mirko Cetkovic
wrote:
> 
> Hello,
> 
> This is my first post for trace. To be honest I''m total newbie
regarding dtrace :)
> I need to trace number of frees and mallocs in my application. Therefore I
wrote my first d script.
> ---------------------------------------------------------------
> BEGIN{
>   printf("malloc/free tracking%s\n", execname);
> }
>  
> pid$target:libc:malloc:entry
> {
>    @table[probefunc] = count();
> }
>  
>  
> pid$target:libc:free:entry
> {
>     @table[probefunc] = count();
> }
>  
> END{
>    printa(@table);
> }
> ---------------------------------------------------------------
> 
> I invoke script  with following command
> 
> [b]dtrace -s mallocfree.d -c ./testapp[/b]
> 
> The problem is that I''m not sure if script is ok, since there is
discrepancy between number fo mallocs and frees.
> For simple C++ application which have only
> 
> int main()
> {
>  return 0;
> }
>  
> Scripts shows 42 mallocs and 58 frees. The problem is in 16 extra frees -
where do they come from ?
> Is it buggy script or there is some other explanation? 
> I would also like to know what code is calling mallocs and frees - is there
easy way to trace those info with dtrace?
> 
> Many thanks,
> 
> Mirko
> 
> 
> --
> This message posted from opensolaris.org
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org
-- 

Rickey C. Weisner 
Software Development and Performance Specialist 
Sun Microsystems, INC
cell phone: 615-308-1147
email: rick.weisner at sun.com

Thank you for your reply.

I have tried your suggestion and something strange happened. I have managed to
crash my solaris box. I just added probes for all allocator functions and when I
added
pid:libc:memalign:entry 
in test script and performed script with simple application

int main()
{
  return 0;
}

Solaris crashed, everything stopped responding and the only solution was reset
button :( I have repeated test and second time Solaris box was crashed as well.

Is it possible that this trivia can kill Solaris ???
I''m using snv_86 on intel Q6600. Even if script is nonsense or buggy it
shouldn''t be able to kill whole OS !? I have executed dtrace as normal
user.

My D trace script is as follows:

---------------------------------------------

BEGIN{
         printf("mem tracking%s\n", execname);
}
 
pid$target:libc:malloc:entry
{
         @table[probefunc] = count();
}
 
pid$target:libc:calloc:entry
{
         @table[probefunc] = count();
}
 
 
pid$target:libc:realloc:entry
{
         @table[probefunc] = count();
}
 
 pid$target:libc:valloc:entry
{
         @table[probefunc] = count();
}
 

pid$target:libc:memalign:entry
{
        @table[probefunc] = count();
}
 
pid$target:libc:free:entry
{
         @table[probefunc] = count();
}
 
END{
         printa(@table);
}

---------------------------------------------

script is invoked 

dtrace -s myscript.d -c ./mytestapp

best regards,
Mirko


--
This message posted from opensolaris.org

Hi Mirko,

This should indeed be impossible. I''m going to continue this
conversation
off-list to gather some data to debug the issue.

Adam

On Mon, May 12, 2008 at 12:20:41PM -0700, Mirko Cetkovic
wrote:
> Thank you for your reply.
> 
> I have tried your suggestion and something strange happened. I have managed
to crash my solaris box. I just added probes for all allocator functions and
when I added
> pid:libc:memalign:entry 
> in test script and performed script with simple application
> 
> int main()
> {
>   return 0;
> }
> 
> Solaris crashed, everything stopped responding and the only solution was
reset button :( I have repeated test and second time Solaris box was crashed as
well.
> 
> Is it possible that this trivia can kill Solaris ???
> I''m using snv_86 on intel Q6600. Even if script is nonsense or
buggy it shouldn''t be able to kill whole OS !? I have executed dtrace
as normal user.
> 
> My D trace script is as follows:
> 
> ---------------------------------------------
> 
> BEGIN{
>          printf("mem tracking%s\n", execname);
> }
>  
> pid$target:libc:malloc:entry
> {
>          @table[probefunc] = count();
> }
>  
> pid$target:libc:calloc:entry
> {
>          @table[probefunc] = count();
> }
>  
>  
> pid$target:libc:realloc:entry
> {
>          @table[probefunc] = count();
> }
>  
>  pid$target:libc:valloc:entry
> {
>          @table[probefunc] = count();
> }
>  
> 
> pid$target:libc:memalign:entry
> {
>         @table[probefunc] = count();
> }
>  
> pid$target:libc:free:entry
> {
>          @table[probefunc] = count();
> }
>  
> END{
>          printa(@table);
> }
> 
> ---------------------------------------------
> 
> script is invoked 
> 
> dtrace -s myscript.d -c ./mytestapp
> 
> best regards,
> Mirko
> 
> 
> --
> This message posted from opensolaris.org
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org
-- 
Adam Leventhal, Fishworks                        http://blogs.sun.com/ahl

Mirko Cetkovic wrote:
> Thank you for your reply.
> 
> I have tried your suggestion and something strange happened. I have
> managed to crash my solaris box. I just added probes for all allocator
if a machine stops responding, we usually say it "hangs" - when we say
a
machine "crashes", we usually refer to the case where the OS goes down
(in
many cases with a panic), and then comes back up by itself (this is 
simplified, but you get the idea ;-)

If you want to help narrow this down, I''d suggest you try the memalign 
probe alone and see how it behaves and whether there''s any output.

HTH
Michael
-- 
Michael Schuster     http://blogs.sun.com/recursion
Recursion, n.: see ''Recursion''

I''d also be curious to know how long you left it...

I''d have expected Dtrace''s watchdog to kick in when they
system went
unresponsive...

Looking forward to Adam''s wrap up on this after he has gathered the
data
he was after. :)

Cheers!

Nathan.


michael schuster wrote:
> Mirko Cetkovic wrote:
>> Thank you for your reply.
>>
>> I have tried your suggestion and something strange happened. I have
>> managed to crash my solaris box. I just added probes for all allocator
> 
> if a machine stops responding, we usually say it "hangs" - when
we say a
> machine "crashes", we usually refer to the case where the OS goes
down (in
> many cases with a panic), and then comes back up by itself (this is 
> simplified, but you get the idea ;-)
> 
> If you want to help narrow this down, I''d suggest you try the
memalign
> probe alone and see how it behaves and whether there''s any output.
> 
> HTH
> Michael

Hi all,

Thanks for your replies.

I will try to give you more detailed insight what is going on when I execute
that script.

@Michael: you are right I wasn''t precise the system hangs 

Script definitely enters into END section since it prints aggregate for malloc
and free items. Terminal hangs and after a while whole computer becomes
unresponsive - only mouse move is working - absolutely no other functionality at
all.
I have left it for couple of minutes in that state (3-5 minutes) and after that
tried to restart X and that action completely hangs machine. I can leave it
longer if you think it can help, but somehow I doubt it will do any good - since
this is the most trivial application to trace.

My box is custom made, I will give you specifications

CPU: Q6600
MB: GA x38-DQ6
RAM: 2x1GB Geil Ultra (paired)
HD: 2x320GB Seagate
GPU: nVidia 8800GTS

I will try memallign probe alone off course :)  

I''m experienced C++ developer who plans to move from Win platform to
Unix and I would really appreciate your help in this migration. I have managed
to convince my chief system architect that Solaris is worth to consider as new
platform for our products. I have to make presentation by the end of the week
and I can''t afford this situation to happen since I''m afraid I
won''t get another chance to convince them if I hang the presentation
box :)
I''m involved in writing middle ware applications in my company -
specialized for networking.
Therefore I was planning to present asynchronous Socketing on Solaris, compare
its performance with Windows AIO with Completion Port model and on top of all
demonstrate Dtrace as ultimate tool for app tracing, administration and
possibility to create scripts which can help first line support team in
diagnosing bottlenecks.

So, I would kindly ask you to feel free to instruct me to do whatever it takes
to provide you sufficient information about this matter.

Best regards
Mirko


--
This message posted from opensolaris.org

Mirko Cetkovic wrote:
> Hi all,
> 
> Thanks for your replies.
> 
> I will try to give you more detailed insight what is going on when I
> execute that script.
> 
> @Michael: you are right I wasn''t precise the system hangs
> 
> Script definitely enters into END section since it prints aggregate for
> malloc and free items. Terminal hangs and after a while whole computer
> becomes unresponsive - only mouse move is working - absolutely no other
> functionality at all.
a few thoughts:

- does the machine respond to ping, or is it reachable via ssh/telnet/rsh?
- what you describe sounds like some part of the machine is still working, 
so I wonder if you''re doing/running *anything* else at the time the
machine
hangs.
- is this easily reproducable and are the results always the same?
- which compiler and switches are you using to build your test app?
- have you tried using the -o option for dtrace (to write output to a file)?
- you can add "set snooping=1" to /etc/system. In the case that the
machine
is so badly hung that the clock stops working, this will panic the machine. 
(works only after next reboot).

HTH
Michael
-- 
Michael Schuster     http://blogs.sun.com/recursion
Recursion, n.: see ''Recursion''

Michael thank you for your prompt reply 

Regarding your questions:

- I don''t know if it was able to ping my box or ssh to it in moment of
hung. It is my home computer and I don''t have other one to try ssh or
ping. I will bring laptop from office and try that tonight.

- it is easily reproducible just boot, start terminal and type
:)
>dtrace -s myscript.d -c ./mytestapp  
- mytestapp is compiled without any switch

CC -o mytestapp mytestapp.cc

- Regarding -o switch and "set snooping=1" I will try it tonight as
well

The issue which bothers me is that it is possible to get unresponsive system
during application tracing which is not in line with statement that I gave to my
bosses that dtrace allows tracing with minimal system impact :))) If it proves
that my stupid example hangs box for minutes it will be enough reason to reject
all my efforts in convincing them to switch to solaris :(

I have to misuse this thread for another question:
Q: Where can I discuss asynchronous socketing on [Open]Solaris ? 

regards
M.


--
This message posted from opensolaris.org

Mirko Cetkovic wrote:
> - mytestapp is compiled without any switch
> 
> CC -o mytestapp mytestapp.cc
why are you using C++ compiler for a simple C program (not that it should 
make a difference ... but who knows?)?
> - Regarding -o switch and "set snooping=1" I will try it tonight
as well
> 
> 
> The issue which bothers me is that it is possible to get unresponsive
> system during application tracing which is not in line with statement
> that I gave to my bosses that dtrace allows tracing with minimal system
> impact :))) If it proves that my stupid example hangs box for minutes it
> will be enough reason to reject all my efforts in convincing them to
> switch to solaris :(
one can overload any system :-)

you need to be careful with sweeping statements, as you''ve seen, since 
there''s frequently some way to prove an exception exists!

I once made the mistake of typing
	pid$target:::
where I meant
	pid$target:::entry,pid$target:::return
(because in the kernel''s fbt provider, which was all I''d seen
at the time,
that would have been the same), and the machine hung trying to instrument 
*every instruction* in the target process.

IMO, DTrace is superior to any other technology in the same area, and, when 
used appropriately, probe impact is minimal.
> I have to misuse this thread for another question: Q: Where can I
> discuss asynchronous socketing on [Open]Solaris ?
that''d be networking-discuss at opensolaris.org

Michael
-- 
Michael Schuster     http://blogs.sun.com/recursion
Recursion, n.: see ''Recursion''

> why are you using C++ compiler for a simple C program
> (not that it should 
> make a difference ... but who knows?)?
I have  created two C++ applications for asynchronous networking in Solaris
tried mem/free counting with dtrace, discovered discrepancy and therefore I
wanted to see what is going on with empty C++ application.
> one can overload any system :-)
> 
> you need to be careful with sweeping statements, as
> you''ve seen, since 
> there''s frequently some way to prove an exception
> exists!
Well we all get reckless sometimes ;) I''m convinced that all this fuss
has nothing to do with dtrace bugs or any other Solaris issue. It is some my
stupid noob error which is causing this odd behavior.
> 
> that''d be networking-discuss at opensolaris.org
> 
thanks - posted there :)

Many thanks
Mirko


--
This message posted from opensolaris.org

I just came home from work and tried that damn script once again and everything
went OK ???!! Yesterday box was dead twice in a row after reboot.
Now I can suspect on HW issue... damn.
Anyway thanks a million to all of you who tried to help me. I do apologize
because of this fuss I made :(

best regards
Mirko


--
This message posted from opensolaris.org