dtrace discuss - Jun 2008 - FW: Memory Leak Problem in My Application running on Solaris 10.

Hi,

 This is regarding Dtrace usability for memory leak detection.

 We have real-time application written C++ which runs on Solaris 10  
  having a problem that''s the my application  grows in size from 130 Mb
to 450Mb in around 15 days.

 So there is two possibilities with the application growth of memory 
 due to Size growth of Dictionary Objects (Like Maps) and Memory Leak.

 (Dictionary Objects we can print the size but its live production 
 site where the problem is happening also there are many such object 
 used the application)

 When I used  Dtrace and libumem ( with mdb )to detect memory leak for
C++ code. 

 When I tried to run dtrace script available in SUN Site it stops
abruptly
 With following error

 # dtrace -s ./CCagg_1.d `pgrep scadadbproc` | c++filt >>
 scadadbproc_dtrace
 dtrace: 91102 drops on CPU 0
 dtrace: processing aborted: Abort due to systemic unresponsiveness

 While suing mdb ( ::findleaks ) 
  It says no memory allocation done and says no leaks found.

1) Is there any way where I can restrict dtrace to look for probe only
certain source files or functions?

2) Is there any way to handle such problems?

3) I got a clue of using "libgc.so" which acts as garbage collector is
   there any problem in using and how reliable and effective is this.
   ( it works fine on sample application but not improvement in real
application ) . I just linked the library "libgc.so" at compile time ?
Anything else need to done for this ?



Your inputs will be highly appreciated. Please do share your thoughts 
and techniques.



 Thanks & Regards

 S L Srikant

 SCADA Business Unit
 Invensys Development Centre India Pvt. Ltd.
 S-210, Plot # 17, Orion Building, Block J
 Vanenburg IT Park, Software Units Layout
 Madhapur, Hyderabad - 500 081.




Hi Jim,

    Thanks you for the inputs .
    I am trying with libumem based on the results I will post on  
    dtrace-discuss at opensolaris.org .

    Once again thank you very much.

rgds
Subject: Re: Regarding DTrace.

Hello - I''d recommend posting to a broader audience. You''ll
get more
people with a lot of expertise. dtrace-discuss at opensolaris.org.

The machine you are running on must be very busy. DTrace implements
a watchdog mechanism to determine if the system is getting extremely
slow
or hung. If dtrace detects an unresponsive system (which it did in your 
case),
it will exit. You have three options here:

1. Run the code in a test enviroment on a machine that is less busy.

2. Run dtrace in destructive mode. This disables the watchdog function.
    (check the dtrace manual for how to do this).

3. Do not use dtrace to troubleshoot this memory leak - use libumem.

Thanks,
/jim


Sana, Srikant wrote:
> Hi Jim ,
>
>   Thanks you very much for the inputs.
>
>   I just tried with dtrace but it gives the following error after
> running  
>   for 2 mins and the size of the output file generated is also around
> 9MB.
>
> # dtrace -s ./CCagg_1.d `pgrep scadadbproc` | c++filt >>
> scadadbproc_dtrace
> dtrace: 91102 drops on CPU 0
> dtrace: processing aborted: Abort due to systemic unresponsiveness
>
> I tried analyzing the output file which where memory allocation is
done
> and freed. 
>
> The memory leak happens in a particular in scenario but the script get
> aborted immediately, the issue is we don''t when the leak happens
to
> identify that we have to run the tool on continuous manner without
> interruption.
>
> Please let me know how I can achieve this. 
>
>
> Thanks & Regards
> S L Srikant
>  
> SCADA Business Unit
> Invensys Development Centre India Pvt. Ltd.
> S-210, Plot # 17, Orion Building, Block J
> Vanenburg IT Park, Software Units Layout
> Madhapur, Hyderabad - 500 081.
>
> Phone:      +91 40 66399721
> Fax:           +91 40 66689000
> Mobile:    +91 9849666508
>  
> eMail:       srikant.sana at ips.invensys.com
>
>
> -----Original Message-----
> From: James.Mauro at Sun.COM [mailto:James.Mauro at Sun.COM] 
> Sent: Wednesday, May 21, 2008 4:33 AM
> To: Sana, Srikant
> Subject: Re: Regarding DTrace.
>
> Hello - there''s a good developers article on using dtrace to find
memory
>
> leaks
> in C++ code here:
>
> http://developers.sun.com/solaris/articles/dtrace_cc.html
>
> There''s also an alternate malloc implementation in Solaris 10
called
> libumem.
> If you link to libumem (just use LD_PRELOAD), it will do heap
management
> out of libumem instead of libc, and libumem offers some very nice
> features
> for chasing memory leaks. You can read how to do this here:
>
> http://access1.sun.com/techarticles/libumem.html
>
> The two WEB sites referenced above speak directly to what you''re
looking
> for, much better than I could so with a long email loaded with
examples.
> ;^)
>
> Hopefully, these tools will help you isolate your memory leak problem.
>
> Thanks,
> /jim
>
>
>
> Sana, Srikant wrote:
>   
>> Hi James,
>>
>> This is regarding Dtrace usability for memory leak detection.
>>
>> I read you article on "Solaris 10 & Open Solaris Performance, 
>> Observability & Debugging " found very interesting for my
problem.
>>
>> We have real-time application written C++ which runs on Solaris 10 ( 
>> SPARC based ) , having a problem that''s the process(
application )
>> grows in size from 130 Mb to 450Mb in around 10 days.
>>
>> So there is two possibilities with the application growth of memory 
>> due to Size growth of Dictionary Objects (Like Maps) and Memory Leak.
>>
>> 1) How Dtrace can help to identify the problem .
>>
>> (Dictoionary Objects we can print the size but its live production 
>> site where the problem is happening also there are many such object 
>> used the application)
>>
>> 2) How I can use Dtrace to detect memory leak for C++ code. Can I
have
>>     
>
>   
>> few examples on it.
>>
>> 3) Also is there any way to handle such problems.
>>
>> I got a clue of using "libgc.so" which acts as garbage
collector is
>> there any problem in using and how reliable and effective is this.
>>
>> I righting directly to you without any support since we are in great 
>> hurry to resolve the issue.
>>
>> Your inputs will be highly appreciated. Please do share your thoughts
>> and techniques.
>>
>> Thanks & Regards
>>
>> S L Srikant
>>
>> SCADA Business Unit
>> Invensys Development Centre India Pvt. Ltd.
>> S-210, Plot # 17, Orion Building, Block J
>> Vanenburg IT Park, Software Units Layout
>> Madhapur, Hyderabad - 500 081.
>>
>> Phone: +91 40 66399721
>> Fax: +91 40 66689000
>> Mobile: +91 9849666508
>>
>> eMail: srikant.sana at ips.invensys.com 
>> <mailto:srikant.sana at ips.invensys.com>
>>
>>
>> *
>>
>>     
>
------------------------------------------------------------------------
>   
>> ** Confidential Notice: *This e-mail and any associated files are 
>> intended solely for the individual or entity to whom they are 
>> addressed. Please do not copy it or use it for any purposes, or 
>> disclose its contents to any other person. Further, this e-mail and 
>> any associated files may be confidential and further may be legally 
>> privileged. This email is from the Invensys Process Systems business 
>> unit of Invensys plc which is a company registered in England and 
>> Wales with its registered office at Portland House, Bressenden Place,
>> London, SW1E 5BF (Registered number 166023). For a list of European 
>> legal entities within the Invensys Process Systems business group, 
>> please click here 
>>
>>     
>
http://www.invensys.com/legal/default.asp?top_nav_id=77&nav_id=80&prev_i
> d=77 
>   
>
<http://www.invensys.com/legal/default.asp?top_nav_id=77&nav_id=80&prev_
> id=77>.
>   
>> If you have received this e-mail in error, you are on notice of its 
>> status. Please notify us immediately by reply e-mail and then delete 
>> this message from your system. Thank you for your co-operation. You 
>> may contact our Helpdesk on +44 (0)20 7821 3859 / 2105 or email 
>> inet.hqhelpdesk at invensys.com <mailto:inet.hqhelpdesk at
invensys.com>.
>> This e-mail and any attachments thereto may be subject to the terms
of
>>     
>
>   
>> any agreements between Invensys (and/or its subsidiaries and 
>> affiliates) and the recipient (and/or its subsidiaries and
>>     
> affiliates).
>   
>
------------------------------------------------------------------------
>   
>>     
>
>
> * Confidentiality Notice:
> This e-mail and any associated files are intended solely for the
individual or entity to whom they are addressed. Please do not copy it
or use it for any purposes, or disclose its contents to any other
person. Further, this e-mail and any associated files may be
confidential and further may be legally privileged. This email is from
the Invensys Process Systems business unit of Invensys plc which is a
company registered in England and Wales with its registered office at
Portland House, Bressenden Place, London, SW1E 5BF (Registered number
166023).  For a list of European legal entities within the Invensys
Process Systems business group, please click here
http://www.invensys.com/legal/default.asp?top_nav_id=77&nav_id=80&prev_i
d=77.
>
> If you have received this e-mail in error, you are on notice of its
status. Please notify us immediately by reply e-mail and then delete
this message from your system. Thank you for your co-operation. You may
contact our Helpdesk on +44 (0)20 7821 3859 / 2105 or email
inet.hqhelpdesk at invensys.com. This e-mail and any attachments thereto
may be subject to the terms of any agreements between Invensys (and/or
its subsidiaries and affiliates) and the recipient (and/or its
subsidiaries and affiliates).
>
>
>   

* Confidentiality Notice:
This e-mail and any associated files are intended solely for the individual or
entity to whom they are addressed. Please do not copy it or use it for any
purposes, or disclose its contents to any other person. Further, this e-mail and
any associated files may be confidential and further may be legally privileged.
This email is from the Invensys Process Systems business unit of Invensys plc
which is a company registered in England and Wales with its registered office at
Portland House, Bressenden Place, London, SW1E 5BF (Registered number 166023). 
For a list of European legal entities within the Invensys Process Systems
business group, please click here
http://www.invensys.com/legal/default.asp?top_nav_id=77&nav_id=80&prev_id=77.

If you have received this e-mail in error, you are on notice of its status.
Please notify us immediately by reply e-mail and then delete this message from
your system. Thank you for your co-operation. You may contact our Helpdesk on
+44 (0)20 7821 3859 / 2105 or email inet.hqhelpdesk at invensys.com. This e-mail
and any attachments thereto may be subject to the terms of any agreements
between Invensys (and/or its subsidiaries and affiliates) and the recipient
(and/or its subsidiaries and affiliates).

Sana, Srikant wrote:
> 
> 
> Hi,
> 
>  This is regarding Dtrace usability for memory leak detection.
> 
>  We have real-time application written C++ which runs on Solaris 10  
>   having a problem that''s the my application  grows in size from
130 Mb
> to 450Mb in around 15 days.
> 
>  So there is two possibilities with the application growth of memory 
>  due to Size growth of Dictionary Objects (Like Maps) and Memory Leak.
> 
>  (Dictionary Objects we can print the size but its live production 
>  site where the problem is happening also there are many such object 
>  used the application)
> 
>  When I used  Dtrace and libumem ( with mdb )to detect memory leak for
> C++ code. 
> 
>  When I tried to run dtrace script available in SUN Site it stops
> abruptly
>  With following error
> 
>  # dtrace -s ./CCagg_1.d `pgrep scadadbproc` | c++filt >>
>  scadadbproc_dtrace
>  dtrace: 91102 drops on CPU 0
>  dtrace: processing aborted: Abort due to systemic unresponsiveness
Where is this D script found? What is it trying to look at?
>  While suing mdb ( ::findleaks ) 
>   It says no memory allocation done and says no leaks found.
Did you run ::findleaks on a crash dump or a live system?
> 1) Is there any way where I can restrict dtrace to look for probe only
> certain source files or functions?
Not so sure about source files, but yes to functions. Eg, if I want
to search for calls to open(2), then I''d use

pid$target::open:

> 2) Is there any way to handle such problems?
Yes, but we need more info from you in order to help you.
> 3) I got a clue of using "libgc.so" which acts as garbage
collector is
>    there any problem in using and how reliable and effective is this.
>    ( it works fine on sample application but not improvement in real
> application ) . I just linked the library "libgc.so" at compile
time ?
> Anything else need to done for this ?
Can''t answer this one, sorry, but have you read the article
http://developers.sun.com/solaris/articles/libgc.html


James C. McPherson
--
Senior Kernel Software Engineer, Solaris
Sun Microsystems
http://blogs.sun.com/jmcp	http://www.jmcp.homeunix.com/blog

>  We have real-time application written C++ which runs on Solaris 10  
>  having a problem that''s the my application  grows in size from
130 Mb
>  to 450Mb in around 15 days.
Just to be sure that you know, free(3c) does not return memory back to
system. Sorry if I repeat well known fact.

============================= man free(3c) ============================     The
argument to free() is a pointer to  a  block  previously
     allocated  by malloc(), calloc(), or realloc(). After free()
     is executed, this space is made available for further  allo-
     cation  by  the application, though not returned to the sys-
     tem. Memory is returned to the system only upon  termination
     of  the  application.  If   ptr is a null pointer, no action
     occurs. If a random number is passed to free(), the  results
     are undefined.
=======================================================================
This means that if you allocate 100Mb buffer, free it and allocate 30Mb
buffer, you will see still 100Mb RSS. C++ delete and delete[] operators
are built on top of the same mechanism.

It is left to the OS to swap out unused memory pages.

[...]
> 1) Is there any way where I can restrict dtrace to look for probe only
> certain source files or functions?
Source files, AFAIK no. Functions yes. I guess that you want to
distinguish between allocations made in some specific functions and the
rest of the allocations ?

> 3) I got a clue of using "libgc.so" which acts as garbage
collector is
>    there any problem in using and how reliable and effective is this.
I would also be interested.
> ( it works fine on sample application but not improvement in real
> application ). I just linked the library "libgc.so" at compile
time ?
> Anything else need to done for this ?
For C++ I think that you have to use -library=gc

> Your inputs will be highly appreciated. Please do share your thoughts 
> and techniques.
I would try to make libumem work, it should give you the data.

Not sure if this helps :)
-- 
	Vlad
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 193 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20080609/6fbb8a13/attachment.bin>

Hi James,

Thank you for the response.

The application is a real-time Data acquisition system running continuously
(24x7x365) written in C++  running on Solaris 10 , Ultra 45 Servers.
It uses Versant OODBMS as database. We are using OSE STL library. We are using
socket communication to get data.
  
1) The dtrace script which I is used is here ( which taken from sun site )
		#!/usr/sbin/dtrace -s
		#pragma D option quiet
	/*
		__1c2k6Fpv_v_ == void operator delete(void*)
		__1c2n6FI_pv_ == void*operator new(unsigned)
	*/
		pid$1::__1c2n6FI_pv_:entry
		{
	        ustack();
		}
		pid$1::__1c2n6FI_pv_:return
		{
	        printf("%s: %x\n", probefunc, arg1);
		}
		pid$1::__1c2k6Fpv_v_:entry
		{
	        printf("%s: %x\n", probefunc, arg0);
		}

2)   Regarding MDB : I generated core file from the live system using gcore, 
                     then used that core file with mdb to findleaks. 

3)   libgc.so usage ,I tried after reading the article you have mentioned.

Here is the few things which found after code analysis , 
	We could see that objects of class LNA_Path , LNA_SimplePath are 
      growing continuously in number.( Found this by using static member in 
      class and incrementing and decrementing the count in constructor and 
      destructor of the class. But not sure if it is the real problem.)

Please let me know if any further information required in this regard about the
application.
Also let me know which tool to use to identify the memory leaks the application.

Rgds
S L Srikant


--
This message posted from opensolaris.org

> Just to be sure that you know, free(3c) does not
> return memory back to
> system. Sorry if I repeat well known fact.
> 
> ============================= man free(3c)
> ============================> The argument to free() is a pointer to  a 
block
>   previously
> allocated  by malloc(), calloc(), or realloc().
>  After free()
> is executed, this space is made available for
> further  allo-
> cation  by  the application, though not returned
>  to the sys-
> tem. Memory is returned to the system only upon
>   termination
> of  the  application.  If   ptr is a null pointer,
>  no action
> occurs. If a random number is passed to free(),
>  the  results
>     are undefined.
> =====================================================Yes i did thought in
that angle so far thank for reminiding .
Since the application is real-time in nature and runs continoulsy , termiantion
is rulled out.  Also we are facing problem in particular site implemntations at
other implentations  it runs fine.
> =============> 
> This means that if you allocate 100Mb buffer, free it
> and allocate 30Mb
> buffer, you will see still 100Mb RSS. C++ delete and
> delete[] operators
> are built on top of the same mechanism.
> It is left to the OS to swap out unused memory pages.
> > 1) Is there any way where I can restrict dtrace to
> look for probe only
> > certain source files or functions?
> 
> Source files, AFAIK no. Functions yes. I guess that
> you want to
> distinguish between allocations made in some specific
> functions and the
> rest of the allocations ?
> 
> 
> > 3) I got a clue of using "libgc.so" which acts as
> garbage collector is 
> >    there any problem in using and how reliable and
> effective is this.
> 
> I would also be interested.
> 
> > ( it works fine on sample application but not
> improvement in real
> > application ). I just linked the library "libgc.so"
> at compile time ?
> > Anything else need to done for this ?
> 
> For C++ I think that you have to use -library=gc
> 
> 
> > Your inputs will be highly appreciated. Please do
> share your thoughts 
> > and techniques.
> 
> I would try to make libumem work, it should give you
> the data.
> 
> Not sure if this helps :)
this whats i have done to use libumem and mdb ...
#UMEM_DEBUG=default;export UMEM_DEBUG=default 
#UMEM_LOGGING=transaction;export UMEM_LOGGING=transaction 
#LD_PRELOAD=libumem.so.1;export LD_PRELOAD=libumem.so.1
# ./myapp
Generate core from live system with pid
#gcore 21250  
bash-3.00$ mdb core.21250 
Loading modules: [ libumem.so.1 libc.so.1 ld.so.1 ]
> ::findleaks
mdb: findleaks: No allocations have occured -- no possible leaks.

Anything else is missing from my sequence of steps ?
Do i need to do add anything during compilation ?

 
> 	Vlad
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.9 (SunOS)
> 
> iEYEARECAAYFAkhNGbcACgkQqVN0MVP42YwBKACgvwl7wn9uDWrrOL
> nF5JSp0W0f
> YXYAnR8Rlc9x1P7fntZXWbic8hv9LsQK
> =y2EK
> -----END PGP SIGNATURE-----
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org

--
This message posted from opensolaris.org

> > ============================= man free(3c)
============================
> Yes i did thought in that angle so far thank for reminiding . Since
> the application is real-time in nature and runs continoulsy ,
> termiantion is rulled out.  Also we are facing problem in particular
> site implemntations at other implentations  it runs fine.
That just screams for the question, what''s different between the
implementations ? Different load, different compilation environment,
different OS versions ? Anything else ?


[...]
> > I would try to make libumem work, it should give you
> > the data.
> > 
> > Not sure if this helps :)
> 
> this whats i have done to use libumem and mdb ...
> #UMEM_DEBUG=default;export UMEM_DEBUG=default 
> #UMEM_LOGGING=transaction;export UMEM_LOGGING=transaction 
> #LD_PRELOAD=libumem.so.1;export LD_PRELOAD=libumem.so.1
> # ./myapp
> Generate core from live system with pid
> #gcore 21250  
> bash-3.00$ mdb core.21250 
> Loading modules: [ libumem.so.1 libc.so.1 ld.so.1 ]
> > ::findleaks
> mdb: findleaks: No allocations have occured -- no possible leaks.
That''s strange, most probably it means that libumem wasn''t
preloaded
successfully. Any chance that your application links the memory
management library statically ? (maybe you are using something else than
libc ?). Can you run pmap on the generated core file ? It should show you few
memory regions used by libumem. Also you can try in mdb

malloc::dis

and you should see

libumem.so.1`malloc:            pushl  %ebp
...

and not 

libc.so.1`malloc:               pushl  %ebp
...

If you run

nm -C yourbinary | less

You should find ''void operator delete(void*)'' and
''void*operator
new(unsigned)''. In my case it''s __1c2k6Fpv_v_ and something
else. Again in mdb
looking at the core file, you should see traces of libumem
> __1c2k6Fpv_v_::dis ! grep call
libCrun.so.1`__1c2k6Fpv_v_+9:   call   +0x0    
<libCrun.so.1`__1c2k6Fpv_v_+0xe>
libCrun.so.1`__1c2k6Fpv_v_+0x21:call   -0x1efe  <PLT=libumem.so.1`free>
libCrun.so.1`__1c2k6Fpv_v_+0x37:call   -0x1f84 
<PLT:__1cG__CrunRex_chk_unexpected6F_v_>
libCrun.so.1`__1c2k6Fpv_v_+0x3c:call   -0x1f79 
<PLT:__1cG__CrunMex_rethrow_q6F_v_>


> Anything else is missing from my sequence of steps ?
Not really.

> Do i need to do add anything during compilation ?
I do not think so.

-- 
	Vlad
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 193 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20080610/664cbac0/attachment.bin>

> > Yes i did thought in that angle so far thank for
> reminiding . Since
> > the application is real-time in nature and runs
> continoulsy ,
> > termiantion is rulled out.  Also we are facing
> problem in particular
> > site implemntations at other implentations  it runs
> fine.
> 
> That just screams for the question, what''s different
> between the
> implementations ? Different load, different
> compilation environment,
> different OS versions ? Anything else ?
OS is Solaris 10 and all SPARC based servers but server model no may vary ,
but i am working on Ultra 45 server. 
The major difference between where we are facing problem from others is
that it through some API provided by our application external applications
pump in data and allow to collect data from it.

> > > I would try to make libumem work, it should give
> you
> > > the data.
> > > 
> > > Not sure if this helps :)
> > 
> > this whats i have done to use libumem and mdb ...
> > #UMEM_DEBUG=default;export UMEM_DEBUG=default 
> > #UMEM_LOGGING=transaction;export
> UMEM_LOGGING=transaction 
> > #LD_PRELOAD=libumem.so.1;export
> LD_PRELOAD=libumem.so.1
> > # ./myapp
> > Generate core from live system with pid
> > #gcore 21250  
> > bash-3.00$ mdb core.21250 
> > Loading modules: [ libumem.so.1 libc.so.1 ld.so.1 ]
> > > ::findleaks
> > mdb: findleaks: No allocations have occured -- no
> possible leaks.
> 
> That''s strange, most probably it means that libumem
> wasn''t preloaded
> successfully. Any chance that your application links
> the memory
> management library statically ? (maybe you are using
> something else than
> libc ?). Can you run pmap on the generated core file
> ? It should show you few
> memory regions used by libumem. Also you can try in
> mdb
> 
> malloc::dis
> 
> and you should see
> 
> libumem.so.1`malloc:            pushl  %ebp
> ...
> 
> and not 
> 
> libc.so.1`malloc:               pushl  %ebp
> ...
> 
Here is library linking  
# ldd /opt/scada/bin/scadadbproc | grep -i libcrun
        libCrun.so.1 =>  /usr/lib/libCrun.so.1
# ldd /opt/scada/bin/scadadbproc | grep -i umem
        libumem.so.1 =>  /lib/libumem.so.1


Here is ouput of malloc::dis  ( its differing from what you are saying )
=================================================================>
malloc::dis
0xb0ab8c:                       and       %o1, 1, %o0
0xb0ab90:                       or        %o0, %g1, %g5
0xb0ab94:                       st        %g5, [%i1 + 4]
0xb0ab98:                       call      +0x7f4        <0xb0b38c>
0xb0ab9c:                       restore
0xb0aba0:                       or        %g2, 1, %l4
0xb0aba4:                       st        %l4, [%i1 + 4]
0xb0aba8:                       st        %g2, [%g3]
0xb0abac:                       ret
0xb0abb0:                       restore
0xb0abb4:                       save      %sp, -0x60, %sp
0xb0abb8:                       sethi     %hi(0x109d800), %i5
0xb0abbc:                       ld        [%i5 + 0x1c], %i4
0xb0abc0:                       cmp       %i4, 0
0xb0abc4:                       be,pn     %icc, +0x14   <0xb0abd8>
0xb0abc8:                       sethi     %hi(0x109d800), %o3
0xb0abcc:                       ld        [%o3 + 0x1c], %o2
0xb0abd0:                       jmp       %o2
0xb0abd4:                       restore
0xb0abd8:                       add       %i0, 0xb, %o4
0xb0abdc:                       cmp       %o4, 0

Also the make file ouput 
=================================================================/opt/SUNWspro/bin/CC
-mt    -O -temp=/tmp/ssana -compat=5 -library=iostream scadadbproc.o
libsa_dbproc.a `echo ..//scada_lib/common/libsl_common.a
..//scada_lib/appdbobjs/libsl_loadman.a
..//scada_lib/configurator/libsl_configurator_system_views.a
..//scada_lib/configurator/libsl_configurator_commands.a
..//scada_lib/configurator/libsl_configurator_db_forms.a
..//scada_lib/configurator/libsl_configurator_iexport.a
..//scada_lib/configurator/libsl_configurator_install.a
..//scada_lib/configurator/libsl_configurator_windows.a
..//scada_lib/configurator/libsl_configurator_data_fields.a
..//scada_lib/configurator/libsl_configurator_agent.a
..//scada_lib/configurator/libsl_configurator_misc.a
..//scada_lib/configurator/libsl_configurator_ComboBox.a
..//scada_lib/configurator/libsl_dbutility.a
..//scada_lib/calcs/libsl_calcs_runtime.a
..//scada_lib/calcs/libsl_calcs_linker.a
..//scada_lib/calcs/libsl_calcs_compiler_compiler.a
..//scada_lib/calcs/libsl_calcs_debugger.a
..//scada_lib/calcs/libsl_calcs_interpreter.a
..//scada_lib/calcs/libsl_calcs_module_parser.a
..//scada_lib/calcs/libsl_calcs_statement_parser.a
..//scada_lib/calcs/libsl_calcs_expression_parser.a
..//scada_lib/calcs/libsl_calcs_types_parser.a
..//scada_lib/calcs/libsl_calcs_dict.a ..//scada_lib/calcs/libsl_calcs_module.a
..//scada_lib/calcs/libsl_calcs_statement.a
..//scada_lib/calcs/libsl_calcs_expression.a
..//scada_lib/calcs/libsl_calcs_connection.a
..//scada_lib/calcs/libsl_calcs_types.a ..//scada_lib/calcs/libsl_calcs_util.a
..//scada_lib/history/libsl_history.a ..//scada_lib/fep/libsl_fep.a
..//scada_lib/fep/libsl_fep_devices.a
..//scada_lib/fep/libsl_fep_file_transfer.a
..//scada_lib/fep/libsl_fep_emulator.a ..//scada_lib/fep/libsl_fep.a
..//scada_lib/fep/libsl_fep_telemetry.a ..//scada_lib/fep/libsl_fep_objects.a
..//scada_lib/fep/libsl_fep_generators.a ..//scada_lib/fep/libsl_fep_stats.a
..//scada_lib/fep/libsl_fep_protocol.a ..//scada_lib/hmi/libsl_hmi_event.a
..//scada_lib/hmi/libsl_hmi_object.a ..//scada_lib/hmi/libsl_hmi_alarm.a
..//scada_lib/hmi/libsl_hmi_presenter.a ..//scada_lib/hmi/libsl_hmi_common.a
..//scada_lib/hmi/libsl_hmi_cpp.a ..//scada_lib/core_scada/libsl_core_scada.a
..//scada_lib/core_scada/libsl_core_scada_points.a
..//scada_lib/core_scada/libsl_core_scada_devices.a
..//scada_lib/core_scada/libsl_core_scada_lists.a
..//scada_lib/core_scada/libsl_core_scada_demand_scan.a
..//scada_lib/core_scada/libsl_core_scada_controls.a
..//scada_lib/core_scada/libsl_core_scada_scanning.a
..//scada_lib/core_scada/libsl_core_scada_cvq.a
..//scada_lib/core_scada/libsl_core_scada_ptw.a
..//scada_lib/core_scada/libsl_core_scada_station_monitor.a
..//scada_lib/core_scada/libsl_core_scada_alarm_system.a
..//scada_lib/core_scada/libsl_core_scada_alarm_flatLine.a
..//scada_lib/core_scada/libsl_core_scada_alarm_digital.a
..//scada_lib/core_scada/libsl_core_scada_alarm_analog.a
..//scada_lib/core_scada/libsl_core_scada_alarm_point.a
..//scada_lib/core_scada/libsl_core_scada_alarm_generic.a
..//scada_lib/core_scada/libsl_core_scada_alarm_common.a
..//scada_lib/hmi/libsl_hmi_event.a ..//scada_lib/hmi/libsl_hmi_object.a
..//scada_lib/hmi/libsl_hmi_alarm.a ..//scada_lib/hmi/libsl_hmi_presenter.a
..//scada_lib/hmi/libsl_hmi_common.a ..//scada_lib/hmi/libsl_hmi_cpp.a
..//scada_lib/dbase/libsl_dbtypes.a ..//scada_lib/dbase/libsl_superagent.a
..//scada_lib/dbase/libsl_logger.a ..//scada_lib/dbase/libsl_thread.a
..//scada_lib/dbase/libsl_stream.a ..//scada_lib/dbase/libsl_interfaces.a
..//scada_lib/dbase/libsl_startupStateMachine.a ..//scada_lib/api/libsl_error.a
..//scada_lib/api/libsl_events.a ..//scada_lib/api/libsl_proxy.a
..//scada_lib/api/libsl_connections.a ..//scada_lib/api/libsl_multicast.a
..//scada_lib/api/libsl_dispatcher.a ..//scada_lib/api/libsl_controls.a
..//scada_lib/api/libsl_histint.a ..//scada_lib/api/libsl_alarmint.a
..//scada_lib/api/libsl_paths.a ..//scada_lib/api/libsl_system.a
..//scada_lib/api/libsl_value.a ..//scada_lib/utilities/libsl_license.a
..//scada_lib/utilities/libsl_scadatrace.a
..//scada_lib/utilities/libsl_commandline.a ..//versant/vod7010/lib/libcxxcls.a
..//versant/vod7010/lib/liboscfe.a -ldl ..//scada_lib/api/libsl_error.a
..//ose/6.1pl1/SPARC_SOL2/lib/SUN5.7/libOTC_6X1pl01_SUN5X7_opt.a
..//ose/6.1pl1/SPARC_SOL2/lib/SUN5.7/libOSE.a -ldl
[u]..//scada_lib/utilities/libsl_malloc.a[/u] | ..//rules/bin/sed
''s=\([^ ]*\)/lib\([^ /]*\)\.so=-L\1 -l\2=g''` -L/usr/dt/lib
-lXm12 -L..//scada_include/configurator/XDesigner/xpm -lXpm -L/usr/openwin/lib
-lXmu -lXt -lX11 -lXext -lkvm -lelf -lkstat -ldl -Bdynamic -lpthread -lrt
-lsocket -lnsl -library=gc  scadadbproc_ccm_executable_version.o -o scadadbproc


==============================================================> If you
run
> 
> nm -C yourbinary | less
> 
> You should find ''void operator delete(void*)'' and
> ''void*operator
> new(unsigned)''. In my case it''s __1c2k6Fpv_v_ and
> something else. Again in mdb
> looking at the core file, you should see traces of
> libumem
> 
> > __1c2k6Fpv_v_::dis ! grep call
> libCrun.so.1`__1c2k6Fpv_v_+9:   call   +0x0
>     <libCrun.so.1`__1c2k6Fpv_v_+0xe>
> run.so.1`__1c2k6Fpv_v_+0x21:call   -0x1efe
>  <PLT=libumem.so.1`free>
> ibCrun.so.1`__1c2k6Fpv_v_+0x37:call   -0x1f84
>  <PLT:__1cG__CrunRex_chk_unexpected6F_v_>
> ibCrun.so.1`__1c2k6Fpv_v_+0x3c:call   -0x1f79
>  <PLT:__1cG__CrunMex_rethrow_q6F_v_>
> 
Here is output from core file 
============================================================>
__1c2k6Fpv_v_::dis ! grep call
libCrun.so.1`__1c2k6Fpv_v_+4:   call      +0x14780      <0xfebea938>
libCrun.so.1`__1c2k6Fpv_v_+0x14:call      +0x1471c     
<PLT:__1cG__CrunRex_chk_unexpected6F_v_>
libCrun.so.1`__1c2k6Fpv_v_+0x1c:call      +0x14720     
<PLT:__1cG__CrunMex_rethrow_q6F_v_>
> __1c2n6FI_pv_::dis ! grep call
libCrun.so.1`__1c2n6FI_pv_+4:   call      +8           
<libCrun.so.1`__1c2n6FI_pv_+0xc>
libCrun.so.1`__1c2n6FI_pv_+0x2c:call      +0x138f8      <0xfebea908>
libCrun.so.1`__1c2n6FI_pv_+0x40:call      +0x13c20     
<PLT:__1cH__CimplPget_new_handler6F_pF_v_>
libCrun.so.1`__1c2n6FI_pv_+0x54:call      +0x139f0     
<PLT:__1cG__CrunIex_alloc6FI_pv_>
libCrun.so.1`__1c2n6FI_pv_+0x5c:call      +0x13c10     
<PLT:__1cDstdJbad_alloc2t6M_v_>
libCrun.so.1`__1c2n6FI_pv_+0x70:call      +0x139ec     
<PLT:__1cG__CrunIex_throw6Fpvpkn0AQstatic_type_info_pF1_v_v_>
libCrun.so.1`__1c2n6FI_pv_+0x98:call      +0x13868     
<PLT:__1cG__CrunRex_chk_unexpected6F_v_>
libCrun.so.1`__1c2n6FI_pv_+0xa0:call      +0x1386c     
<PLT:__1cG__CrunMex_rethrow_q6F_v_>

============================================================What could be the
reason that i am getting no allocation done in mdb,
please let know anything else i am missing ?

> 
> > Anything else is missing from my sequence of steps
> ?
> 
> Not really.
> 
> 
> > Do i need to do add anything during compilation ?
> 
> I do not think so.
> 
> -- 
> 	Vlad
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.9 (SunOS)
> 
> iEYEARECAAYFAkhON1UACgkQqVN0MVP42YzSJwCfZrUbJ3hOdv7AV9
> 2WN/kICuwe
> 2+MAoIQSfUsrLIG/Gfa8EsEN1shCzVox
> =lzPF
> -----END PGP SIGNATURE-----
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org

--
This message posted from opensolaris.org