dtrace discuss - Nov 2007 - Exhaustive tracing with pid provider?

Hi,

I tried, naively, to do something like:
#!/usr/sbin/dtrace -qs



pid$$1:*::entry
{
  printf("start %s(arg0=%d)\n",probefunc,arg0);
}

pid$$1:*::return
{
  printf("end %s\n",probefunc);
}

But dtrace said:
dtrace: failed to compile script dtrace.all_arg.d: line 5: failed to
create probe in process 10422: Not enough space

Is there a way around that? At the moment I am creating one probe per
function of interest, but I always need more functions and need to run
my program again and again.

Speed is not an issue as the program is under Purify already... So you
can tell I am OK with a slow program ;-)

  Thanks,

  Pierre-Olivier

Hi Pierre-Olivier:

You can increase the max number of probes by modifying the /kernel/drv/ 
fasttrap.conf
Set the fasttrap-ma-probes to a higher number. After editing you can  
either reboot
or call "update_drv fasttrap"

Also you do not need the * in probe description. It defaults to wild  
card. Also the
$$1 is the first argument to your script in string format. As you are  
using an integer,
just $1 should do.

HTHs

Angelo


On Nov 7, 2007, at 12:53 PM, Pierre-Olivier Gaillard wrote:
> Hi,
>
> I tried, naively, to do something like:
> #!/usr/sbin/dtrace -qs
>
>
>
> pid$$1:*::entry
> {
>  printf("start %s(arg0=%d)\n",probefunc,arg0);
> }
>
> pid$$1:*::return
> {
>  printf("end %s\n",probefunc);
> }
>
> But dtrace said:
> dtrace: failed to compile script dtrace.all_arg.d: line 5: failed to
> create probe in process 10422: Not enough space
>
> Is there a way around that? At the moment I am creating one probe per
> function of interest, but I always need more functions and need to run
> my program again and again.
>
> Speed is not an issue as the program is under Purify already... So you
> can tell I am OK with a slow program ;-)
>
>  Thanks,
>
>  Pierre-Olivier
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20071107/33f7760e/attachment.html>

On Nov 7, 2007, at 1:06 PM, Angelo Rajadurai wrote:
> Hi Pierre-Olivier:
>
> You can increase the max number of probes by modifying the /kernel/ 
> drv/fasttrap.conf
> Set the fasttrap-ma-probes to a higher number. After editing you can  
> either reboot
> or call "update_drv fasttrap"
Sorry typo that should be fasttrap-max-probes

-Angelo

>
> Also you do not need the * in probe description. It defaults to wild  
> card. Also the
> $$1 is the first argument to your script in string format. As you  
> are using an integer,
> just $1 should do.
>
> HTHs
>
> Angelo
>
>
> On Nov 7, 2007, at 12:53 PM, Pierre-Olivier Gaillard wrote:
>
>> Hi,
>>
>> I tried, naively, to do something like:
>> #!/usr/sbin/dtrace -qs
>>
>>
>>
>> pid$$1:*::entry
>> {
>>  printf("start %s(arg0=%d)\n",probefunc,arg0);
>> }
>>
>> pid$$1:*::return
>> {
>>  printf("end %s\n",probefunc);
>> }
>>
>> But dtrace said:
>> dtrace: failed to compile script dtrace.all_arg.d: line 5: failed to
>> create probe in process 10422: Not enough space
>>
>> Is there a way around that? At the moment I am creating one probe per
>> function of interest, but I always need more functions and need to  
>> run
>> my program again and again.
>>
>> Speed is not an issue as the program is under Purify already... So  
>> you
>> can tell I am OK with a slow program ;-)
>>
>>  Thanks,
>>
>>  Pierre-Olivier
>> _______________________________________________
>> dtrace-discuss mailing list
>> dtrace-discuss at opensolaris.org
>
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20071107/90aaae85/attachment.html>

Thanks a lot. That sounds helpful.
How many probes can I have (on a Sparc)?
Would a couple millions be OK?

  Thanks for your help,

On Nov 7, 2007 1:06 PM, Angelo Rajadurai <Angelo.Rajadurai at sun.com>
wrote:
> Hi Pierre-Olivier:
>
> You can increase the max number of probes by modifying the
> /kernel/drv/fasttrap.conf
> Set the fasttrap-ma-probes to a higher number. After editing you can either
> reboot
> or call "update_drv fasttrap"
>
> Also you do not need the * in probe description. It defaults to wild card.
> Also the
> $$1 is the first argument to your script in string format. As you are using
> an integer,
> just $1 should do.
>
> HTHs
>
> Angelo
>
>
>
>
>
> On Nov 7, 2007, at 12:53 PM, Pierre-Olivier Gaillard wrote:
>
>
> Hi,
>
> I tried, naively, to do something like:
> #!/usr/sbin/dtrace -qs
>
>
>
> pid$$1:*::entry
> {
>   printf("start %s(arg0=%d)\n",probefunc,arg0);
> }
>
> pid$$1:*::return
> {
>   printf("end %s\n",probefunc);
> }
>
> But dtrace said:
> dtrace: failed to compile script dtrace.all_arg.d: line 5: failed to
> create probe in process 10422: Not enough space
>
> Is there a way around that? At the moment I am creating one probe per
> function of interest, but I always need more functions and need to run
> my program again and again.
>
> Speed is not an issue as the program is under Purify already... So you
> can tell I am OK with a slow program ;-)
>
>   Thanks,
>
>   Pierre-Olivier
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org
>
>

Pierre-Olivier Gaillard wrote:
> Thanks a lot. That sounds helpful.
> How many probes can I have (on a Sparc)?
> Would a couple millions be OK?
I can''t answer that specific question, but I am wondering what
you''re
trying to achieve with the amount of data these probes potentially accumulate.

Michael
-- 
Michael Schuster
Recursion, n.: see ''Recursion''

I am investigating a memory problem (access to memory that was
released previously) that Purify can''t analyze.

As I don''t know the program I am analyzing, I thought I could generate
a big trace of all function calls and search through it.

 The resulting file might be way too big though : I did a trace with
all malloc and free calls and it already takes 155MB. But on the other
hand, if malloc and free represent 1% of all calls I will get a 15GB
trace file which is not that much (I have filled 120GB hard drives
with Linux Trace Toolkit traces and the result was helpful).

 Thanks,

  Pierre-Olivier




On Nov 7, 2007 1:28 PM, Michael Schuster <Michael.Schuster at sun.com>
wrote:
> Pierre-Olivier Gaillard wrote:
> > Thanks a lot. That sounds helpful.
> > How many probes can I have (on a Sparc)?
> > Would a couple millions be OK?
>
> I can''t answer that specific question, but I am wondering what
you''re
> trying to achieve with the amount of data these probes potentially
accumulate.
>
> Michael
> --
> Michael Schuster
> Recursion, n.: see ''Recursion''
>

It''s not dtrace-related, but you should check out the following:

On Wed, Nov 07, 2007 at 01:45:33PM -0500, Pierre-Olivier Gaillard
wrote:
> I am investigating a memory problem (access to memory that was
> released previously) that Purify can''t analyze.
> 
> As I don''t know the program I am analyzing, I thought I could
generate
> a big trace of all function calls and search through it.
Have you tried libumem?  It''s frickin'' awesome for catching
problems like
this (and part of OpenSolaris for free)!

Run your program like so (assuming sh,ksh,bash):

	LD_PRELOAD=libumem.so UMEM_DEBUG=default <your-program>

or with csh,tcsh:

   ( setenv LD_PRELOAD libumem.so; setenv UMEM_DEBUG default ;
<your-program>)

And watch libumem catch your program in the act!  It will dump core, and you
can use MDB''s post-mortem UMEM analysis tools like
"::findleaks" and
"::bufctl".

There''s got to be a page with examples, I just don''t have it
off the top of
my head.

Dan

Yikes!

Me think that this is no way to debug a problem. Why collect data that
you will never use! Just collect what you are interested in. DTrace
aggregates are excellent for this type of debugging. So you don''t
spend the rest of your life going through 15 GB of text data.

Also you may want to looks DTrace speculation. Why do manually
when you can program it out.

Also there are nice tools like libumem that can probably help you
find the memory issue you are tracking see
http://blogs.sun.com/ahl/entry/solaris_10_top_11_20

Hope this helps

Angelo


On Nov 7, 2007, at 1:45 PM, Pierre-Olivier Gaillard wrote:
> I am investigating a memory problem (access to memory that was
> released previously) that Purify can''t analyze.
>
> As I don''t know the program I am analyzing, I thought I could
generate
> a big trace of all function calls and search through it.
>
> The resulting file might be way too big though : I did a trace with
> all malloc and free calls and it already takes 155MB. But on the other
> hand, if malloc and free represent 1% of all calls I will get a 15GB
> trace file which is not that much (I have filled 120GB hard drives
> with Linux Trace Toolkit traces and the result was helpful).
>
> Thanks,
>
>  Pierre-Olivier
>
>
>
>
> On Nov 7, 2007 1:28 PM, Michael Schuster <Michael.Schuster at
sun.com>
> wrote:
>> Pierre-Olivier Gaillard wrote:
>>> Thanks a lot. That sounds helpful.
>>> How many probes can I have (on a Sparc)?
>>> Would a couple millions be OK?
>>
>> I can''t answer that specific question, but I am wondering what
you''re
>> trying to achieve with the amount of data these probes potentially  
>> accumulate.
>>
>> Michael
>> --
>> Michael Schuster
>> Recursion, n.: see ''Recursion''
>>

Pierre-Olivier Gaillard wrote:
> I am investigating a memory problem (access to memory that was
> released previously) that Purify can''t analyze.
>
>   
Have you tried libumem (3LIB)? It has special debug options which are 
designed to catch exactly this kind of problem. Particularly look at 
umem_debug(3MALLOC).

You should be able to LD_PRELOAD it.

Chris
> As I don''t know the program I am analyzing, I thought I could
generate
> a big trace of all function calls and search through it.
>
>  The resulting file might be way too big though : I did a trace with
> all malloc and free calls and it already takes 155MB. But on the other
> hand, if malloc and free represent 1% of all calls I will get a 15GB
> trace file which is not that much (I have filled 120GB hard drives
> with Linux Trace Toolkit traces and the result was helpful).
>
>  Thanks,
>
>   Pierre-Olivier
>
>
>
>
> On Nov 7, 2007 1:28 PM, Michael Schuster <Michael.Schuster at
sun.com> wrote:
>   
>> Pierre-Olivier Gaillard wrote:
>>     
>>> Thanks a lot. That sounds helpful.
>>> How many probes can I have (on a Sparc)?
>>> Would a couple millions be OK?
>>>       
>> I can''t answer that specific question, but I am wondering what
you''re
>> trying to achieve with the amount of data these probes potentially
accumulate.
>>
>> Michael
>> --
>> Michael Schuster
>> Recursion, n.: see ''Recursion''
>>
>>     
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org
>

Dan McDonald <danmcd at sun.com> writes:
> It''s not dtrace-related, but you should check out the following:
>
> On Wed, Nov 07, 2007 at 01:45:33PM -0500, Pierre-Olivier Gaillard wrote:
>> I am investigating a memory problem (access to memory that was
>> released previously) that Purify can''t analyze.
>> 
>> As I don''t know the program I am analyzing, I thought I could
>> generate a big trace of all function calls and search through it.
>
> Have you tried libumem?  It''s frickin'' awesome for
catching problems
> like this (and part of OpenSolaris for free)!
>
> Run your program like so (assuming sh,ksh,bash):
>
> 	LD_PRELOAD=libumem.so UMEM_DEBUG=default <your-program>
>
> or with csh,tcsh:
>
>    ( setenv LD_PRELOAD libumem.so; setenv UMEM_DEBUG default ;
>    <your-program>)
>
> And watch libumem catch your program in the act!  It will dump core,
> and you can use MDB''s post-mortem UMEM analysis tools like
> "::findleaks" and "::bufctl".
>
> There''s got to be a page with examples, I just don''t have
it off the
> top of my head.
Here''s one:
http://blogs.sun.com/pnayak/entry/finding_memory_leaks_within_solaris

And a list of others:
http://www.genunix.org/wiki/index.php/libumem_Resources

Boyd