dtrace discuss - Mar 2006 - Dtrace not working for processes running after a couple of hours

Hi, all,
 
I''m a newbie on dtrace and the problem I am having is: 
    For processes which have been running for more than a couple of
hours, dtrace doesn''t seem to be able to trace(I waited for at least
one
hour and didn''t see anything happening). However, it works every time
for a relatively new process.
 
Any insight or help will be appreciated.
 
This is my machine:
    SunOS costello 5.10 Generic sun4u sparc SUNW,Sun-Fire-V440
 
Thanks,
Jianbing
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20060301/a042b930/attachment.html>

My first thought would be that, for whatever probes you''ve
instrumented,
they''re not firing for the long running process.  It''s hard to
be sure
without seeing the script or command line you are using to start 
DTrace.  Are you running one of the canned scripts?

Chip

jianbing.chen at exgate.tek.com wrote:
> Hi, all,
>  
> I''m a newbie on dtrace and the problem I am having is:
>     For processes which have been running for more than a couple of 
> hours, dtrace doesn''t seem to be able to trace(I waited for at
least
> one hour and didn''t see anything happening). However, it works
every
> time for a relatively new process.
>  
> Any insight or help will be appreciated.
>  
> This is my machine:
>     SunOS costello 5.10 Generic sun4u sparc SUNW,Sun-Fire-V440
>  
> Thanks,
> Jianbing
>
>------------------------------------------------------------------------
>
>_______________________________________________
>dtrace-discuss mailing list
>dtrace-discuss at opensolaris.org
>

On Wed, Mar 01, 2006 at 05:37:35PM -0600, jianbing.chen at exgate.tek.com
wrote:
> Hi, all,
>  
> I''m a newbie on dtrace and the problem I am having is: 
>     For processes which have been running for more than a couple of
> hours, dtrace doesn''t seem to be able to trace(I waited for at
least one
> hour and didn''t see anything happening). However, it works every
time
> for a relatively new process.
What probes are you tracing?  I''ve seen processes run for hours without
making a single system call, for example, and it''s even possible that a
process could run for hours without taking any page faults or receiving
signals.  Attaching to it with a debugger and single-stepping it may
help.  Watching it go on and off cpu with dtrace may help show that
it''s
executing something.

Chip,

I tried a couple scripts. One is the caned one "dapptrace" and the
other
is the following:

#!/usr/sbin/dtrace -s

#pragma D option flowindent

pid$1::$2:entry
{
    self->trace = 1;
    self->in = timestamp;
}

pid$1::$2:return
/self->trace/
{
    @[probefunc] = quantize(timestamp - self->in);
    self->trace = 0;
}

pid$1:::entry, pid$1:::return
/self->trace/
{
}

Thanks for the reply,
Jianbing



-----Original Message-----
From: Chip Bennett [mailto:cbennett at laurustech.com]
Sent: Wednesday, March 01, 2006 5:47 PM
To: Chen, Jianbing
Cc: dtrace-discuss at opensolaris.org
Subject: Re: [dtrace-discuss] Dtrace not working for processes running
after a couple of hours

My first thought would be that, for whatever probes you''ve
instrumented,
they''re not firing for the long running process.  It''s hard to
be sure
without seeing the script or command line you are using to start DTrace.
Are you running one of the canned scripts?

Chip

jianbing.chen at exgate.tek.com wrote:
> Hi, all,
> 
> I''m a newbie on dtrace and the problem I am having is:
>     For processes which have been running for more than a couple of
> hours, dtrace doesn''t seem to be able to trace(I waited for at
least
> one hour and didn''t see anything happening). However, it works
every
> time for a relatively new process.
> 
> Any insight or help will be appreciated.
> 
> This is my machine:
>     SunOS costello 5.10 Generic sun4u sparc SUNW,Sun-Fire-V440
> 
> Thanks,
> Jianbing
>
>-----------------------------------------------------------------------
>-
>
>_______________________________________________
>dtrace-discuss mailing list
>dtrace-discuss at opensolaris.org
>


-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20060301/e8d3d06d/attachment.html>

Pid$1::func, which is some user library function.

BTW, the function call that I''m tracing is getting called close to 100
times per second.

The interesting part is that for a process with less than 1 hour age, it
works every time.

Thanks,
Jianbing
 

-----Original Message-----
From: Nicolas Williams [mailto:Nicolas.Williams at sun.com] 
Sent: Wednesday, March 01, 2006 5:51 PM
To: Chen, Jianbing
Cc: dtrace-discuss at opensolaris.org
Subject: Re: [dtrace-discuss] Dtrace not working for processes running
after a couple of hours

On Wed, Mar 01, 2006 at 05:37:35PM -0600, jianbing.chen at exgate.tek.com
wrote:
> Hi, all,
>  
> I''m a newbie on dtrace and the problem I am having is: 
>     For processes which have been running for more than a couple of 
> hours, dtrace doesn''t seem to be able to trace(I waited for at
least
> one hour and didn''t see anything happening). However, it works
every
> time for a relatively new process.
What probes are you tracing?  I''ve seen processes run for hours without
making a single system call, for example, and it''s even possible that a
process could run for hours without taking any page faults or receiving
signals.  Attaching to it with a debugger and single-stepping it may
help.  Watching it go on and off cpu with dtrace may help show that
it''s
executing something.

Just for grins I wrote a small C program that called one function, and 
then ran your script against it.  I was seeing the function entry until 
I compiled using "gcc -O3", which will take functions and try to 
"inline" them.  This is going to cause the function to not form a
proper
stack trace.  The interesting part is that my function name was still 
available to match the probe, but there was no action output.

Could something like that be happening to you?  What language is your 
program in?

Chip

jianbing.chen at exgate.tek.com wrote:
>Pid$1::func, which is some user library function.
>
>BTW, the function call that I''m tracing is getting called close to
100
>times per second.
>
>The interesting part is that for a process with less than 1 hour age, it
>works every time.
>
>Thanks,
>Jianbing
> 
>
>-----Original Message-----
>From: Nicolas Williams [mailto:Nicolas.Williams at sun.com] 
>Sent: Wednesday, March 01, 2006 5:51 PM
>To: Chen, Jianbing
>Cc: dtrace-discuss at opensolaris.org
>Subject: Re: [dtrace-discuss] Dtrace not working for processes running
>after a couple of hours
>
>On Wed, Mar 01, 2006 at 05:37:35PM -0600, jianbing.chen at exgate.tek.com
>wrote:
>  
>
>>Hi, all,
>> 
>>I''m a newbie on dtrace and the problem I am having is: 
>>    For processes which have been running for more than a couple of 
>>hours, dtrace doesn''t seem to be able to trace(I waited for at
least
>>one hour and didn''t see anything happening). However, it works
every
>>time for a relatively new process.
>>    
>>
>
>What probes are you tracing?  I''ve seen processes run for hours
without
>making a single system call, for example, and it''s even possible
that a
>process could run for hours without taking any page faults or receiving
>signals.  Attaching to it with a debugger and single-stepping it may
>help.  Watching it go on and off cpu with dtrace may help show that
it''s
>executing something.
>_______________________________________________
>dtrace-discuss mailing list
>dtrace-discuss at opensolaris.org
>  
>

Chip,

It''s in c++ and with "gcc -g -O2". Still, it''s hard
to explain the non-deterministic behavior with the only difference on the
lifespan of the process.

Thanks for helping out, 
Jianbing



-----Original Message-----
From: Chip Bennett [mailto:cbennett at laurustech.com]
Sent: Wed 3/1/2006 6:57 PM
To: Chen, Jianbing
Cc: dtrace-discuss at opensolaris.org
Subject: Re: [dtrace-discuss] Dtrace not working for processes running after a
couple of hours
 
Just for grins I wrote a small C program that called one function, and 
then ran your script against it.  I was seeing the function entry until 
I compiled using "gcc -O3", which will take functions and try to 
"inline" them.  This is going to cause the function to not form a
proper
stack trace.  The interesting part is that my function name was still 
available to match the probe, but there was no action output.

Could something like that be happening to you?  What language is your 
program in?

Chip

jianbing.chen at exgate.tek.com wrote:
>Pid$1::func, which is some user library function.
>
>BTW, the function call that I''m tracing is getting called close to
100
>times per second.
>
>The interesting part is that for a process with less than 1 hour age, it
>works every time.
>
>Thanks,
>Jianbing
> 
>
>-----Original Message-----
>From: Nicolas Williams [mailto:Nicolas.Williams at sun.com] 
>Sent: Wednesday, March 01, 2006 5:51 PM
>To: Chen, Jianbing
>Cc: dtrace-discuss at opensolaris.org
>Subject: Re: [dtrace-discuss] Dtrace not working for processes running
>after a couple of hours
>
>On Wed, Mar 01, 2006 at 05:37:35PM -0600, jianbing.chen at exgate.tek.com
>wrote:
>  
>
>>Hi, all,
>> 
>>I''m a newbie on dtrace and the problem I am having is: 
>>    For processes which have been running for more than a couple of 
>>hours, dtrace doesn''t seem to be able to trace(I waited for at
least
>>one hour and didn''t see anything happening). However, it works
every
>>time for a relatively new process.
>>    
>>
>
>What probes are you tracing?  I''ve seen processes run for hours
without
>making a single system call, for example, and it''s even possible
that a
>process could run for hours without taking any page faults or receiving
>signals.  Attaching to it with a debugger and single-stepping it may
>help.  Watching it go on and off cpu with dtrace may help show that
it''s
>executing something.
>_______________________________________________
>dtrace-discuss mailing list
>dtrace-discuss at opensolaris.org
>  
>