dtrace discuss - Oct 2007 - DTrace debugging and dynamic variable drops

I am fairly new to dtrace and having trouble with dtrace debugging in
general and dynamic variable drops in particular - below are some
questions pertaining to this.  My version of Dtrace is 1.1 and it is
being run on Solaris 10 SPARC, i.e.:

[ken at walleye dtrace]$ dtrace -V 
dtrace: Sun D 1.1 
[ken at walleye dtrace]$
uname -a SunOS walleye 5.10 Generic_118833-33 sun4u sparc SUNW,Ultra-60


I read the slides 21 & 22 describing dynamic variable drops here:

http://blogs.sun.com/roller/resources/bmc/dtrace_tips.pdf

which are helpful for understanding the root cause of the problem, but
not particularly instructive as to how to solve it, i.e. tuning
dynvarsize and cleanrate most likely will not address the root cause of
my problem.


Questions

1.  Can one get dtrace to definitively indicate which thread-local
variables exist?  That is, my current approach to finding active
thread-local variables is to inspect the source.  It there a better
way?

2.  How does one zero a string type thread-local variable?  Assign a
null string (e.g. self->mystring = "";)?

3.  Similarly, how does one zero out a struct or union type
thread-local variable?  Explicitly zero each field of the struct or one
field of the union?

4.  Do clause-local variables need to be zeroed or is their storage
automatically reclaimed when the clause exits?  Aside: what is the
difference between a clause and a probe?

5.  I''ve primarily used printf() to debug my dtrace scripts which is
fairly cumbersome.  Is there a tutorial describing how to do dtrace
debugging?  The stop() and breakpoint() calls imply one can use mdb(1)
or similar debugger, but indicate extreme care needs to be taken or the
system may become wedged.  Some of the dtrace(1M) options (e.g. -f -n)
imply additional tracing info can be gotten, but I''ve failed to gain
anything useful from them.

6.  On a separate topic, I''ve read about the need for a stable network
provider and that some work has been done on this.  Is a network
provider available?


--
This message posted from opensolaris.org

Ken,

See notes below on the questions that I know something about.

Chip

Ken Nelson wrote:
> I am fairly new to dtrace and having trouble with dtrace debugging in
> general and dynamic variable drops in particular - below are some
> questions pertaining to this.  My version of Dtrace is 1.1 and it is
> being run on Solaris 10 SPARC, i.e.:
>
> [ken at walleye dtrace]$ dtrace -V 
> dtrace: Sun D 1.1 
> [ken at walleye dtrace]$
> uname -a SunOS walleye 5.10 Generic_118833-33 sun4u sparc SUNW,Ultra-60
>
>
> I read the slides 21 & 22 describing dynamic variable drops here:
>
> http://blogs.sun.com/roller/resources/bmc/dtrace_tips.pdf
>
> which are helpful for understanding the root cause of the problem, but
> not particularly instructive as to how to solve it, i.e. tuning
> dynvarsize and cleanrate most likely will not address the root cause of
> my problem.
>
>
> Questions
>
> 1.  Can one get dtrace to definitively indicate which thread-local
> variables exist?  That is, my current approach to finding active
> thread-local variables is to inspect the source.  It there a better
> way?
>   
Not that I know of.
> 2.  How does one zero a string type thread-local variable?  Assign a
> null string (e.g. self->mystring = "";)?
>   
You always assign numeric 0 to a thread-local variable to release it, 
regardless of its type.
> 3.  Similarly, how does one zero out a struct or union type
> thread-local variable?  Explicitly zero each field of the struct or one
> field of the union?
>   
That''s an interesting one.  I hadn''t thought of that.  The
manual says
to always assign a 0, and that works for all the cases I''ve ever used
or
seen,  (integers, strings, pointers to structures, associative arrays, 
etc.), but I never seen anyone use a thread-local structure.  What''s 
interesting is that it is syntactically correct to do the following:

struct abc
{
   int x;
   int y;
} zzz;

syscall::open:entry
{
   self->x = zzz;
}

but you can''t assign 0 to self->x without incurring a type mismatch 
violation.  I don''t know the answer to this one.  At the same time, I 
can''t think of a reason I would need to create a thread local 
structure.  Pointer to structure, yes.
> 4.  Do clause-local variables need to be zeroed or is their storage
> automatically reclaimed when the clause exits?  Aside: what is the
> difference between a clause and a probe?
>   
Clause local variables are re-claimed at the end of the clause.  
However, when released, the memory doesn''t get cleared, so if you refer
to a clause-local variable without initializing it first, you might get 
the last value of some other clause local variable.

A probe is the hook in the OS that causes the clause to execute.  You 
specify which probes to instrument (or activate) with the tuple above 
the clause (i.e. syscall::open:entry).  The clause is the D code in 
braces that gets executed when the probe fires.  You can have many 
probes associated with one clause, or many clauses associated with one 
probe.
> 5.  I''ve primarily used printf() to debug my dtrace scripts which
is
> fairly cumbersome.  Is there a tutorial describing how to do dtrace
> debugging?  The stop() and breakpoint() calls imply one can use mdb(1)
> or similar debugger, but indicate extreme care needs to be taken or the
> system may become wedged.  Some of the dtrace(1M) options (e.g. -f -n)
> imply additional tracing info can be gotten, but I''ve failed to
gain
> anything useful from them.
>   
The stop action causes the process that fired the probe to stop the next 
time it leaves the kernel (i.e. after it finishes the system call that 
caused the probe to fire).  This doesn''t really help you debug D 
scripts.  And breakpoint puts the kernel in debug mode, which doesn''t 
really help with debugging D programs.

I think you may be confused about the purpose of the -f and -n options.  
Their purpose is just to specify how the command line probe tuples are 
specified.  For example, using -f means that the probe tuple that 
follows will end with the function part (provider:module:function, 
rather than provider:module:function:name).

Here''s the bottom line.  It''s hard for DTrace to include
traditional
debugging info (breakpoints, code trace, etc.) because the compiled D 
program (the DIF code) runs in kernel mode.  As such, it can''t process 
interrupts.  Actions which would cause an interrupt (like trace or 
printf) are executed by the dtrace process "out-of-band" of the kernel
DIF code.  The kernel DIF code places trace data into a buffer, and then 
the dtrace process pulls it out of the buffer later and prints
it.
> 6.  On a separate topic, I''ve read about the need for a stable
network
> provider and that some work has been done on this.  Is a network
> provider available?
>
>
> --
> This message posted from opensolaris.org
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org
>

G''Day Ken,

On Mon, Oct 22, 2007 at 07:59:48AM -0700, Ken Nelson wrote:
[...]
> 6.  On a separate topic, I''ve read about the need for a stable
network
> provider and that some work has been done on this.  Is a network
> provider available?
Not yet; I was working on this, and yes, it''s important to get
completed.
I''ve been called to work on other important things for a bit. 
I''m looking
at returning to this and completing it by the end of this year.

Brendan

-- 
Brendan
[CA, USA]

On Mon, Oct 22, 2007 at 07:59:48AM -0700, Ken Nelson
wrote:
> 1.  Can one get dtrace to definitively indicate which thread-local
> variables exist?  That is, my current approach to finding active
> thread-local variables is to inspect the source.  It there a better
> way?
For a running process, you can attach do it with mdb(1) (mdb -p <pid>) and
then do ::nm -t tls. You can see the value by doing <thread-id>::tls
<symbol>.
> 2.  How does one zero a string type thread-local variable?  Assign a
> null string (e.g. self->mystring = "";)?
You should be able to do self->mystring = NULL;
> 3.  Similarly, how does one zero out a struct or union type
> thread-local variable?  Explicitly zero each field of the struct or one
> field of the union?
This is a long-standing bug there there''s no way to delete such an
object.
> 4.  Do clause-local variables need to be zeroed or is their storage
> automatically reclaimed when the clause exits?
Clause-local variables don''t necessarily start out at zero when a probe
fires.
> Aside: what is the difference between a clause and a probe?
A clause is everything between curly braces. A probe is an activation point
in the DTrace framework. Officially we refer to ''clause-local''
variables, but
that''s a little inaccurate since they retain their value for the
duration of
a probe''s firing. For that reason I''ve described them as
''probe-local'' (e.g.
here http://blogs.sun.com/ahl/entry/dtrace_boot_camp). Most accurately, they
would probably be called ''probe-activation-local'' or something
similarly
obscure.
> 5.  I''ve primarily used printf() to debug my dtrace scripts which
is
> fairly cumbersome.  Is there a tutorial describing how to do dtrace
> debugging?  The stop() and breakpoint() calls imply one can use mdb(1)
> or similar debugger, but indicate extreme care needs to be taken or the
> system may become wedged.  Some of the dtrace(1M) options (e.g. -f -n)
> imply additional tracing info can be gotten, but I''ve failed to
gain
> anything useful from them.
Both stop() and breakpoint() impact the target rather than the D script
itself. We''ve discussed adding debugging framework for D scripts (e.g.
the
putative ''dint'' -- D lint -- and D stabs), but
haven''t made any progress
in that direction because there have been more pressing needs.

Adam

-- 
Adam Leventhal, FishWorks                        http://blogs.sun.com/ahl

Adam Leventhal wrote:
> On Mon, Oct 22, 2007 at 07:59:48AM -0700, Ken Nelson wrote:
>   
>
>> Aside: what is the difference between a clause and a probe?
>>     
>
> A clause is everything between curly braces. A probe is an activation point
> in the DTrace framework. Officially we refer to
''clause-local'' variables, but
> that''s a little inaccurate since they retain their value for the
duration of
> a probe''s firing. For that reason I''ve described them as
''probe-local'' (e.g.
> here http://blogs.sun.com/ahl/entry/dtrace_boot_camp). Most accurately,
they
> would probably be called ''probe-activation-local'' or
something similarly
> obscure.
>
>   
That''s right:  The extent of clause-locals span all of the clause 
executions for the firing of one probe, rather than just one clause.  I 
forgot about that.  Thanks Adam.

Sometimes DTrace users are lulled into thinking that the extent of 
clause-locals is larger.  Since DTrace doesn''t clear them, sometimes
the
old value is still present the next time the probe fires, or even when a 
different probe fires that executes the same clause, but you can''t rely
on that value.

Chip