dtrace discuss - Oct 2007 - fbt provider not firing for entry to cv_wait_sig_swap

I think I know what is going on, but I just want to confirm it.

I have a dtrace script (the funky one that iterates over the
speculation buffers) that has an fbt entry probe for the kernel
function cv_wait_sig_swap. Only it didn''t trace anything even though
there was a process that hung in the function. Getting a thread stack
trace, I discovered that cv_wait_sig_swap didn''t appear on the stack,
but cv_wait_sig_swap_core did. Looking at cv_wait_sig_swap and
cv_wait_sig_swap, I found that cv_wait_sig_swap is really just a
wrapper that calls cv_wait_sig_swap_core with the addition of
a single new parameter that is always set to NULL. So, the compiler
does not create a new frame for it, it just sets the parameter and
calls cv_wait_sig_swap_core, presumably leaving the old return address
in place.

So, my guess is that dtrace instruments something in the entry code
that is not being generated for cv_wait_sig_swap. Can someone confirm
that for me? I changed the script to trace cv_wait_sig_swap* so I would
be sure to get it if either one fires.

Also, is there any way to see if there is anything in the speculation
buffers for a running dtrace script? Since the problem is intermittent,
if there is another hang without the buffer being committed, I would
like to confirm that the buffer is used so I can determine if the
problem is in the speculative tracing or if it is only in the iterative
code that prints the speculation out.

Thanks.
-- 
blu

"You''ve added a new disk. Do you want to replace your current
drive, protect your data from a drive failure or expand your
storage capacity?" - Disk management as it should be.
----------------------------------------------------------------------
Brian Utterback - Solaris RPE, Sun Microsystems, Inc.
Ph:877-259-7345, Em:brian.utterback-at-ess-you-enn-dot-kom

On Mon, Oct 15, 2007 at 10:12:09AM -0400, Brian Utterback
wrote:
> I have a dtrace script (the funky one that iterates over the
> speculation buffers) that has an fbt entry probe for the kernel
> function cv_wait_sig_swap. Only it didn''t trace anything even
though
> there was a process that hung in the function. Getting a thread stack
> trace, I discovered that cv_wait_sig_swap didn''t appear on the
stack,
> but cv_wait_sig_swap_core did. Looking at cv_wait_sig_swap and
> cv_wait_sig_swap, I found that cv_wait_sig_swap is really just a
> wrapper that calls cv_wait_sig_swap_core with the addition of
> a single new parameter that is always set to NULL. So, the compiler
> does not create a new frame for it, it just sets the parameter and
> calls cv_wait_sig_swap_core, presumably leaving the old return address
> in place.
Yes. On SPARC, the compiler seems to generate a tail-call in cv_wait_sig_swap()
to cv_wait_sig_swap_core(). This means that you won''t see
cv_wait_sig_swap_core() in the stack backtrace.

Adam

-- 
Adam Leventhal, FishWorks                        http://blogs.sun.com/ahl

Adam Leventhal wrote:
> Yes. On SPARC, the compiler seems to generate a tail-call in
cv_wait_sig_swap()
> to cv_wait_sig_swap_core(). This means that you won''t see
> cv_wait_sig_swap_core() in the stack backtrace.
> 
> Adam
Okay. But does that mean that the fbt::cv_wait_sig_swap:entry probe
will not fire?

-- 
blu

"You''ve added a new disk. Do you want to replace your current
drive, protect your data from a drive failure or expand your
storage capacity?" - Disk management as it should be.
----------------------------------------------------------------------
Brian Utterback - Solaris RPE, Sun Microsystems, Inc.
Ph:877-259-7345, Em:brian.utterback-at-ess-you-enn-dot-kom

> Okay. But does that mean that the fbt::cv_wait_sig_swap:entry probe
> will not fire?
It should and it did for me:

# dtrace -n ''cv_wait_sig_swap:entry''
dtrace: description ''cv_wait_sig_swap:entry'' matched 1 probe
CPU     ID                    FUNCTION:NAME
   0  10242           cv_wait_sig_swap:entry
   0  10242           cv_wait_sig_swap:entry

Thanks
Bhaskar

Hmm Brian, your mileage may vary.

equally it hasn''t worked on S10 Sparc for me, I''ve not tested
on Nevada or S10 x64.

I''ve needed to dtrace entry to cv_wait_sig_swap_core() instead.


-George


Bhaskar.Sarkar at sun.com wrote:

*>> Okay. But does that mean that the fbt::cv_wait_sig_swap:entry probe
*>> will not fire?
*>
*>It should and it did for me:
*>
*># dtrace -n ''cv_wait_sig_swap:entry''
*>dtrace: description ''cv_wait_sig_swap:entry'' matched 1
probe
*>CPU     ID                    FUNCTION:NAME
*>   0  10242           cv_wait_sig_swap:entry
*>   0  10242           cv_wait_sig_swap:entry
*>


George Shepherd
http://clem.uk/~georges/
=============================================================================  
Solaris Revenue Product Engineering:    |  SUN Microsystems
       Core team  -Internet                |  Guillemont Park
   Email: George.Shepherd at Sun.COM          |  Camberley GU17 9QG
   Disclaimer: Less is more, more or less  |  United Kingdom 
==============================================================================