Howdy,
I read Adam''s BLOG and the lockstat section of the DTrace users guide
in
an attempt to correlate execnames to non-zero values in the mpstat smtx
column. My first attempt to solve this question with DTrace produced
the following (per Adam''s BLOG):
$ dtrace -n ''lockstat:::*spin { @num[execname] = count(); }''
dtrace: description ''lockstat:::*spin '' matched 5 probes
^C
$
dtrace didn''t print anything when I exit''ed with ^C, but smtx
was greater
than zero while I was attempting to capture the entity responsible for the
non-zero value:
$ mpstat 5
CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt
idl
0 6538 0 0 545 376 288 42 0 1 0 6009 26 74 0
0
0 6110 0 0 409 309 251 43 0 1 0 5718 34 66 0
0
0 6330 0 0 546 445 299 60 0 4 0 5782 35 65 0
0
0 6281 2 0 406 306 283 49 0 2 0 5667 35 65 0
0
This got me curious, so I read mpstat.c, and saw that mpstat used the
mutex_adenters kstat value to determine smtx. I dumped mutex_adenters into
the source browser and saw that mutex_adenters was incremented by
CPU_STATS_ADDQ in mutex_vector_enter*. Does anyone happen to know which
lock events will cause program execution to enter mutex_vector_enter
(and hence indirectly update the value reported by mpstat''s smtx
value)?
Is DTrace able to capture these events? There is probably a better way to
determine what is causing smtx to be non-zero, so I thought I would ask
the folks on the list.
Thanks for any insight,
- Ryan
* Whoever wrote "Big Theory Statement for mutual exclusion locking
primitives" in mutex.c did a kickass job!
============================================================== Matty :
UNIX Administrator : GPG ID: 92D5DFFF
Home/BLOG : http://daemons.net/~matty
Public Key : http://daemons.net/~matty/public_key.txt
Fingerprint : 4BEC 6145 30A6 BCE6 5602 FF11 4954 165D 92D5 DFFF
===============================================================
Hi Ryan, To continue your analysis a little more: if you look at the expansion of the macro that increments the mutex_adenters kstat, you''ll see that this is the source of information for the sysinfo provider. To enable a probe that corresponds precisely to what you''re seeing in mpstat, you can enable the sysinfo:::mutex_adenters probe.> * Whoever wrote "Big Theory Statement for mutual exclusion locking > primitives" in mutex.c did a kickass job!That would be the prolific and indefatigable Jeff Bonwick. Adam On Wed, Sep 14, 2005 at 12:18:41AM -0400, Matty wrote:> > Howdy, > > I read Adam''s BLOG and the lockstat section of the DTrace users guide in > an attempt to correlate execnames to non-zero values in the mpstat smtx > column. My first attempt to solve this question with DTrace produced > the following (per Adam''s BLOG): > > $ dtrace -n ''lockstat:::*spin { @num[execname] = count(); }'' > dtrace: description ''lockstat:::*spin '' matched 5 probes > ^C > > $ > > dtrace didn''t print anything when I exit''ed with ^C, but smtx was greater > than zero while I was attempting to capture the entity responsible for the > non-zero value: > > $ mpstat 5 > CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt > idl > 0 6538 0 0 545 376 288 42 0 1 0 6009 26 74 0 > 0 > 0 6110 0 0 409 309 251 43 0 1 0 5718 34 66 0 > 0 > 0 6330 0 0 546 445 299 60 0 4 0 5782 35 65 0 > 0 > 0 6281 2 0 406 306 283 49 0 2 0 5667 35 65 0 > 0 > > This got me curious, so I read mpstat.c, and saw that mpstat used the > mutex_adenters kstat value to determine smtx. I dumped mutex_adenters into > the source browser and saw that mutex_adenters was incremented by > CPU_STATS_ADDQ in mutex_vector_enter*. Does anyone happen to know which > lock events will cause program execution to enter mutex_vector_enter > (and hence indirectly update the value reported by mpstat''s smtx value)? > Is DTrace able to capture these events? There is probably a better way to > determine what is causing smtx to be non-zero, so I thought I would ask > the folks on the list. > > Thanks for any insight, > - Ryan > > * Whoever wrote "Big Theory Statement for mutual exclusion locking > primitives" in mutex.c did a kickass job! > > ==============================================================> Matty : UNIX Administrator : GPG ID: 92D5DFFF > Home/BLOG : http://daemons.net/~matty > Public Key : http://daemons.net/~matty/public_key.txt > Fingerprint : 4BEC 6145 30A6 BCE6 5602 FF11 4954 165D 92D5 DFFF > ==============================================================> _______________________________________________ > dtrace-discuss mailing list > dtrace-discuss at opensolaris.org-- Adam Leventhal, Solaris Kernel Development http://blogs.sun.com/ahl
On Wed, Sep 14, 2005 at 12:18:41AM -0400, Matty wrote:> > Howdy, > > I read Adam''s BLOG and the lockstat section of the DTrace users guide in > an attempt to correlate execnames to non-zero values in the mpstat smtx > column. My first attempt to solve this question with DTrace produced > the following (per Adam''s BLOG): > > $ dtrace -n ''lockstat:::*spin { @num[execname] = count(); }'' > dtrace: description ''lockstat:::*spin '' matched 5 probes > ^CThis will only fire if someone spun for the lock without blocking; on a single-processor system, we (almost always) block. adaptive-block will fire in that case.> dtrace didn''t print anything when I exit''ed with ^C, but smtx was greater > than zero while I was attempting to capture the entity responsible for the > non-zero value: > > $ mpstat 5 > CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt > idl > 0 6538 0 0 545 376 288 42 0 1 0 6009 26 74 0 > 0 > 0 6110 0 0 409 309 251 43 0 1 0 5718 34 66 0 > 0 > 0 6330 0 0 546 445 299 60 0 4 0 5782 35 65 0 > 0 > 0 6281 2 0 406 306 283 49 0 2 0 5667 35 65 0 > 0 > > This got me curious, so I read mpstat.c, and saw that mpstat used the > mutex_adenters kstat value to determine smtx. I dumped mutex_adenters into > the source browser and saw that mutex_adenters was incremented by > CPU_STATS_ADDQ in mutex_vector_enter*. Does anyone happen to know which > lock events will cause program execution to enter mutex_vector_enter > (and hence indirectly update the value reported by mpstat''s smtx value)? > Is DTrace able to capture these events? There is probably a better way to > determine what is causing smtx to be non-zero, so I thought I would ask > the folks on the list.For each bump of mutex_adenters, exactly one of lockstat:::adaptive-spin or lockstat:::adaptive-block will fire. Just add lockstat:::adaptive-block to your probe spec, and it should work. Part of the confusion is from the mpstat(1M) manpage; characterizing smtx as as "spin on mutex" is inaccurate; it''s really "fast adaptive path failed". Presumably, the meaning is historical. Cheers, - jonathan -- Jonathan Adams, Solaris Kernel Development