dtrace discuss - Jul 2007 - What am I missing in the dtrace script?

I''ve asked around, I tried searching on-line, but no go.

I would think what I am seeing is impossible or that there must
be a bug in dtrace on MP systems (at least on 64-bit amd systems)
as this was run on a Sun Fire x4500 (4-way Opteron - aka "Thumper")

Here is the script:


#! /usr/sbin/dtrace -s

BEGIN
{
       total_queued = 0;
       queuing_failed = 0;
       queuing_succeeded = 0;
}

::vdev_queue_io_to_issue:return
{
       ++total_queued;
}

::vdev_queue_io_to_issue:return
/ args[1] == 0 /
{
       ++queuing_failed;
}

::vdev_queue_io_to_issue:return
/ args[1] != 0 /
{
       ++queuing_succeeded;
}

END
{
       printf("\nTotal %d succeeded %d failed %d\n",
               total_queued, queuing_succeeded, queuing_failed);
}

And below is the output:

^C
CPU     ID                    FUNCTION:NAME
 3      2                             :END
Total 970818 succeeded 485824 failed 485584

Note that:

# bc
485824+485584
971408

So the sum of the two cases with qualifiers do not sum
to the the total.  I''ve seen this before and running the script
with all but one CPU off line seems to "fix" the problem.

With all but one processor off line:

^C
CPU     ID                    FUNCTION:NAME
 0      2                             :END
Total 959314 succeeded 479657 failed 479657


# bc
479657+479657
959314

It is also suspect that "succeeded" and "failed" have
the exact same value, but I haven''t analyzed that.

Is it a known issue that probes might be double counted or
missed on MP systems?  Am I crazy?  Am I trying to do
something that dtrace doesn''t really support?
--
This message posted from opensolaris.org

Brian D. Horn wrote:
> I''ve asked around, I tried searching on-line, but no go.
> 
> I would think what I am seeing is impossible or that there must
> be a bug in dtrace on MP systems (at least on 64-bit amd systems)
> as this was run on a Sun Fire x4500 (4-way Opteron - aka
"Thumper")
> 
> Here is the script:
> 
> 
> #! /usr/sbin/dtrace -s
> 
> BEGIN
> {
>        total_queued = 0;
>        queuing_failed = 0;
>        queuing_succeeded = 0;
> }
[...]
> And below is the output:
> 
> ^C
> CPU     ID                    FUNCTION:NAME
>  3      2                             :END
> Total 970818 succeeded 485824 failed 485584
> 
> Note that:
> 
> # bc
> 485824+485584
> 971408
> 
> So the sum of the two cases with qualifiers do not sum
> to the the total.  I''ve seen this before and running the script
> with all but one CPU off line seems to "fix" the problem.
[...]
> Is it a known issue that probes might be double counted or
> missed on MP systems?  Am I crazy?  Am I trying to do
> something that dtrace doesn''t really support?
you''re using global variables. I don''t think DTrace does any 
synchronisation about its own globals, ie you may be missing some 
updates. running with only once CPU eliminates any possible race 
condition, so you see this go away.

use thread - local variables (self-> ...), that should help.

HTH
Michael

-- 
Michael Schuster	Sun Microsystems, Inc.
recursion, n: see ''recursion''

On Tue, Jul 10, 2007 at 03:27:46PM -0700, Brian D. Horn
wrote:
> I''ve asked around, I tried searching on-line, but no go.
> 
> I would think what I am seeing is impossible or that there must
> be a bug in dtrace on MP systems (at least on 64-bit amd systems)
> as this was run on a Sun Fire x4500 (4-way Opteron - aka
"Thumper")
> 
> Here is the script:
> 
> 
> #! /usr/sbin/dtrace -s
> 
> BEGIN
> {
>        total_queued = 0;
>        queuing_failed = 0;
>        queuing_succeeded = 0;
> }
> 
> ::vdev_queue_io_to_issue:return
> {
>        ++total_queued;
> }
> 
> ::vdev_queue_io_to_issue:return
> / args[1] == 0 /
> {
>        ++queuing_failed;
> }
> 
> ::vdev_queue_io_to_issue:return
> / args[1] != 0 /
> {
>        ++queuing_succeeded;
> }
> 
> END
> {
>        printf("\nTotal %d succeeded %d failed %d\n",
>                total_queued, queuing_succeeded, queuing_failed);
> }
> 
> And below is the output:
> 
> ^C
> CPU     ID                    FUNCTION:NAME
>  3      2                             :END
> Total 970818 succeeded 485824 failed 485584
> 
> Note that:
> 
> # bc
> 485824+485584
> 971408
> 
> So the sum of the two cases with qualifiers do not sum
> to the the total.  I''ve seen this before and running the script
> with all but one CPU off line seems to "fix" the problem.
When you''re using global variables, they are modified without any
locking
whatsoever -- so those increments are racing with one another on different
CPUs.  To achieve what you''re trying to achieve, use aggregations, and
then printa() to process them:

::vdev_queue_io_to_issue:return
{
	@total_queued = count();
}

::vdev_queue_io_to_issue:return
/args[1] == 0/
{
	@queuing_failed = count();
}

::vdev_queue_io_to_issue:return
/args[1] != 0/
{
	@queuing_succeeded = count();
}

END
{
	printa("\nTotal %@d succeeded %@d failed %@d\n",
	    @total_queued, @queuing_succeeded, @queuing_failed);
}

	- Bryan

--------------------------------------------------------------------------
Bryan Cantrill, Solaris Kernel Development.       http://blogs.sun.com/bmc

Bryan Cantrill wrote:
> On Tue, Jul 10, 2007 at 03:27:46PM -0700, Brian D. Horn wrote:
>   
>> I''ve asked around, I tried searching on-line, but no go.
>>
>> I would think what I am seeing is impossible or that there must
>> be a bug in dtrace on MP systems (at least on 64-bit amd systems)
>> as this was run on a Sun Fire x4500 (4-way Opteron - aka
"Thumper")
>>
>> Here is the script:
>>
>>
>> #! /usr/sbin/dtrace -s
>>
>> BEGIN
>> {
>>        total_queued = 0;
>>        queuing_failed = 0;
>>        queuing_succeeded = 0;
>> }
>>
>> ::vdev_queue_io_to_issue:return
>> {
>>        ++total_queued;
>> }
>>
>> ::vdev_queue_io_to_issue:return
>> / args[1] == 0 /
>> {
>>        ++queuing_failed;
>> }
>>
>> ::vdev_queue_io_to_issue:return
>> / args[1] != 0 /
>> {
>>        ++queuing_succeeded;
>> }
>>
>> END
>> {
>>        printf("\nTotal %d succeeded %d failed %d\n",
>>                total_queued, queuing_succeeded, queuing_failed);
>> }
>>
>> And below is the output:
>>
>> ^C
>> CPU     ID                    FUNCTION:NAME
>>  3      2                             :END
>> Total 970818 succeeded 485824 failed 485584
>>
>> Note that:
>>
>> # bc
>> 485824+485584
>> 971408
>>
>> So the sum of the two cases with qualifiers do not sum
>> to the the total.  I''ve seen this before and running the
script
>> with all but one CPU off line seems to "fix" the problem.
>>     
>
> When you''re using global variables, they are modified without any
locking
> whatsoever -- so those increments are racing with one another on different
> CPUs.  To achieve what you''re trying to achieve, use aggregations,
and
> then printa() to process them:
>   
BTW, where is it documented that globals are not lock protected but 
aggregates are?
The closest thing I saw as a in a "dos and don''ts" section
that is was
recommended
to use aggregates when possible, but that was about it.
> ::vdev_queue_io_to_issue:return
> {
> 	@total_queued = count();
> }
>
> ::vdev_queue_io_to_issue:return
> /args[1] == 0/
> {
> 	@queuing_failed = count();
> }
>
> ::vdev_queue_io_to_issue:return
> /args[1] != 0/
> {
> 	@queuing_succeeded = count();
> }
>
> END
> {
> 	printa("\nTotal %@d succeeded %@d failed %@d\n",
> 	    @total_queued, @queuing_succeeded, @queuing_failed);
> }
>
> 	- Bryan
>
> --------------------------------------------------------------------------
> Bryan Cantrill, Solaris Kernel Development.       http://blogs.sun.com/bmc
>

On 7/11/07, Brian D. Horn <Brian.Horn at sun.com>
wrote:
> BTW, where is it documented that globals are not lock protected but
> aggregates are?
aggregates need not be lock protected.  It works on a per-cpu buffer.

See
<http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/uts/common/sys/dtrace_impl.h#247>



-- 
Just me,
Wire ...
Blog: <prstat.blogspot.com>

Wee Yeh Tan wrote:
> On 7/11/07, Brian D. Horn <Brian.Horn at sun.com> wrote:
>> BTW, where is it documented that globals are not lock protected but
>> aggregates are?
>
> aggregates need not be lock protected.  It works on a per-cpu buffer.
>
> See 
>
<http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/uts/common/sys/dtrace_impl.h#247>
>
>
Ok, then the question becomes "Where is documented that aggregates are 
MP safe, but scalars are
not MP safe?"

On 7/13/07, Brian D. Horn <Brian.Horn at sun.com>
wrote:
> Ok, then the question becomes "Where is documented that aggregates are
> MP safe, but scalars are
> not MP safe?"
Hah... I sort of saw that coming.  I recall seeing this somewhere
though I cannot remember exactly where.  Didn''t see it in
Bryan''s
DTrace gotchas.

Does this warrant a bug against the documentation?  Perhaps a note in
the variables section saying that "Global variables are not MP safe."?


-- 
Just me,
Wire ...
Blog: <prstat.blogspot.com>

Wee Yeh Tan wrote:
> On 7/13/07, Brian D. Horn <Brian.Horn at sun.com> wrote:
>> Ok, then the question becomes "Where is documented that aggregates
are
>> MP safe, but scalars are
>> not MP safe?"
>
> Hah... I sort of saw that coming.  I recall seeing this somewhere
> though I cannot remember exactly where.  Didn''t see it in
Bryan''s
> DTrace gotchas.
>
> Does this warrant a bug against the documentation?  Perhaps a note in
> the variables section saying that "Global variables are not MP
safe."?
>
I don''t know whether it is a bug or not yet.  I may have missed some
piece of documentation or seen old documentation.  It also begs the
question of "Why was it decided that scalars should not be MP safe?"
(assuming that is was decided as such!).

On Thu, Jul 12, 2007 at 05:47:50PM -0700, Brian D. Horn
wrote:
> Wee Yeh Tan wrote:
> > On 7/13/07, Brian D. Horn <Brian.Horn at sun.com> wrote:
> >> Ok, then the question becomes "Where is documented that
aggregates are
> >> MP safe, but scalars are
> >> not MP safe?"
> >
> > Hah... I sort of saw that coming.  I recall seeing this somewhere
> > though I cannot remember exactly where.  Didn''t see it in
Bryan''s
> > DTrace gotchas.
> >
> > Does this warrant a bug against the documentation?  Perhaps a note in
> > the variables section saying that "Global variables are not MP
safe."?
> >
> I don''t know whether it is a bug or not yet.  I may have missed
some
> piece of documentation or seen old documentation.  It also begs the
> question of "Why was it decided that scalars should not be MP
safe?"
> (assuming that is was decided as such!).
It _was_ decided as such, and it''s because global scalars -- MP-safe or
not -- do not scale.  Indeed, we could quite easily make increment
and decrement MP-safe, but I have resisted it because it sends the 
wrong message:  global scalars are almost always the wrong tool for the
job; as I instructed you, you should be using aggregations for this
instead.  If you''d like to file a bug against the documentation for
this
to be more clearly spelled out, feel free...

	- Bryan

--------------------------------------------------------------------------
Bryan Cantrill, Solaris Kernel Development.       http://blogs.sun.com/bmc

Brian D. Horn wrote:
> Wee Yeh Tan wrote:
>> On 7/13/07, Brian D. Horn <Brian.Horn at sun.com> wrote:
>>> Ok, then the question becomes "Where is documented that
aggregates are
>>> MP safe, but scalars are
>>> not MP safe?"
>> Hah... I sort of saw that coming.  I recall seeing this somewhere
>> though I cannot remember exactly where.  Didn''t see it in
Bryan''s
>> DTrace gotchas.
>>
>> Does this warrant a bug against the documentation?  Perhaps a note in
>> the variables section saying that "Global variables are not MP
safe."?
>>
> I don''t know whether it is a bug or not yet.  I may have missed
some
> piece of documentation or seen old documentation.  It also begs the
> question of "Why was it decided that scalars should not be MP
safe?"
> (assuming that is was decided as such!).
The question here isn''t whether "scalaers are mt-safe",
it''s whether
or not the ++ and -- operators are atomic.

Would you expect

foo = foo + 1;

to work safely?

How about:

bar = foo;

foo = bar + 1;

There''s no implicit locking in DTrace.  Aggregations are specifically
designed to be safe, and are documented as such:
> For performance-related questions, aggregated data is often more 
 > useful than individual data points. DTrace provides several
built-in
> aggregating functions. When an aggregating function is applied to 
> subsets of a collection of data, then applied again to the results 
> of the analysis of those subsets, the results are identical to the 
> results returned by the aggregating function when it is applied to 
> the collection as a whole.
> 
> The DTrace facility stores a running count of data items for 
> aggregations. The aggregating functions store only the current 
> intermediate result and the new element that the function is being 
> applied to. The intermediate results are allocated on a per-CPU basis.
> Because this allocation scheme does not require locks, the implementation 
> is inherently scalable.


-- 
Bart Smaalders			Solaris Kernel Performance
barts at cyber.eng.sun.com		http://blogs.sun.com/barts

Bart Smaalders wrote:
> Brian D. Horn wrote:
>> Wee Yeh Tan wrote:
>>> On 7/13/07, Brian D. Horn <Brian.Horn at sun.com> wrote:
>>>> Ok, then the question becomes "Where is documented that
aggregates are
>>>> MP safe, but scalars are
>>>> not MP safe?"
>>> Hah... I sort of saw that coming.  I recall seeing this somewhere
>>> though I cannot remember exactly where.  Didn''t see it in
Bryan''s
>>> DTrace gotchas.
>>>
>>> Does this warrant a bug against the documentation?  Perhaps a note
in
>>> the variables section saying that "Global variables are not MP
safe."?
>>>
>> I don''t know whether it is a bug or not yet.  I may have
missed some
>> piece of documentation or seen old documentation.  It also begs the
>> question of "Why was it decided that scalars should not be MP
safe?"
>> (assuming that is was decided as such!).
>
> The question here isn''t whether "scalaers are mt-safe",
it''s whether
> or not the ++ and -- operators are atomic.
>
> Would you expect
>
> foo = foo + 1;
>
> to work safely?
>
> How about:
>
> bar = foo;
>
> foo = bar + 1;
>
> There''s no implicit locking in DTrace.  Aggregations are
specifically
> designed to be safe, and are documented as such:
>
>> For performance-related questions, aggregated data is often more 
> > useful than individual data points. DTrace provides several built-in
>> aggregating functions. When an aggregating function is applied to 
>> subsets of a collection of data, then applied again to the results of 
>> the analysis of those subsets, the results are identical to the 
>> results returned by the aggregating function when it is applied to 
>> the collection as a whole.
>>
>> The DTrace facility stores a running count of data items for 
>> aggregations. The aggregating functions store only the current 
>> intermediate result and the new element that the function is being 
>> applied to. The intermediate results are allocated on a per-CPU basis.
>> Because this allocation scheme does not require locks, the 
>> implementation is inherently scalable.
Ah ha!  This statement was what I was looking for and not finding.
I think a more clear statement should be in the documentation, but at 
least there is something!

Now, is in absolutely clear what *is* an aggregate?  I would think that 
global dynamic arrays
are not, correct?
>
>
>

Brian D. Horn wrote:
> Now, is in absolutely clear what *is* an aggregate?  I would think that 
> global dynamic arrays
> are not, correct?
If it has an @, it''s an aggregation.  Global arrays are not indeed
not aggregations.

- Bart



-- 
Bart Smaalders			Solaris Kernel Performance
barts at cyber.eng.sun.com		http://blogs.sun.com/barts

Brian D. Horn wrote:
> Bart Smaalders wrote:
>> Brian D. Horn wrote:
>>> Wee Yeh Tan wrote:
>>>> On 7/13/07, Brian D. Horn <Brian.Horn at sun.com> wrote:
>>>>> Ok, then the question becomes "Where is documented
that aggregates
>>>>> are
>>>>> MP safe, but scalars are
>>>>> not MP safe?"
>>>> Hah... I sort of saw that coming.  I recall seeing this
somewhere
>>>> though I cannot remember exactly where.  Didn''t see it
in Bryan''s
>>>> DTrace gotchas.
>>>>
>>>> Does this warrant a bug against the documentation?  Perhaps a
note in
>>>> the variables section saying that "Global variables are
not MP safe."?
>>>>
>>> I don''t know whether it is a bug or not yet.  I may have
missed some
>>> piece of documentation or seen old documentation.  It also begs the
>>> question of "Why was it decided that scalars should not be MP
safe?"
>>> (assuming that is was decided as such!).
>>
>> The question here isn''t whether "scalaers are
mt-safe", it''s whether
>> or not the ++ and -- operators are atomic.
>>
>> Would you expect
>>
>> foo = foo + 1;
>>
>> to work safely?
>>
>> How about:
>>
>> bar = foo;
>>
>> foo = bar + 1;
>>
>> There''s no implicit locking in DTrace.  Aggregations are
specifically
>> designed to be safe, and are documented as such:
As far as what my expectations were, well... I wasn''t sure.  I was
aware
that
some sort of locking would be required for that to work, so I asked someone
(without naming the "guilty") who I thought would know what the story
was.
I was told that it should do the locking "appropriately".  I suspect
that
what that person was thinking about was things like the args and
the return values (in fbt probes), but I''m guessing.  So, in short, I 
questioned
how it worked, asked and I allowed myself to be misled.
>>
>>> For performance-related questions, aggregated data is often more 
>> > useful than individual data points. DTrace provides several
built-in
>>> aggregating functions. When an aggregating function is applied to 
>>> subsets of a collection of data, then applied again to the results 
>>> of the analysis of those subsets, the results are identical to the 
>>> results returned by the aggregating function when it is applied to 
>>> the collection as a whole.
>>>
>>> The DTrace facility stores a running count of data items for 
>>> aggregations. The aggregating functions store only the current 
>>> intermediate result and the new element that the function is being 
>>> applied to. The intermediate results are allocated on a per-CPU
basis.
>>> Because this allocation scheme does not require locks, the 
>>> implementation is inherently scalable.
>
> Ah ha!  This statement was what I was looking for and not finding.
> I think a more clear statement should be in the documentation, but at 
> least there is something!
>
> Now, is in absolutely clear what *is* an aggregate?  I would think 
> that global dynamic arrays
> are not, correct?
>>
>>
>>
>
>

Bart Smaalders wrote On 07/13/07 09:46 AM,:
> Brian D. Horn wrote:
> 
>>Wee Yeh Tan wrote:
>>
>>>On 7/13/07, Brian D. Horn <Brian.Horn at sun.com> wrote:
>>>
>>>>Ok, then the question becomes "Where is documented that
aggregates are
>>>>MP safe, but scalars are
>>>>not MP safe?"
>>>
>>>Hah... I sort of saw that coming.  I recall seeing this somewhere
>>>though I cannot remember exactly where.  Didn''t see it in
Bryan''s
>>>DTrace gotchas.
>>>
>>>Does this warrant a bug against the documentation?  Perhaps a note
in
>>>the variables section saying that "Global variables are not MP
safe."?
>>>
>>
>>I don''t know whether it is a bug or not yet.  I may have missed
some
>>piece of documentation or seen old documentation.  It also begs the
>>question of "Why was it decided that scalars should not be MP
safe?"
>>(assuming that is was decided as such!).
> 
> 
> The question here isn''t whether "scalaers are mt-safe",
it''s whether
> or not the ++ and -- operators are atomic.
> 
> Would you expect
> 
> foo = foo + 1;
> 
> to work safely?
> 
> How about:
> 
> bar = foo;
> 
> foo = bar + 1;
> 
> There''s no implicit locking in DTrace.  Aggregations are
specifically
> designed to be safe, and are documented as such:


Isn''t it also true that there is no explicit locking available in
DTrace,
therefore isn''t the user lulled into believeing that this is all taken
care of automatically by DTrace.  I''m not saying it should, only that
the absense of locking needs to be more explicit. The fact that aggregates
are definately documented to be MP-safe doesn''t relieve us from
documenting
what happens in other circumstances.

-Pete.





> 
> 
>>For performance-related questions, aggregated data is often more 
> 
>  > useful than individual data points. DTrace provides several built-in
> 
>>aggregating functions. When an aggregating function is applied to 
>>subsets of a collection of data, then applied again to the results 
>>of the analysis of those subsets, the results are identical to the 
>>results returned by the aggregating function when it is applied to 
>>the collection as a whole.
>>
>>The DTrace facility stores a running count of data items for 
>>aggregations. The aggregating functions store only the current 
>>intermediate result and the new element that the function is being 
>>applied to. The intermediate results are allocated on a per-CPU basis.
>>Because this allocation scheme does not require locks, the
implementation
>>is inherently scalable.
> 
> 
> 
>