dtrace discuss - Apr 2006 - Calling all DTrace advocates...

All,

Recently, Red Monk analyst Stephen O''Grady has posted about SystemTap,
asking if it is the "Linux response" to DTrace:

  http://www.redmonk.com/sogrady/archives/001494.html

Stephen has asked for those that have used either DTrace or SystemTap to
share their experiences, so I would encourage you to comment on his blog
if you''re in the mood.

For those of you who don''t know Stephen (which I assume includes many
of
you), he''s a very good guy, and in my experience he''s one of
the few
industry analysts who is interested in how a technology is actually
_used_ -- and not just how great it is claimed to be -- so your
experiences will mean quite a bit to him.  Indeed, you will notice that
his blog entry refers to previous blog entries by DTrace veterans Robert
Milkowski and James Dickens.  Many of you have been tireless advocates
of DTrace; your advocacy use of DTrace continue to be a great
inspiration to the DTrace team -- thank you!

	- Bryan

--------------------------------------------------------------------------
Bryan Cantrill, Solaris Kernel Development.       http://blogs.sun.com/bmc

As a DTrace fan I like to hear more stories being told in our community. We
really need more stories. I would love to hear such kind of stories that the
root cause of the problem get identified by the help of DTrace. Is there any
stories list Sun can release to the OpenSolaris so that  we can appreciate how
powerful the Dtrace is?
 
 
This message posted from opensolaris.org

That list would be too large ;-)

Dtrace is so much  an integral part  of our work,  we hardly
think about  that anymore.  DTrace, as  a tool, started when
the developpers began to  provide frequent builds of Solaris
enabled with Dtrace. People started to use it and then, bugs
investigated with the help  of  DTrace would be  tagged with
the Dtrace keyword and or have a DTrace mail list attached.

Over time, that usage just dwindled down by itself since,
there were just too many issues that had being investigated
and there was no need to track such a large list down.

In some respect, It''s not like  DTrace is that sophisticated
in  what it does (_how_ it  does it, is a different matter),
but the lack  of it, makes an OS  so  limited: "I''d like  to
hear success  stories  about people  fixing things on  their
house with a Hammer & Screwdriver set".

But the Dtrace tool  box can also  be used in  sophisticated
ways, so I guess there are lots of good stories to be told...

-r



Qiang Liu writes:
 > As a DTrace fan I like to hear more stories being told in our
 > community. We really need more stories. I would love to hear such kind
 > of stories that the root cause of the problem get identified by the
 > help of DTrace. Is there any stories list Sun can release to the
 > OpenSolaris so that  we can appreciate how powerful the Dtrace is? 
 >  
 >  
 > This message posted from opensolaris.org
 > _______________________________________________
 > dtrace-discuss mailing list
 > dtrace-discuss at opensolaris.org

I''m sure many folks have some good campfire stories to tell over a beer
while building s''mores.

I''ll never forget my first experience using DTrace in the field. I went
out there to give a presentation; afterward, they cornered me and asked 
about a performance problem they were seeing on their workload migrating 
from Solaris 8 to Solaris 10. They logged into their system and I started 
providing them with short DTrace scripts to run on the spot. Going back 
and forth over about an hour (with me constantly digging into the DTrace 
guide, because I''m *far* from an expert at using DTrace) we had it 
narrowed down to a change in behavior in the IP stack between 8 and 10 
which exposed a bug in one of their third-party device drivers. Without 
DTrace we never could have guessed in a million years that an obscure bug 
in a seemingly insignificant device driver was causing a flood of 
involuntary context switches!

Since then I''ve considered observability first whenever starting any
new
project... you can''t improve what you can''t see.

- Eric

On 4/13/06, Roch Bourbonnais - Performance Engineering
<Roch.Bourbonnais at sun.com> wrote:
> That list would be too large ;-)
This is so true.  In fact, I would even label DTrace as addictive.

2 minutes ago, I just resolved a "weird" system behaviour cornering a
process that''s been calling "/usr/lib/sendmail" every 2
minutes.  I
could search the cronlist of all users and match up their process to
the calling time but I took the easy way out of just making DTrace
tell me which guy it is and just sit around for 1 minute and 30
seconds (the 1-liner D script took 30 seconds).  Going through
cronjobs of all the users will take much longer and end up barking up
the wrong tree anyway.  Turns out some guy''s job had a bad time and
cron was trying to inform him.

And yes, this is on a production system.


--
Just me,
Wire ...

Eric Lowe wrote:
> I''m sure many folks have some good campfire stories to tell over a
> beer while building s''mores.
Beer _and_ s''mores ??

Chip Bennett
Laurus Technologies, Sun Partner

On Sat, Apr 15, 2006 at 01:13:19AM +0800, Wee Yeh Tan
wrote:
> On 4/13/06, Roch Bourbonnais - Performance Engineering
> <Roch.Bourbonnais at sun.com> wrote:
> > That list would be too large ;-)
> 
> This is so true.  In fact, I would even label DTrace as addictive.
When we designed DTrace, we set out to build the tool that we had always
wanted; one of the most gratifying things about DTrace is that we ourselves
use it every day to solve problems -- usually problems that we can''t 
even remember how we solved before DTrace.  (And usually we have to remind
ourselves that we basically didn''t solve them, or not quickly anyway:
problems took exponentially longer to solve -- and we know that many
problems went unsolved, because DTrace has been used to find so many legacy
problems.)  Indeed, the fact that we -- and you -- have used DTrace all along
to solve actual problems is what informed (and continues to inform) our
design.  And when DTrace _doesn''t_ solve your problem -- or is missing
some
feature -- it thus always helps us to know the problem that you were trying
to solve in the first place...

So for me, at least, DTrace is more than addictive:  it''s required. 
(Much
to my disappointment, I''ve discovered when losing my cell phone earlier
this week that my phone is in the same category:  how did we ever coordinate
any kind of meeting before the mid 1990s?)  And for me, DTrace isn''t
alone:
MDB, libumem, ::findleaks, and the p-tools are all tools that are simply
not optional when dealing with software...

	- Bryan

--------------------------------------------------------------------------
Bryan Cantrill, Solaris Kernel Development.       http://blogs.sun.com/bmc

On Sat, Apr 15, 2006 at 01:13:19AM +0800, Wee Yeh Tan
wrote:
> On 4/13/06, Roch Bourbonnais - Performance Engineering
> <Roch.Bourbonnais at sun.com> wrote:
> > That list would be too large ;-)
> 
> This is so true.  In fact, I would even label DTrace as addictive.
> 
> 2 minutes ago, I just resolved a "weird" system behaviour
cornering a
> process that''s been calling "/usr/lib/sendmail" every 2
minutes.
Last week I needed to determine if nfsmapidd upcalls, on NFS clients,
happen in the context of the thread that did the syscall that triggered
the NFSv4 GETATTR/SETATTR.  A trivial script, aggregating stack() and
ustack() when the four functions that call door_ki_upcall() to talk to
nfsmapidd, showed that this is [generally] the case (I haven''t proved
that the opposite is never the case (client-side), of course).

Nico
--

Hi.  My name is Dan, and here''s my DTrace testimony...

On Fri, Apr 14, 2006 at 01:04:36PM -0500, Nicolas Williams
wrote:
> On Sat, Apr 15, 2006 at 01:13:19AM +0800, Wee Yeh Tan wrote:
> > On 4/13/06, Roch Bourbonnais - Performance Engineering
> > <Roch.Bourbonnais at sun.com> wrote:
> > > That list would be too large ;-)
> > 
> > This is so true.  In fact, I would even label DTrace as addictive.
I needed to monitor progress in a brute-force MD5 passphrase cracker.  3
non-trivial lines of Dtrace later, I could see how far along the cracker was
by merely checking the input string of the workhorse function of the cracker.

I suspect with an O(minutes) worth of effort (mostly spent RTFM), I could
measure realtime cracks/sec without interrupting the effort.

Dan

On Wed, 12 Apr 2006, Qiang Liu wrote:
> As a DTrace fan I like to hear more stories being told in our community. We
really need more stories. I would love to hear such kind of stories that the
root cause of the problem get identified by the help of DTrace. Is there any
stories list Sun can release to the OpenSolaris so that  we can appreciate how
powerful the Dtrace is?
I have solved a variety of problems with DTrace. Here is my latest win 
(10-minutes by DTrace, countless hours with truss and shell mangling):

http://daemons.net/~matty/blog/?p=377

Dtrace is some sweet livin''
- Ryan
--
UNIX Administrator
http://daemons.net/~matty

On Wed, 12 Apr 2006, Qiang Liu wrote:
> As a DTrace fan I like to hear more stories being told in our community. We
really need more stories. I would love to hear such kind of stories that the
root cause of the problem get identified by the help of DTrace. Is there any
stories list Sun can release to the OpenSolaris so that  we can appreciate how
powerful the Dtrace is?
I have solved a variety of problems with DTrace. Here is my latest win 
(10-minutes by DTrace, countless hours with shell utilities and a version 
of truss w/o the "-E" option):

http://daemons.net/~matty/blog/?p=377

Dtrace is some sweet livin''
- Ryan

--
UNIX Administrator
http://daemons.net/~matty

And then there was that ld.so.1 security bug that could be detected and
prevented with some simple D scripts...

> > I''m sure many folks have some good campfire stories to tell
over
> a 
> > beer while building s''mores.
> 
> Beer _and_ s''mores ??
> 
You still need something to was the s''mores down...
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Joel.Buckley.vcf
Type: text/x-vcard
Size: 348 bytes
Desc: Card for Joel Buckley <Joel.Buckley at Sun.COM>
URL:
<http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20060414/c654c955/attachment.vcf>

On Fri 14 Apr 2006 at 02:55PM, Matty wrote:
> 
> On Wed, 12 Apr 2006, Qiang Liu wrote:
> 
> >As a DTrace fan I like to hear more stories being told in our
community.
> >We really need more stories. I would love to hear such kind of stories 
> >that the root cause of the problem get identified by the help of
DTrace.
> >Is there any stories list Sun can release to the OpenSolaris so that 
we
> >can appreciate how powerful the Dtrace is?
> 
> I have solved a variety of problems with DTrace. Here is my latest win 
> (10-minutes by DTrace, countless hours with shell utilities and a version 
> of truss w/o the "-E" option):
> 
> http://daemons.net/~matty/blog/?p=377
Your blog points up that ''hostname'' is poorly implemented.  I
filed:

        6413595 hostname(1) is needlessly slow

And marked it oss-bite-size.  This would be a good small project for
anyone interested; I''ll happily sponsor this.

        -dp

-- 
Daniel Price - Solaris Kernel Engineering - dp at eng.sun.com - blogs.sun.com/dp