dtrace discuss - Jan 2008 - DTrace in Perl: What probes should we have?

As of patch 32953 dtrace support is in bleadperl (5.11.0). The probes  
are based on Alan Burlinson''s original blog post on the subject:

     http://blogs.sun.com/alanbur/date/20050909

By guarding the probes with PERL_SUB_*_ENABLED the performance hit is  
unmeasurable.

All the necessary bits already existed in the wild. I just assembled  
them and made the necessary changes to Perl''s Configure script. The  
patched Perl works with the examples in DTraceToolkit.

Currently we have probes on subroutine entry and return.

Next I''d like to solicit input about what other probes would be  
useful. I have Sven Dowideit''s patch that adds probes on new__sv,  
del__sv (creation and deletion of values) and, main__enter, main__exit  
(interpreter lifecycle) and load__module (require, use etc).

Where else might we usefully probe? What would people find useful?

-- notes --

I''m crossposting this to what I hope are a few interested lists. If  
anyone reading this needs a primer on dtrace this video is highly  
recommended:

   http://video.google.com/videoplay?docid=-8002801113289007228

It''s 90 minutes of your life but it''ll be time well spent :)

If anyone would like to try bleadperl it is available here:

rsync -avz --exclude .svn/ --delete \
    rsync://ftp.linux.activestate.com/perl-current/ bleadperl

You can configure it like this:

./Configure -de -Dusedevel -Dinc_version_list=none \
     -Dprefix=$install -Dldflags=-Dman3ext=3pm \
     -Duseithreads -Duseshrplib -Uversiononly \
     -Dusedtrace

where $install is your preferred install base.

Right now ./Configure displays a harmless error about ''!''
being an
unknown command just after the questions. There''s a patch in the  
pipeline to fix that.

-- 
Andy Armstrong, Hexten

On 11 Jan 2008, at 20:45, Andy Armstrong wrote:
> Right now ./Configure displays a harmless error about ''!''
being an
> unknown command just after the questions. There''s a patch in the
> pipeline to fix that.

Fixed in blead (#32963) now.

-- 
Andy Armstrong, Hexten

excellent stuff Andy!

To start with the obvious :)

Due to the lack of ustack helper, I am hoping to add a stack probe - I 
really could do with being able to write a dtrace that shows me a perl 
stack trace on file system access - yesterday I spent a borish day using 
inotifywait and some perl tracing on linux :/


Another other thing I''m missing is the arguments to ops. Oh, and can I 
trace exception throws and catches?


Sven

Andy Armstrong wrote:
> As of patch 32953 dtrace support is in bleadperl (5.11.0). The probes  
> are based on Alan Burlinson''s original blog post on the subject:
>
>      http://blogs.sun.com/alanbur/date/20050909
>
> By guarding the probes with PERL_SUB_*_ENABLED the performance hit is  
> unmeasurable.
>
> All the necessary bits already existed in the wild. I just assembled  
> them and made the necessary changes to Perl''s Configure script.
The
> patched Perl works with the examples in DTraceToolkit.
>
> Currently we have probes on subroutine entry and return.
>
> Next I''d like to solicit input about what other probes would be  
> useful. I have Sven Dowideit''s patch that adds probes on new__sv,
> del__sv (creation and deletion of values) and, main__enter, main__exit  
> (interpreter lifecycle) and load__module (require, use etc).
>
> Where else might we usefully probe? What would people find useful?
>
> -- notes --
>
> I''m crossposting this to what I hope are a few interested lists.
If
> anyone reading this needs a primer on dtrace this video is highly  
> recommended:
>
>    http://video.google.com/videoplay?docid=-8002801113289007228
>
> It''s 90 minutes of your life but it''ll be time well spent
:)
>
> If anyone would like to try bleadperl it is available here:
>
> rsync -avz --exclude .svn/ --delete \
>     rsync://ftp.linux.activestate.com/perl-current/ bleadperl
>
> You can configure it like this:
>
> ./Configure -de -Dusedevel -Dinc_version_list=none \
>      -Dprefix=$install -Dldflags=-Dman3ext=3pm \
>      -Duseithreads -Duseshrplib -Uversiononly \
>      -Dusedtrace
>
> where $install is your preferred install base.
>
> Right now ./Configure displays a harmless error about ''!''
being an
> unknown command just after the questions. There''s a patch in the  
> pipeline to fix that.
>
>   

-- 
Professional Wiki Innovation and Support 
Sven Dowideit - http://DistributedINFORMATION.com 
A WikiRing Partner http://wikiring.com

On Sat, 2008-01-12 at 11:21 +1100, Sven Dowideit wrote:
> excellent stuff Andy!
> 
> To start with the obvious :)
> 
> Due to the lack of ustack helper, I am hoping to add a stack probe
 Two things: first of all DTrace could really benefit from a 
general purpose mechanism of hooking up custom stack helpers
with the rest of the system. Frankly, the way jstack is done 
doesn''t strike me as too generic a mechanism.

  Second, I would like to encourage those who do DTrace integration
into Perl to take a look at what JavaScript is doing. They don''t
have a custom stack helper either but the set of probes they offer
makes stack synthesis a trivial (albeit costly) matter.

Thanks,
Roman.

Roman Shaposhnik wrote:
> On Sat, 2008-01-12 at 11:21 +1100, Sven Dowideit wrote:
>   
>> excellent stuff Andy!
>>
>> To start with the obvious :)
>>
>> Due to the lack of ustack helper, I am hoping to add a stack probe
>>     
>
>  Two things: first of all DTrace could really benefit from a 
> general purpose mechanism of hooking up custom stack helpers
> with the rest of the system. Frankly, the way jstack is done 
> doesn''t strike me as too generic a mechanism.
>
>   Second, I would like to encourage those who do DTrace integration
> into Perl to take a look at what JavaScript is doing. They don''t
> have a custom stack helper either but the set of probes they offer
> makes stack synthesis a trivial (albeit costly) matter.
>
> Thanks,
> Roman.
>   
Yeah, I''m pondering (naively) the idea of activating the perl debugger 
using isenabled.

-- 
Professional Wiki Innovation and Support 
Sven Dowideit - http://DistributedINFORMATION.com 
A WikiRing Partner http://wikiring.com

On 12 Jan 2008, at 01:11, Sven Dowideit wrote:
>> Thanks,
>> Roman.
>>
> Yeah, I''m pondering (naively) the idea of activating the perl  
> debugger using isenabled.

It''s pretty easy to walk up the Perl stack if that helps.

(followups trimmed)

-- 
Andy Armstrong, Hexten

On Fri, Jan 11, 2008 at 04:39:25PM -0800, Roman Shaposhnik wrote:
> > Due to the lack of ustack helper, I am hoping to add a stack probe
> 
>  Two things: first of all DTrace could really benefit from a 
> general purpose mechanism of hooking up custom stack helpers
> with the rest of the system. Frankly, the way jstack is done 
> doesn''t strike me as too generic a mechanism.
Do you have something in mind?

regards
john

On Fri, Jan 11, 2008 at 04:39:25PM -0800, Roman Shaposhnik
wrote:
> On Sat, 2008-01-12 at 11:21 +1100, Sven Dowideit wrote:
> > excellent stuff Andy!
> > 
> > To start with the obvious :)
> > 
> > Due to the lack of ustack helper, I am hoping to add a stack probe
> 
>  Two things: first of all DTrace could really benefit from a 
> general purpose mechanism of hooking up custom stack helpers
> with the rest of the system. Frankly, the way jstack is done 
> doesn''t strike me as too generic a mechanism.
Do you actually understand how it''s implemented?  It''s
actually quite
general -- witness the presence of ustack helpers for both Python and
PHP.  (Indeed, the underlying mechanism is _so_ general that for some
time we weren''t sure if we had fallen prey to false generality.)  The
problem -- such as it is -- is not the lack of generality, it''s that
they
are brutally difficult to write, in large part because the problem that
ustack helpers are trying to solve is a brutally difficult problem.
(Most VMs are not prepared to perform stack introspection in an arbitrary
context.)
>   Second, I would like to encourage those who do DTrace integration
> into Perl to take a look at what JavaScript is doing. They don''t
> have a custom stack helper either but the set of probes they offer
> makes stack synthesis a trivial (albeit costly) matter.
While the JavaScript approach is better than nothing, it is _not_
preferable to the ustack approach -- but a JavaScript ustack helper is
prohibitively difficult at the moment because the information required
simply doesn''t exist when it''s needed.

So, Perl folks:  if you can do it, a ustack helper is the way to go. 
It''s
brutal, but the payoff is substantial, as it will be much easier to connect
misbehaving Perl to the symptoms of that misbehavior elsewhere in the
system.  If you''re interested in seeing a demo of this, check out John
Levon''s blog on the Python ustack helper that he implemented:

  http://blogs.sun.com/levon/entry/python_and_dtrace_in_build

And my description (and video) of demonstrating John''s outstanding work
at Google:

  http://blogs.sun.com/bmc/entry/dtrace_at_google

	- Bryan

--------------------------------------------------------------------------
Bryan Cantrill, Sun Microsystems FishWorks.       http://blogs.sun.com/bmc

On Sat, Jan 12, 2008 at 01:13:29AM +0000, Andy Armstrong
wrote:
> On 12 Jan 2008, at 01:11, Sven Dowideit wrote:
> >> Thanks,
> >> Roman.
> >>
> > Yeah, I''m pondering (naively) the idea of activating the perl
> > debugger using isenabled.
> 
> 
> It''s pretty easy to walk up the Perl stack if that helps.
"Pretty easy" might be relative.  In probe context, all we can do is
loads from the registers and from the address space -- that''s it.  (And
that
includes bootstrapping stack iteration.)  So you need to have all
metastructures available at a known location, and have your helper be
able to get from those structures into a string that represents a frame.
This is certainly possible (viz. helpers in Java and Python with a 
prototype PHP helper), but it''s arduous.  So hopefully we have the same
definition of "pretty easy", in which case a ustack helper for Perl
will
be possible...

	- Bryan

--------------------------------------------------------------------------
Bryan Cantrill, Sun Microsystems FishWorks.       http://blogs.sun.com/bmc

On 12 Jan 2008, at 01:35, Bryan Cantrill wrote:
> "Pretty easy" might be relative.  In probe context, all we can do
is
> loads from the registers and from the address space -- that''s it.
> (And that
> includes bootstrapping stack iteration.)  So you need to have all
> metastructures available at a known location, and have your helper be
> able to get from those structures into a string that represents a  
> frame.
> This is certainly possible (viz. helpers in Java and Python with a
> prototype PHP helper), but it''s arduous.  So hopefully we have the
> same
> definition of "pretty easy", in which case a ustack helper for
Perl
> will
> be possible...

OK, "pretty easy until you actually understand the problem" :)

Where does the source of the Java and Python helpers live?

-- 
Andy Armstrong, Hexten

On 12 Jan 2008, at 01:32, Bryan Cantrill wrote:
> So, Perl folks:  if you can do it, a ustack helper is the way to  
> go.  It''s
> brutal, but the payoff is substantial, as it will be much easier to  
> connect
> misbehaving Perl to the symptoms of that misbehavior elsewhere in the
> system.  If you''re interested in seeing a demo of this, check out
John
> Levon''s blog on the Python ustack helper that he implemented:
>
>  http://blogs.sun.com/levon/entry/python_and_dtrace_in_build

I read that Bryan. I got the impression - maybe incorrectly - that it  
relies on the Python interpreter recursing when it enters a Python  
function - so that a walk up the C stack visits all entries in the  
Python stack. Did I get that wrong?

Perl doesn''t recurse in C when it calls a Perl subroutine so walking  
up the stack won''t yield a Perl call chain.

-- 
Andy Armstrong, Hexten

On Sat, 2008-01-12 at 01:30 +0000, John Levon wrote:
> On Fri, Jan 11, 2008 at 04:39:25PM -0800, Roman Shaposhnik wrote:
> 
> > > Due to the lack of ustack helper, I am hoping to add a stack
probe
> > 
> >  Two things: first of all DTrace could really benefit from a 
> > general purpose mechanism of hooking up custom stack helpers
> > with the rest of the system. Frankly, the way jstack is done 
> > doesn''t strike me as too generic a mechanism.
> 
> Do you have something in mind?
  Discalimer: I can''t really claim to be an expert in DTrace, my
main familiarity with it comes from working on a tool that sometimes
pushes DTrace to its limits and that way I''m forced to explore how
these limits could be overcome. If what I have to say below is a
wrong perception of the technology I''d love to be educated by the
subject experts.

  My main beef with stack helpers is that they have to be executed 
inside the kernel and are subject to the usual DTrace restrictions.
I would like to see a possibility of offloading more work to user
space in a generic fashion. For example, we have a libcollector.so
facility that is capable of reconstructing stacks under the most
challenging of situations (inside the function preamble, etc.) it is 
way more accurate in what it reports back than ustack() it is also
capable of more stack unwinding than jstack(). Of course, it runs in
user space.

  We do have some preliminary ideas on how such functionality
can be hooked to  the rest of DTrace machinery. If there''s a 
significant portion of the DTrace community that is interested in
extending DTrace functionality in a way that would make it more 
reliant on userspace -- perhaps it is a good thing to have on the 
DTrace conference agenda.

Thanks,
Roman.

On Fri, 2008-01-11 at 17:32 -0800, Bryan Cantrill wrote:
> On Fri, Jan 11, 2008 at 04:39:25PM -0800, Roman Shaposhnik wrote:
> > On Sat, 2008-01-12 at 11:21 +1100, Sven Dowideit wrote:
> > > excellent stuff Andy!
> > > 
> > > To start with the obvious :)
> > > 
> > > Due to the lack of ustack helper, I am hoping to add a stack
probe
> > 
> >  Two things: first of all DTrace could really benefit from a 
> > general purpose mechanism of hooking up custom stack helpers
> > with the rest of the system. Frankly, the way jstack is done 
> > doesn''t strike me as too generic a mechanism.
> 
> Do you actually understand how it''s implemented?  
  Not fully, but I''ve spent a great deal of time staring at
jhelper.d  ;-) 

  Seriously, though -- my experience comes mostly from reading the 
source. And the source code for DTrace can get pretty tricky. If you
have any suggestion for me as to what would be a good intro into
stack helpers -- I''d appreciate that.
> It''s actually quite
> general -- witness the presence of ustack helpers for both Python and
> PHP.  (Indeed, the underlying mechanism is _so_ general that for some
> time we weren''t sure if we had fallen prey to false generality.) 
  I guess we are talking about two different kinds of generality here.
What I have in mind is the generality that would allow DTrace to be
able to offload some of the heavylifting to the userspace. I DO
understand the ramifications of that and I know full well that pushing
DTrace into userspace might be perceived as the deviation from the
original design. To some extent it might even reduce the generality
that you seem to be referring to.
> >   Second, I would like to encourage those who do DTrace integration
> > into Perl to take a look at what JavaScript is doing. They
don''t
> > have a custom stack helper either but the set of probes they offer
> > makes stack synthesis a trivial (albeit costly) matter.
> 
> While the JavaScript approach is better than nothing, it is _not_
> preferable to the ustack approach 
  Agreed. Still it is WAY better than nothing ;-)

Thanks,
Roman.

On Fri, Jan 11, 2008 at 06:23:14PM -0800, Roman Shaposhnik
wrote:
> 
>   I guess we are talking about two different kinds of generality here.
> What I have in mind is the generality that would allow DTrace to be
> able to offload some of the heavylifting to the userspace. I DO
> understand the ramifications of that and I know full well that pushing
> DTrace into userspace might be perceived as the deviation from the
> original design. To some extent it might even reduce the generality
> that you seem to be referring to.
> 
The issue is not one of design, but of the constraints on the problem.
DTrace actions must be executed in probe context.  Heavy lifting can be
done via post-processing in userland (symbol translation, system()
actions, etc), but the data gathering must be done in arbitrary context.
How would you call arbitrary userspace code from, say, an interrupt
handler? 

- Eric

--
Eric Schrock, FishWorks                        http://blogs.sun.com/eschrock

On Sat, Jan 12, 2008 at 02:03:42AM +0000, Andy Armstrong wrote:
> >  http://blogs.sun.com/levon/entry/python_and_dtrace_in_build
> 
> I read that Bryan. I got the impression - maybe incorrectly - that it  
> relies on the Python interpreter recursing when it enters a Python  
> function - so that a walk up the C stack visits all entries in the  
> Python stack. Did I get that wrong?
Nope, that''s right.
> Perl doesn''t recurse in C when it calls a Perl subroutine so
walking
> up the stack won''t yield a Perl call chain.
I always got the impression the difficulty was in walking the stack for
the relevant Perl thread at all, not just the C vs. Perl stack issue.

Whilst the C stack issue is a problem, it''s not necessarily a major
one.
Presuming (and I really don''t know) that it''s easy to get to
the top of
the Perl stack from the available initial state, and it''s feasible to
traverse the relevant structures, in theory we could extend the ustack()
implementation with a yield()-style approach: that is, the helper is
called multiple times and each time yields a relevant string until
"done".

I''m sure Bryan will jump and tell me I''m crazy.

BTW this has been discussed before:

http://mail.opensolaris.org/pipermail/dtrace-discuss/2007-September/004572.html

regards
john

On Jan 11, 2008, at 6:26 PM, Eric Schrock wrote:
> On Fri, Jan 11, 2008 at 06:23:14PM -0800, Roman Shaposhnik wrote:
>>
>>   I guess we are talking about two different kinds of generality  
>> here.
>> What I have in mind is the generality that would allow DTrace to be
>> able to offload some of the heavylifting to the userspace. I DO
>> understand the ramifications of that and I know full well that  
>> pushing
>> DTrace into userspace might be perceived as the deviation from the
>> original design. To some extent it might even reduce the generality
>> that you seem to be referring to.
>>
>
> The issue is not one of design, but of the constraints on the problem.
   If we are talking about *u*stack, than I disagree with the way
you''re
setting up constraints. The *u*stack is by definition a userspace  
related
operation. Not only that, but calling ustack() is declared void for a  
reason.
Calling it changes the output of the DTrace but has no way of affecting
the rest of the DTrace script''s internal logic.
> DTrace actions must be executed in probe context.
   Please don''t redefine the problem. We are not talking about
arbitrary
actions here.
> Heavy lifting can be
> done via post-processing in userland (symbol translation, system()
> actions, etc), but the data gathering must be done in arbitrary  
> context.
   What data gathering? Could you, please, be more specific? What data
there is that can''t be leveraged from the userspace? And again,
I''m
talking specifically about ustack() here.
> How would you call arbitrary userspace code from, say, an interrupt
> handler?
   I''m not suggesting that at all. And in fact for Project D-Light we  
devised
a mechanism for getting much more accurate stacktraces from userspace
compared to what DTrace is capable of. E.g.:

self uint32_t stkIdx;

collector$1:::ustack
{
     printf("0 %d %d %u\n", tid, self->stkIdx++, arg0<<32| 
(arg1&0xffffffff));
}

syscall::read:return
/pid == $1/
{
     fname = (self->fd == 0 ) ? "<stdin>" :
fnames[self->fd];
     printf("1 %d %d %d %d \"%s\" %d %d\n",
            cpu,
            tid,
            timestamp,
            0,
            fname,
            arg1,
            self->stkIdx);
     raise(29);
}

   The awkwardness here comes from the fact that we have to:
      * make libcollector available in the address space of the process
      * use a raise(29) in order to poke userspace instead of  
something more
         DTrace specific.

   Otherwise it works very nice. We get all the benefits of userspace  
code:
we can match dlopen/dlclose events and such and our Java stack are
better than jstack().

Thanks,
Roman.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20080111/0ae4b306/attachment.html>

On 12 Jan 2008, at 03:54, John Levon wrote:
> Whilst the C stack issue is a problem, it''s not necessarily a
major
> one.
> Presuming (and I really don''t know) that it''s easy to get
to the top
> of
> the Perl stack from the available initial state, and it''s feasible
to
> traverse the relevant structures, in theory we could extend the  
> ustack()
> implementation with a yield()-style approach: that is, the helper is
> called multiple times and each time yields a relevant string until
> "done".

I think "available initial state" is where I get stuck.

So far I''ve just stuffed some probes into Perl. I need to equip myself
with a far better understanding of how dtrace is implemented so I have  
at least a vague clue what I''m talking about here...

If dtrace has enough information to locate the current Perl thread  
context then finding and walking the stack is relatively easy.

Of course almost anything in the Perl interpreter is only really easy  
if you have access to all the preprocessor macros defined the same way  
they were when Perl was built. Perl uses macros like it wishes it was  
written in Lisp :)

Before wasting any more of everyone''s time with uninformed speculation
I''m going to go and learn more about how dtrace is implemented.

-- 
Andy Armstrong, Hexten

On 12 Jan 2008, at 00:21, Sven Dowideit wrote:
> Another other thing I''m missing is the arguments to ops.
D''you mean the ops (like OP_ENTERSUB) that Perl compiles to?
They''re a
fixed size structure but the interpretation of the fields depends on  
which op it is.
> Oh, and can I trace exception throws and catches?

OP_ENTERTRY and OP_LEAVETRY bracket a block eval and OP_DIE executes a  
die (throw).

-- 
Andy Armstrong, Hexten

I think there''s a misunderstanding here about how ustack() works.  The
constraints of the problem are best summarized as:

	At probe time, record enough data in the buffer such that you
	can reconstruct a userland stack at a later point in dtrace(1M).

	All information that is dependent on the current state of the
	process must be recorded, as the state of the process will have
	changed by the time dtrace(1M) executes.  Any operations
	involved must be safe from arbitrary probe context.  If there
	is some portion that does not depend on the state of the
	process, or depends on optional state that changes infrequently
	(such as the symbol table), this work can be offloaded to
	dtrace(1M) asynchronously (though how to do so for arbitrary
	providers is a complicated problem by itself).

What you have described is not ustack(), but a provider-assisted stack
probe.  This has been done in the past and can be quite useful, but it
is not ustack().  For one, it requires that you record data (the stack
trace from the provider) at all times, even if it won''t end up being
used.  For another, it requires that you instantiate the provider probes
for anything that *may* need a userland stack, i.e. instrument every
process on the system.

The benefit of ustack() is that you can ask a question like
"io:::start{@[execname, ustack()] = count()}".  This only instruments
a
single probe, and the stack is calculated as necessary.  Imagine a world
where every provider had a ''ustack'' probe.  In order to run
the above
script, you would need to instrument every process on the system to
record the ustack at all times just in case they might do I/O.

When you''re drilling down on a specific process where you know the
providers involved,  provider stack traces are quite useful.  For
environments where a ustack() helper is not possible, it is a great
solution, and I would encourage provider developers to create such a
probe for when a truly heavy hammer is needed.  No additional DTrace
infrastructure is needed; simply provide the stack as a string to an
is-enabled USDT probe.  But this is a very different way of observing
processes, and saying things like "ustack() should be generalized to
leverage userspace" doesn''t really apply.  These are two different
tracing methodologies designed for two different tasks.

If you have a way of extending ustack() such that it meets your needs
while satisfying the constraints of the problem, please share it.

- Eric

--
Eric Schrock, FishWorks                        http://blogs.sun.com/eschrock

On Fri, Jan 11, 2008 at 06:12:57PM -0800, Roman Shaposhnik
wrote:
>   My main beef with stack helpers is that they have to be executed 
> inside the kernel and are subject to the usual DTrace restrictions.
> I would like to see a possibility of offloading more work to user
> space in a generic fashion. For example, we have a libcollector.so
> facility that is capable of reconstructing stacks under the most
> challenging of situations (inside the function preamble, etc.) it is 
> way more accurate in what it reports back than ustack() it is also
> capable of more stack unwinding than jstack(). Of course, it runs in
> user space.
All respect to libcollector.so, but the conditions under which DTrace must
gather a stack are far more constrained than those of a user-land program.
The DTrace code from probe context can''t do I/O or block -- it can only
load and store. Even looping is a bit of a stretch.

Adam

-- 
Adam Leventhal, FishWorks                        http://blogs.sun.com/ahl

On Jan 12, 2008, at 12:26 PM, Adam Leventhal wrote:
> On Fri, Jan 11, 2008 at 06:12:57PM -0800, Roman Shaposhnik wrote:
>>   My main beef with stack helpers is that they have to be executed
>> inside the kernel and are subject to the usual DTrace restrictions.
>> I would like to see a possibility of offloading more work to user
>> space in a generic fashion. For example, we have a libcollector.so
>> facility that is capable of reconstructing stacks under the most
>> challenging of situations (inside the function preamble, etc.) it is
>> way more accurate in what it reports back than ustack() it is also
>> capable of more stack unwinding than jstack(). Of course, it runs in
>> user space.
>
> All respect to libcollector.so, but the conditions under which  
> DTrace must
> gather a stack are far more constrained than those of a user-land  
> program.
> The DTrace code from probe context can''t do I/O or block -- it can
> only
> load and store. Even looping is a bit of a stretch.
   I don''t disagree at all! And I''m not trying to say that
*stack()
should be replaced.
I''m merely trying to explore whether the best of both worlds approach  
is doable.
What we''ve done for Project D-Light clearly shows that for some  
applications
it is doable. At this point I really would like to explore whether  
something like
that could be generalized. See my earlier email to Eric. I understand  
his
argument perfectly well. But I''m still not convinced that it makes  
the idea
void. If convincing me is perceived as a waste of time -- that''s ok.  
I guess
I should get back to you when I earn my Solaris kernel guru  
degree :-) If
on the other hand, I''m not explaining what I want clearly -- I can  
try again.

Thanks,
Roman.

On Jan 12, 2008, at 10:16 AM, Eric Schrock wrote:
> I think there''s a misunderstanding here about how ustack() works.
   There''s perhaps some of that (and as I pointed out, I would really
like to be educated more on this subject) but the more I think about
it the more it feels to me that most of the time I don''t need what
ustack() has to offer. I''d much rather have a similar functionality
at much lower price. More on that later.
>   The constraints of the problem are best summarized as:
>
> 	At probe time, record enough data in the buffer such that you
> 	can reconstruct a userland stack at a later point in dtrace(1M).
   Understood. But, there''s very little reason to pay such a hefty
price in most cases where ustack() gets used. That''s the generality
I was alluding to. The mechanism I need is similar to ustack in 90%
of the cases but it should be much less constrained. The example I gave
in my previous email points into a direction I need. The control flow
in my example is like that:

     1. a probe X fires (process gets frozen at this point --  
actually I would
         really like to understand whether  process has any chance of
         going on in such a situation).
     2. control flow gets transfered to the userspace. In my case via  
signal, but
         I''d rather have a generic mechanism for doing it. Note that  
the process
         is still frozen.
     3. signal gets delivered to the process and that triggers all of  
the heavy lifting
         for accurately recording stack information. As I pointed out  
before, since
         we are not constrained we can perform a much more precise  
analysis
         and we ARE demonstrably better than what a general purpose  
ustack()
         can deliver (if you are interested in details I can provide  
them in a separate
         email). We are also capable of querying arbitrary virtual  
machines like
         JVM, OpenMP, etc. Also note that the control flow inside the  
process is
         still stuck at the same point where it was at step 1.
     4. At the end of the stack analysis routine a dedicate USDT  
probe "pstack" fires with
         an argument that would let DTrace script regain control  
understanding from
         where it was transferred to the userspace. Thus it is almost  
like we''re calling
         userspace routines from within the DTrace code:

               X:X:X:X {
                      /* do some DTrace stuff */
                      "call user space"
                      /* I really would like to have an option of  
pstack probe "transferring" control
                          to this point, but perhaps I''m asking too  
much */
               }

               Y:Y:Y:pstack {
                      /* Here I have an access to the very accurate  
stack info */
                }

    Again, for the tool we are building relying on signal trickery is  
not ideal, but
we can survive. If, however, this approach of "transferring" control  
back and forth
between userspace and kernel can be generalized -- I think everybody  
would
benefit from such a mechanism.

    Ok, here comes the biggest rub (from now on I''ll refer to it as  
THE RUB ;-)) currently
there''s not way that I know of to achieve the following two things:
      * poke (send signal) not the current process but an arbitrary one
      * somehow pass an argument through the poking mechanism
    Personally, I''m not enough of a Solaris kernel guru to see  
whether these two
issues are solvable or not.  However, if these are solvable than what  
we can have
a custom helper be part of a DTrace command (or something like that)  
instead
of being part of address space of each process that might need a  
ustack().
> 	All information that is dependent on the current state of the
> 	process must be recorded, as the state of the process will have
> 	changed by the time dtrace(1M) executes.
    Sorry, but I don''t see this as a constraint of the problem. I see  
this
as a constraint of an implementation. DTrace, being a kernel beast,
has a full control over the process state. Stoping and resuming
control flow of the process is not out of the question.
>   Any operations
> 	involved must be safe from arbitrary probe context.
    Could you, please, elaborate on this one a bit more?
>   If there
> 	is some portion that does not depend on the state of the
> 	process, or depends on optional state that changes infrequently
> 	(such as the symbol table), this work can be offloaded to
> 	dtrace(1M) asynchronously (though how to do so for arbitrary
> 	providers is a complicated problem by itself).
    Understood.
> What you have described is not ustack(), but a provider-assisted stack
> probe.  This has been done in the past and can be quite useful, but it
> is not ustack().  For one, it requires that you record data (the stack
> trace from the provider) at all times, even if it won''t end up
being
> used.
    Agree. Although I don''t see it as a big deal. Getting back to my  
example
every time we do signal(29) we are requesting the user stack to be
recorded. There is a slight chance that it won''t get used. But in our
experience it is a slim one. Besides, given that way we are compressing
stacks this unused information is very cheap.
>   For another, it requires that you instantiate the provider probes
> for anything that *may* need a userland stack, i.e. instrument every
> process on the system.
> The benefit of ustack() is that you can ask a question like
> "io:::start{@[execname, ustack()] = count()}".  This only  
> instruments a
> single probe, and the stack is calculated as necessary.
> Imagine a world
> where every provider had a ''ustack'' probe.  In order to
run the above
> script, you would need to instrument every process on the system to
> record the ustack at all times just in case they might do I/O.
   Agreed. See THE RUB ;-) Also, please note that what you are
describing is not quite what I had in mind. In my case, having
a collector$1:::ustack USDT probe is a secondary issue. The stack
will get recorded by just us doing signal(29). The reason we need
the USDT probe at all is to transfer the control flow back to D script.
> When you''re drilling down on a specific process where you know the
> providers involved,  provider stack traces are quite useful.  For
> environments where a ustack() helper is not possible, it is a great
> solution, and I would encourage provider developers to create such a
> probe for when a truly heavy hammer is needed.  No additional DTrace
> infrastructure is needed; simply provide the stack as a string to an
> is-enabled USDT probe.
   But that''s not what I''m talking about.
> But this is a very different way of observing
> processes, and saying things like "ustack() should be generalized to
> leverage userspace" doesn''t really apply.  These are two
different
> tracing methodologies designed for two different tasks.
>
> If you have a way of extending ustack() such that it meets your needs
> while satisfying the constraints of the problem, please share it.

    Ok, let me try to be extra clear: what I need is a mechanism for
passing the control flow back and forth between the kernel executing
the action statements and a particular userspace process in the system.
One place where such mechanism seems to be extremely useful is
designing custom stack helpers which don''t suffer from the huge
disadvantage you''ve outlined above -- the need to instrument every
process on the system.

    We have some ideas of how this could be achieved, but we
lack sufficient expertise in Solaris kernel to suggest a detailed plan
of how this could be implemented in DTrace.

    And finally, I remember Bryan once saying that one of the reasons
he did DTrace in the first place was because a lot of folks thought
it would be impossible to do. I really like this attitude and I hope
my lack of deep kernel knowledge won''t be a reason that a potentially
good idea gets discarded.

Thanks,
Roman.

> > I think there''s a misunderstanding here about how ustack()
works.
> 
>    There''s perhaps some of that (and as I pointed out, I would
really
> like to be educated more on this subject) 
I''m not sure that you actually do want to be more educated; several
people
are trying to educate you here, and it doesn''t seem to be taking...
> but the more I think about
> it the more it feels to me that most of the time I don''t need what
> ustack() has to offer. I''d much rather have a similar
functionality
> at much lower price. More on that later.
> 
> >   The constraints of the problem are best summarized as:
> >
> > 	At probe time, record enough data in the buffer such that you
> > 	can reconstruct a userland stack at a later point in dtrace(1M).
> 
>    Understood. But, there''s very little reason to pay such a hefty
> price in most cases where ustack() gets used. That''s the
generality
> I was alluding to. The mechanism I need is similar to ustack in 90%
> of the cases but it should be much less constrained. 
I''m not sure where you''re getting "90% of the cases"
from -- perhaps you
meant 90% of the cases that you happened to be interested in?
> The example I gave
> in my previous email points into a direction I need. The control flow
> in my example is like that:
> 
>      1. a probe X fires (process gets frozen at this point --  
> actually I would
>          really like to understand whether  process has any chance of
>          going on in such a situation).
And here is the fundamental flaw in your logic:  you are assuming that
the probe that is firing is at user-level.  But this is not the case for
many probes that are quite valuable for understanding the system --
probes related to I/O, CPU scheduling, networking, interprocess
communication, etc.  For these probes, you cannot simply transfer control
to user-level, for a myriad of safety reasons.  Indeed, in these contexts
you can''t even synchronously stop the LWP -- you can only set a bit
indicating that the LWP should stop itself before returning to user-level.
>      2. control flow gets transfered to the userspace. In my case via  
> signal, but
>          I''d rather have a generic mechanism for doing it. Note
that
> the process
>          is still frozen.
>      3. signal gets delivered to the process and that triggers all of  
> the heavy lifting
>          for accurately recording stack information. As I pointed out  
> before, since
>          we are not constrained we can perform a much more precise  
> analysis
>          and we ARE demonstrably better than what a general purpose  
> ustack()
>          can deliver (if you are interested in details I can provide  
> them in a separate
>          email). We are also capable of querying arbitrary virtual  
> machines like
>          JVM, OpenMP, etc. Also note that the control flow inside the  
> process is
>          still stuck at the same point where it was at step 1.
>      4. At the end of the stack analysis routine a dedicate USDT  
> probe "pstack" fires with
>          an argument that would let DTrace script regain control  
> understanding from
>          where it was transferred to the userspace. Thus it is almost  
> like we''re calling
>          userspace routines from within the DTrace code:
> 
>                X:X:X:X {
>                       /* do some DTrace stuff */
>                       "call user space"
>                       /* I really would like to have an option of  
> pstack probe "transferring" control
>                           to this point, but perhaps I''m asking
too
> much */
>                }
> 
>                Y:Y:Y:pstack {
>                       /* Here I have an access to the very accurate  
> stack info */
>                 }
> 
>     Again, for the tool we are building relying on signal trickery is  
> not ideal, but
> we can survive. If, however, this approach of "transferring"
control
> back and forth
> between userspace and kernel can be generalized -- I think everybody  
> would
> benefit from such a mechanism.
Again, this just isn''t the way the world works -- when you are deep
in the context of the kernel, you cannot simply revector control to
user-land for arbitrary execution.  
>     Ok, here comes the biggest rub (from now on I''ll refer to it
as
> THE RUB ;-)) currently
> there''s not way that I know of to achieve the following two
things:
>       * poke (send signal) not the current process but an arbitrary one
>       * somehow pass an argument through the poking mechanism
>     Personally, I''m not enough of a Solaris kernel guru to see  
> whether these two
> issues are solvable or not.  However, if these are solvable than what  
> we can have
> a custom helper be part of a DTrace command (or something like that)  
> instead
> of being part of address space of each process that might need a  
> ustack().
> 
> > 	All information that is dependent on the current state of the
> > 	process must be recorded, as the state of the process will have
> > 	changed by the time dtrace(1M) executes.
> 
>     Sorry, but I don''t see this as a constraint of the problem. I
see
> this
> as a constraint of an implementation. DTrace, being a kernel beast,
> has a full control over the process state. Stoping and resuming
> control flow of the process is not out of the question.
Yes it is, actually:  DTrace would lose its ability to instrument arbitrary
contexts if it demanded synchronous interaction with complicated subsystems
like process control.
> >   Any operations
> > 	involved must be safe from arbitrary probe context.
> 
>     Could you, please, elaborate on this one a bit more?
> 
> >   If there
> > 	is some portion that does not depend on the state of the
> > 	process, or depends on optional state that changes infrequently
> > 	(such as the symbol table), this work can be offloaded to
> > 	dtrace(1M) asynchronously (though how to do so for arbitrary
> > 	providers is a complicated problem by itself).
> 
>     Understood.
> 
> > What you have described is not ustack(), but a provider-assisted stack
> > probe.  This has been done in the past and can be quite useful, but it
> > is not ustack().  For one, it requires that you record data (the stack
> > trace from the provider) at all times, even if it won''t end
up being
> > used.
> 
>     Agree. Although I don''t see it as a big deal. Getting back to
my
> example
> every time we do signal(29) we are requesting the user stack to be
> recorded. There is a slight chance that it won''t get used. But in
our
> experience it is a slim one. Besides, given that way we are compressing
> stacks this unused information is very cheap.
> 
> >   For another, it requires that you instantiate the provider probes
> > for anything that *may* need a userland stack, i.e. instrument every
> > process on the system.
> > The benefit of ustack() is that you can ask a question like
> > "io:::start{@[execname, ustack()] = count()}".  This only  
> > instruments a
> > single probe, and the stack is calculated as necessary.
> > Imagine a world
> > where every provider had a ''ustack'' probe.  In order
to run the above
> > script, you would need to instrument every process on the system to
> > record the ustack at all times just in case they might do I/O.
> 
>    Agreed. See THE RUB ;-) Also, please note that what you are
> describing is not quite what I had in mind. In my case, having
> a collector$1:::ustack USDT probe is a secondary issue. The stack
> will get recorded by just us doing signal(29). The reason we need
> the USDT probe at all is to transfer the control flow back to D script.
> 
> > When you''re drilling down on a specific process where you
know the
> > providers involved,  provider stack traces are quite useful.  For
> > environments where a ustack() helper is not possible, it is a great
> > solution, and I would encourage provider developers to create such a
> > probe for when a truly heavy hammer is needed.  No additional DTrace
> > infrastructure is needed; simply provide the stack as a string to an
> > is-enabled USDT probe.
> 
>    But that''s not what I''m talking about.
> 
> > But this is a very different way of observing
> > processes, and saying things like "ustack() should be generalized
to
> > leverage userspace" doesn''t really apply.  These are two
different
> > tracing methodologies designed for two different tasks.
> >
> > If you have a way of extending ustack() such that it meets your needs
> > while satisfying the constraints of the problem, please share it.
> 
> 
>     Ok, let me try to be extra clear: what I need is a mechanism for
> passing the control flow back and forth between the kernel executing
> the action statements and a particular userspace process in the system.
You''re asking for the operating system equivalent of a perpetual motion
machine:  if an instrumentation framework allowed this kind of revectoring
of control, then there would -- by definition -- be kernel contexts that
would be uninstrumentable.  Specifically, one could not instrument the
contexts that are responsible for revectoring control into and the
execution of user-level instructions -- which includes CPU scheduling,
the VM system, I/O, the filesystem, etc.  As the design center of DTrace is
very much to be able to instrument these delicate contexts, probe-context
user-level revectoring mechanisms are therefore impossible.
>     We have some ideas of how this could be achieved, but we
> lack sufficient expertise in Solaris kernel to suggest a detailed plan
> of how this could be implemented in DTrace.
> 
>     And finally, I remember Bryan once saying that one of the reasons
> he did DTrace in the first place was because a lot of folks thought
> it would be impossible to do. I really like this attitude and I hope
> my lack of deep kernel knowledge won''t be a reason that a
potentially
> good idea gets discarded.
It''s not a "potentially good idea", but rather a traditional
debugger-centric
notion that reflects a fundamental misunderstanding of the constraints
of DTrace.  Which is not to say that one couldn''t implement a system
around
these notions, but rather that such a system would be so different as to
longer be DTrace.  Indeed, it its features and limitations, it would look
much more like a conventional, process-oriented debugger -- which should be
no surprise given your background and bias...

	- Bryan

--------------------------------------------------------------------------
Bryan Cantrill, Sun Microsystems FishWorks.       http://blogs.sun.com/bmc

On Sat, Jan 12, 2008 at 10:46:02AM +0000, Andy Armstrong wrote:
> If dtrace has enough information to locate the current Perl thread  
> context then finding and walking the stack is relatively easy.
It''s up to the helper, not dtrace. You have the IP counter and the
stack
pointer: can you find the Perl thread from that? If not, all is lost. If
not, then see my previous post. It''s entirely up to the Perl stack, or
what the Perl stack can reasonably be extended to do.

regards
john

On Jan 12, 2008, at 4:29 PM, Bryan Cantrill wrote:
>
>>> I think there''s a misunderstanding here about how ustack()
works.
>>
>>    There''s perhaps some of that (and as I pointed out, I would
really
>> like to be educated more on this subject)
>
> I''m not sure that you actually do want to be more educated;
several
> people
> are trying to educate you here, and it doesn''t seem to be
taking...
    I''m sorry if I wasted your time. It wasn''t my intent. Now,
since
this will be,
perhaps, the last email between us I just hope you do realize that
starting an answer with the sentence like that significantly devalues  
your
subsequent arguments. Why bother writing the rest if the point of the
reply is to flame?
>>     Again, for the tool we are building relying on signal trickery is
>> not ideal, but
>> we can survive. If, however, this approach of "transferring"
control
>> back and forth
>> between userspace and kernel can be generalized -- I think everybody
>> would
>> benefit from such a mechanism.
>
> Again, this just isn''t the way the world works -- when you are
deep
> in the context of the kernel, you cannot simply revector control to
> user-land for arbitrary execution.
    I don''t think you followed my example (the actual D code). If you
did you would have seen that I specifically picked a kernel level
probe (syscall::read) to illustrate how our crude but useful mechanism
works. Nobody is suggesting immediately revectoring the control
flow. As in my example a signal doesn''t get delivered immediately.
However it is guaranteed to be delivered before the process in question
gets any chance to chug forward. I agree with the points you made later
in your email, that instrumenting delicate places deep inside the kernel
can not rely on such vectoring of the control flow. I agree with that.
The same way these places can not rely on, lets say system(..)
destructive action. Yet I can use system(...) in such a handler,  
can''t I?
And even with all the restrictions (and yes I do know about them)
it happens to be quite useful at times. As I pointed out in my email
to Adam -- I was merely trying to suggest a mechanism that would
complement stack() and ustack().
>>
>>     Ok, let me try to be extra clear: what I need is a mechanism for
>> passing the control flow back and forth between the kernel executing
>> the action statements and a particular userspace process in the  
>> system.
>
> You''re asking for the operating system equivalent of a perpetual  
> motion
> machine.
    Analogies cut both ways. You of all people should know that. Trying
to invent a perpetual motion machine is, perhaps unwise.  Talking about
it and carrying out though experiments proved to be quite beneficial to
physics.
> It''s not a "potentially good idea", but rather a
traditional
> debugger-centric
> notion that reflects a fundamental misunderstanding of the constraints
> of DTrace.  Which is not to say that one couldn''t implement a  
> system around
> these notions, but rather that such a system would be so different  
> as to
> longer be DTrace.  Indeed, it its features and limitations, it  
> would look
> much more like a conventional, process-oriented debugger -- which  
> should be
> no surprise given your background and bias...
   Sorry. For a minute there I forgot that DTrace was perfect. And it  
is just
silly me trying to do stupid things (like zeroing out struct elements  
or even
strings) and asking questions that don''t even deserve an answer. Sorry.
It won''t happen again.

Thanks,
Roman.

> >>     Again, for the tool we are building relying on signal trickery
is
> >> not ideal, but
> >> we can survive. If, however, this approach of
"transferring" control
> >> back and forth
> >> between userspace and kernel can be generalized -- I think
everybody
> >> would
> >> benefit from such a mechanism.
> >
> > Again, this just isn''t the way the world works -- when you
are deep
> > in the context of the kernel, you cannot simply revector control to
> > user-land for arbitrary execution.
> 
>     I don''t think you followed my example (the actual D code). If
you
> did you would have seen that I specifically picked a kernel level
> probe (syscall::read) to illustrate how our crude but useful mechanism
> works. 
Yes, but you already have -- and have long had -- a mechanism to do exactly
this:  /proc.  If you would like to stop a process on every system call --
or more generally if you are more interested in stopping and controlling
processes than in observing their dynamic interactions with the system
at-large -- use /proc and be done with it.
> Nobody is suggesting immediately revectoring the control
> flow. As in my example a signal doesn''t get delivered immediately.
> However it is guaranteed to be delivered before the process in question
> gets any chance to chug forward. I agree with the points you made later
> in your email, that instrumenting delicate places deep inside the kernel
> can not rely on such vectoring of the control flow. I agree with that.
> The same way these places can not rely on, lets say system(..)
> destructive action. Yet I can use system(...) in such a handler,  
> can''t I?
Yes, but system() doesn''t the work the way you appear to think that it
does:  system() merely records which command you wish to be executed --
the actual execution of that command does not occur until that datum
is processed (asynchronously) back in user-land.  The target process is
not stopped, and the command certainly does not execute synchronously
with respect to probe context (and nor could it).
> >>     Ok, let me try to be extra clear: what I need is a mechanism
for
> >> passing the control flow back and forth between the kernel
executing
> >> the action statements and a particular userspace process in the  
> >> system.
> >
> > You''re asking for the operating system equivalent of a
perpetual
> > motion
> > machine.
> 
>     Analogies cut both ways. You of all people should know that. Trying
> to invent a perpetual motion machine is, perhaps unwise.  Talking about
> it and carrying out though experiments proved to be quite beneficial to
> physics.
I''m not sure how much grant money you could get to investigate
perpetual
motion machines, but you''d like to spend your time and energy on the
software
equivalent, don''t let me get in your way...
> > It''s not a "potentially good idea", but rather a
traditional
> > debugger-centric
> > notion that reflects a fundamental misunderstanding of the constraints
> > of DTrace.  Which is not to say that one couldn''t implement a
> > system around
> > these notions, but rather that such a system would be so different  
> > as to
> > longer be DTrace.  Indeed, it its features and limitations, it  
> > would look
> > much more like a conventional, process-oriented debugger -- which  
> > should be
> > no surprise given your background and bias...
> 
>    Sorry. For a minute there I forgot that DTrace was perfect. And it  
> is just
> silly me trying to do stupid things (like zeroing out struct elements  
> or even
> strings) and asking questions that don''t even deserve an answer.
Sorry.
> It won''t happen again.
I was merely pointing out that we have different constraints on our
problem than you find on more traditional debuggers.  Those different
constraints convey both strength and weakness:  on the one hand, DTrace
is able to instrument contexts and answer system-level questions for which
traditional debuggers are useless, but on the other, DTrace has a much
more difficult time ascertaining information that is immediately useful
to the developer.  And if I was curt, it was out of frustration that the
misunderstanding of those constraints was coupled with an unyielding
assertion of a notion that completely violates them...

	- Bryan

--------------------------------------------------------------------------
Bryan Cantrill, Sun Microsystems FishWorks.       http://blogs.sun.com/bmc