dtrace discuss - Sep 2009 - Aggregation key membership test, and dynamic adopting of tracing functionality

Hi

I''m trying to write a DTrace script (or a set of scripts) that will
adopt what it is tracing of over time, based on previously traced data.
For example, my DTrace script may start by collecting some basic
statistics about a set of events, storing them in an aggregation keyed
by event type, and after a while, I would like to start collecting more
detailed information only about the event types that occur most
frequently.  There are various reasons why I would like this kind of
dynamic behavior:

- Collecting the detailed information may be more expensive, as it
  requires copying large data structures from the program process space.
  Thus, I do not want to collect it for *all* event types.

- My TDTrace script can affect what information is made available by the
  traced program, by writing data in the program space and asking to
  collect detailed information only about the events of interest.

- I may want to stop the program and attach a debugger only when a
  particular event type occurs.

[Note that I cannot simply stop the program and run it again with a
different script, as event types tend to change from run to run.]

Now, while I can easily use an aggregation to collect the basic
information, and throw out (using trunc()) unimportant event types,
DTrace does not allow accessing this data and using it in the script''s
predicates.  Note that in my case, even checking whether a key is in an
aggregation would be sufficient -- I do not have to access the actual
aggregated values.

My questions:

 - Is there any way for a DTrace script to test whether a particular key
   belongs to an aggregation?

 - If not, do you have any other idea how can I achieve this kind of
   dynamically adapting tracing behavior? Here are two ideas that I have
   in mind that I would love to get your feedback on:

   - Use some kind of feedback loop: after truncating the aggregation,
     print it, and trace the "write" system calls (that implements the
     printouts) to "catch" the printed keys, and store them in some
     global variables used by the script.

     This option looks quite complicated to me, and I have no idea if it
     can actually work, especially since I''m running the script using
     the Java DTrace API, so printouts are actually processed by the
     java program...

   - Generate another script that runs in parallel and collect
     information about the events type of interest: with this option, we
     print the truncated aggregation (or scan it using the java DTrace
     API), generate another script that only traces the events we''re
     interested in, and run the latter in parallel to the first script
     (that keeps aggregating the basic statistics for all event types).

     I think that this may work, but I''m worried about one issue, which
     is having two scripts with permissions to run destructive
     operations operating on the same process.  Will DTrace allow two
     scripts, both enabling identical pid<x> probes, to run in parallel
     with permissions to run destructive operations?  (The two
     destructive operations that I need are copyout() and stop().)

Thanks,
Yossi

On Sun, Sep 13, 2009 at 4:54 PM, Yossi Lev (Sun Labs) <Yosef.Lev at
sun.com> wrote:
>
> My questions:
>
> - If not, do you have any other idea how can I achieve this kind of
> ?dynamically adapting tracing behavior?
For something this complicated, you might want to look at writing your
own DTrace consumer.  There isn''t really any good documentation on
this, but I have some notes written up if you''re interested.  I
don''t
know that it would be any easier doing this within a D script, if you
could even figure out how to do what you want.  But if you write a
custom DTrace consumer, you could get access to the data without
having to play games with catching the output of dtrace(1M).

Chad

Chad Mynhier wrote:
> On Sun, Sep 13, 2009 at 4:54 PM, Yossi Lev (Sun Labs) <Yosef.Lev at
sun.com> wrote:
>   
>> My questions:
>>
>> - If not, do you have any other idea how can I achieve this kind of
>>  dynamically adapting tracing behavior?
>>     
>
> For something this complicated, you might want to look at writing your
> own DTrace consumer.  
I assume that by that you mean programming directly to the libdtrace 
library --- the one whose interface is said not to be public and 
undocumented --- right?  I saw warnings in various websites regarding 
programming to this interface, especially because it is not public and 
might change in future releases.  Is that still true?

In any case, I would love it if you can send me your notes on it...
> There isn''t really any good documentation on
> this, but I have some notes written up if you''re interested.  I
don''t
> know that it would be any easier doing this within a D script, if you
> could even figure out how to do what you want.  But if you write a
> custom DTrace consumer, you could get access to the data without
> having to play games with catching the output of dtrace(1M).
>   
Well, accessing the data is not really the problem.  I''m using the 
DTrace Java API, that allows you (I think) to take snapshots of 
aggregation while the DTrace script is running.  The problem is putting 
this data back in predicates --- once I started running a script, I 
cannot modify it, so even though I can get the information of what event 
types needed to be monitored, I cannot make the script monitor them for 
me.  This is why I suggested generating another script, and running it 
in parallel.

Do you think that the two scripts solution should work? Here is the high 
level idea:

- A first script starts the program (with the -c switch), and has 
pid$target probes that copyout() some info to let the program know to 
start gathering and reporting basic information about all event types.

- A second script is generated based on the info gathered by the first 
(gathered by the Java DTrace API).  This script has pic<x> probes (where 
x is the process id spawn by the first script), and executes some 
additional copyout() operations to notify the program which event types 
should be monitored in details.

Note that they both have to run destructive operations...

Thanks,
Yossi

> Chad
>