Yossi Lev (Sun Labs)
2009-Sep-13 20:54 UTC
[dtrace-discuss] Aggregation key membership test, and dynamic adopting of tracing functionality
Hi I''m trying to write a DTrace script (or a set of scripts) that will adopt what it is tracing of over time, based on previously traced data. For example, my DTrace script may start by collecting some basic statistics about a set of events, storing them in an aggregation keyed by event type, and after a while, I would like to start collecting more detailed information only about the event types that occur most frequently. There are various reasons why I would like this kind of dynamic behavior: - Collecting the detailed information may be more expensive, as it requires copying large data structures from the program process space. Thus, I do not want to collect it for *all* event types. - My TDTrace script can affect what information is made available by the traced program, by writing data in the program space and asking to collect detailed information only about the events of interest. - I may want to stop the program and attach a debugger only when a particular event type occurs. [Note that I cannot simply stop the program and run it again with a different script, as event types tend to change from run to run.] Now, while I can easily use an aggregation to collect the basic information, and throw out (using trunc()) unimportant event types, DTrace does not allow accessing this data and using it in the script''s predicates. Note that in my case, even checking whether a key is in an aggregation would be sufficient -- I do not have to access the actual aggregated values. My questions: - Is there any way for a DTrace script to test whether a particular key belongs to an aggregation? - If not, do you have any other idea how can I achieve this kind of dynamically adapting tracing behavior? Here are two ideas that I have in mind that I would love to get your feedback on: - Use some kind of feedback loop: after truncating the aggregation, print it, and trace the "write" system calls (that implements the printouts) to "catch" the printed keys, and store them in some global variables used by the script. This option looks quite complicated to me, and I have no idea if it can actually work, especially since I''m running the script using the Java DTrace API, so printouts are actually processed by the java program... - Generate another script that runs in parallel and collect information about the events type of interest: with this option, we print the truncated aggregation (or scan it using the java DTrace API), generate another script that only traces the events we''re interested in, and run the latter in parallel to the first script (that keeps aggregating the basic statistics for all event types). I think that this may work, but I''m worried about one issue, which is having two scripts with permissions to run destructive operations operating on the same process. Will DTrace allow two scripts, both enabling identical pid<x> probes, to run in parallel with permissions to run destructive operations? (The two destructive operations that I need are copyout() and stop().) Thanks, Yossi
Chad Mynhier
2009-Sep-13 22:23 UTC
[dtrace-discuss] Aggregation key membership test, and dynamic adopting of tracing functionality
On Sun, Sep 13, 2009 at 4:54 PM, Yossi Lev (Sun Labs) <Yosef.Lev at sun.com> wrote:> > My questions: > > - If not, do you have any other idea how can I achieve this kind of > ?dynamically adapting tracing behavior?For something this complicated, you might want to look at writing your own DTrace consumer. There isn''t really any good documentation on this, but I have some notes written up if you''re interested. I don''t know that it would be any easier doing this within a D script, if you could even figure out how to do what you want. But if you write a custom DTrace consumer, you could get access to the data without having to play games with catching the output of dtrace(1M). Chad
Yossi Lev (Sun Labs)
2009-Sep-14 12:40 UTC
[dtrace-discuss] Aggregation key membership test, and dynamic adopting of tracing functionality
Chad Mynhier wrote:> On Sun, Sep 13, 2009 at 4:54 PM, Yossi Lev (Sun Labs) <Yosef.Lev at sun.com> wrote: > >> My questions: >> >> - If not, do you have any other idea how can I achieve this kind of >> dynamically adapting tracing behavior? >> > > For something this complicated, you might want to look at writing your > own DTrace consumer.I assume that by that you mean programming directly to the libdtrace library --- the one whose interface is said not to be public and undocumented --- right? I saw warnings in various websites regarding programming to this interface, especially because it is not public and might change in future releases. Is that still true? In any case, I would love it if you can send me your notes on it...> There isn''t really any good documentation on > this, but I have some notes written up if you''re interested. I don''t > know that it would be any easier doing this within a D script, if you > could even figure out how to do what you want. But if you write a > custom DTrace consumer, you could get access to the data without > having to play games with catching the output of dtrace(1M). >Well, accessing the data is not really the problem. I''m using the DTrace Java API, that allows you (I think) to take snapshots of aggregation while the DTrace script is running. The problem is putting this data back in predicates --- once I started running a script, I cannot modify it, so even though I can get the information of what event types needed to be monitored, I cannot make the script monitor them for me. This is why I suggested generating another script, and running it in parallel. Do you think that the two scripts solution should work? Here is the high level idea: - A first script starts the program (with the -c switch), and has pid$target probes that copyout() some info to let the program know to start gathering and reporting basic information about all event types. - A second script is generated based on the info gathered by the first (gathered by the Java DTrace API). This script has pic<x> probes (where x is the process id spawn by the first script), and executes some additional copyout() operations to notify the program which event types should be monitored in details. Note that they both have to run destructive operations... Thanks, Yossi> Chad >