dtrace discuss - Aug 2007 - Closed Loop Control Systems utilizing DTrace for monitoring

I am looking a writing a piece of control software for
closed loop management of system resources.

Is it possible for DTrace to monitor a measure and
launch a non-DTrace procedure when an event occurs?
e.g. monitor.d calls "/bin/speedup ..." or "/bin/slowdown
..."
of the DTrace monitoring uncovers busy or idle
resources and continues to monitor the system...

Or is it possible for a monitoring software to utilize
DTrace to obtain a non-String result?  Can DTrace
return an Object or Structure?

Is there a way to avoid DTrace Startup/Shutdown
resource utilizations, if a DTrace script needs to be
run several times?

Is there a good method for a DTrace procedure to
trigger, say every millisecond, to compare system
status on several kernel values?

Cheers,
Joel.

Joel,

Er, yes...

Joel Buckley wrote:
> I am looking a writing a piece of control software for
> closed loop management of system resources.
> 
> Is it possible for DTrace to monitor a measure and
> launch a non-DTrace procedure when an event occurs?
> e.g. monitor.d calls "/bin/speedup ..." or "/bin/slowdown
..."
> of the DTrace monitoring uncovers busy or idle
> resources and continues to monitor the system...
If you allow destructive actions, DTrace can execute arbitrary commands, via the
exec() action.

For example:

	syscall::exec:entry
	/execname != "dtrace"/
	{
		exec("logger -t dtrace somebody just did an exec");
	}

integrates a DTrace probe with syslog.  Not particularly well, one might
argue...
> Or is it possible for a monitoring software to utilize
> DTrace to obtain a non-String result?  Can DTrace
> return an Object or Structure?
DTrace can output anything it can get to via the context of a user or kernel 
thread, at the time an enabled probe in that thread fires.  So, the answer is 
yes, but you may find it a task to get the data you want to another process in a
form that is usable.  Becoming a DTrace consumer via libdtrace may be the 
easiest way.
> Is there a way to avoid DTrace Startup/Shutdown
> resource utilizations, if a DTrace script needs to be
> run several times?
Not sure about that one, I will let others answer.
> Is there a good method for a DTrace procedure to
> trigger, say every millisecond, to compare system
> status on several kernel values?
On a single CPU:

	profile:::tick-1000

On every CPU:

	profile:::profile-1000

Regards,
Tim

-- 
Tim Cook
Performance and Applications Engineering
<> Sun Microsystems

On Thu, Aug 02, 2007 at 03:49:00PM -0700, Tim Cook - PAE
wrote:
> Joel Buckley wrote:
> > Is it possible for DTrace to monitor a measure and
> > launch a non-DTrace procedure when an event occurs?
> > e.g. monitor.d calls "/bin/speedup ..." or
"/bin/slowdown ..."
> > of the DTrace monitoring uncovers busy or idle
> > resources and continues to monitor the system...
> 
> If you allow destructive actions, DTrace can execute arbitrary commands,
via the
> exec() action.
> 
> For example:
> 
> 	syscall::exec:entry
> 	/execname != "dtrace"/
> 	{
> 		exec("logger -t dtrace somebody just did an exec");
> 	}
> 
> integrates a DTrace probe with syslog.  Not particularly well, one might
argue...
But it''d be a lot better to just parse the output of the script into
syslog() calls (e.g., in a Perl5 script, or in a Java program using the
Java DTrace interfaces).

And yes, you can have your D scripts general XML output if you like.

Keep in mind though that DTrace is robust in its effect on production
systems: it will drop events if need be.  That also makes it not robust
for auditing purposes.
> > Is there a way to avoid DTrace Startup/Shutdown
> > resource utilizations, if a DTrace script needs to be
> > run several times?
> 
> Not sure about that one, I will let others answer.
Just keep the script running.  The only thing you can''t dynamically add
is PID provider probes for specific processes (AFAIK).  But then again
you could just use the Java interface to DTrace.

On Thu, Aug 02, 2007 at 06:10:33PM -0500, Nicolas Williams
wrote:
> 
> But it''d be a lot better to just parse the output of the script
into
> syslog() calls (e.g., in a Perl5 script, or in a Java program using the
> Java DTrace interfaces).
> 
If you use the Java DTrace API, there is no output to parse since your
program gets the data from DTrace in the form of Java objects.
> And yes, you can have your D scripts general XML output if you like.
> 
All of the data classes in the Java DTrace API are both Serializable and
writable by java.beans.XMLEncoder, so generating XML is done for you.
> Keep in mind though that DTrace is robust in its effect on production
> systems: it will drop events if need be.  That also makes it not robust
> for auditing purposes.
> 
By default, a drop throws a ConsumerException which terminates your Java
program if unhandled. You can override the dataDropped() method to
behave differently if you like:

    consumer.addConsumerListener(new ConsumerAdapter() {
	public void dataDropped(DropEvent e) {
	    ... log data drops ...
	}
	... other methods
    });

The default behavior of terminating your program means you don''t have
to
worry about silently losing data. Ideally, you would use the Consumer
setOption() method to increase the size of your data buffers to avoid
drops.
> > > Is there a way to avoid DTrace Startup/Shutdown
> > > resource utilizations, if a DTrace script needs to be
> > > run several times?
> > 
> > Not sure about that one, I will let others answer.
> 
> Just keep the script running.  The only thing you can''t
dynamically add
> is PID provider probes for specific processes (AFAIK).  But then again
> you could just use the Java interface to DTrace.
In the Java DTrace API, a stopped Consumer cannot be restarted and a
closed Consumer cannot be reopened. However, as long as your DTrace
program is running, you can accumulate data in aggregations for as long
as you like, then clear() or trunc() them to start over.

Tom

On Fri, Aug 03, 2007 at 01:29:49AM -0700, Tom Erickson
wrote:
> On Thu, Aug 02, 2007 at 06:10:33PM -0500, Nicolas Williams wrote:
> > 
> > But it''d be a lot better to just parse the output of the
script into
> > syslog() calls (e.g., in a Perl5 script, or in a Java program using
the
> > Java DTrace interfaces).
> > 
> If you use the Java DTrace API, there is no output to parse since your
> program gets the data from DTrace in the form of Java objects.
Yes, I know, but I was also referring to output from dtrace(1).  Please
do forgive my inexact wording!  :)
> > And yes, you can have your D scripts general XML output if you like.
> > 
> All of the data classes in the Java DTrace API are both Serializable and
> writable by java.beans.XMLEncoder, so generating XML is done for you.
Here I specifically referred to D scripts run via dtrace(1).
> > Keep in mind though that DTrace is robust in its effect on production
> > systems: it will drop events if need be.  That also makes it not
robust
> > for auditing purposes.
> > 
> By default, a drop throws a ConsumerException which terminates your Java
> program if unhandled. You can override the dataDropped() method to
> behave differently if you like:
That does not make DTrace robust for auditing.  You find out about
drops, but that''s about it.
> > Just keep the script running.  The only thing you can''t
dynamically add
> > is PID provider probes for specific processes (AFAIK).  But then again
> > you could just use the Java interface to DTrace.
> 
> In the Java DTrace API, a stopped Consumer cannot be restarted and a
> closed Consumer cannot be reopened. However, as long as your DTrace
> program is running, you can accumulate data in aggregations for as long
> as you like, then clear() or trunc() them to start over.
I''d really like to be able to add PID providers probes dynamically to a
running D script.

Nico
--