[dtrace-discuss] Guidelines for Long Running DTrace Scripts

Robert Milkowski
Mon, 16 Mar 2009 23:36:02 -0000

Hello Ben,

Thursday, March 12, 2009, 5:04:40 AM, you wrote:

BR> There are several forms of data collection that I simply can gather in
BR> no other way than to use DTrace, and am therefore considering
BR> implementing several perpetual scripts, potentially SMF controlled.

BR> To date I''ve been reluctant to just leave scripts running for
very long
BR> periods of time (hours, days, weeks)... but I''m not entirely
sure why.

BR> Are there any general guidelines or best practice for doing so?
BR> Possibly experiences from others who have done this?

Just some thoughts, probably obvious:

 - make sure you discard (assign 0) all variables no longer used
 otherwise your script will probably be memory leaking; it''s common to
 not to think about it for one liners or short lived scripts but could
 be an issue for long time running ones

 - you probably want to monitor if any drops are happening

 - using a cyclic buffer could be useful

 - some of the dtrace buffers probably will need tuning - best to observe a
 script for some time and look for errors from dtrace

 - try to avoid any string comparisons in predicates (well not
 necessarily linked to long running scripts...) and only use direct
 comparisons without need to dereferencing pointers, etc

 - depends on what you are monitoring dtrace -Z could be useful as
 your application might restart, etc.

 - be very precise at probe definitions to monitor only what you
 really need otherwise a non-typical app/os behavior could induce big
 overhead from dtrace

 - observe allocated and resident memory sets of dtrace process for
 each script if they are not growing above acceptable levels

 - monitor any other errors coming from dtrace

 - libdtrace could be useful? (don''t know, just a thought)

 - "system is unresponsive" could be tuned IIRC and maybe you want it
 to be more aggressive so dtrace scripts exit quicker than default (so
 minimizing bad impact on a system)

 - dtrace speculations could be very useful to minimize amount of
 output and focus only on interesting ones

 - use walltimestamp and/or timestamp provided by dtrace on multi-cpu servers
instead
 on relaying on syslog or anything else external to dtrace - if real
 order of events is required to be know everything else will fail
 sooner or later (again, not necessarily related only to long running
 scripts)

 - remember dtrace can drop events by design and you neeed to take it
 into account - it is not an auditing framework afterall

 - when average numbers are good enough profile-N could be much more
 lightweight than measuring every event but you can miss some
 potentially interesting data...

 - never forget then there are other tools than dtrace and sometimes
 it is easier (better) to achieve something by using them than dtrace



--
Best regards,
 Robert Milkowski