dtrace discuss - Feb 2008 - Dynamic variable drops when using dtrace to monitor IO performance

Hi;

I''m trying to track I/O performance of an application accessing a solid
state storage device which should have sub 15us latencies...I was looking at
using iosnoop from the dtrace toolkit, as iostat is no use at this level of
granularity.

Unfortunately, the app is very I/O heavy and after a couple of seconds the I/O
snoop craps out, giving me thousands of dynamic variable drops.

Is there any way to fix this, or am I simply using iosnoop for something it
can''t handle?

rgds,

Russell


--
This message posted from opensolaris.org

Russ wrote:
> Hi;
> 
> I''m trying to track I/O performance of an application accessing a
solid state storage device which should have sub 15us latencies...I was looking
at using iosnoop from the dtrace toolkit, as iostat is no use at this level of
granularity.
> 
> Unfortunately, the app is very I/O heavy and after a couple of seconds the
I/O snoop craps out, giving me thousands of dynamic variable drops.
> 
> Is there any way to fix this, or am I simply using iosnoop for something it
can''t handle?
search for the error message you get in the discussions on opensolaris.org 
- I''ve seen a few suggestions there, but can''t remember them
now, sorry.

HTH
-- 
Michael Schuster
Recursion, n.: see ''Recursion''

1) Decrease the number of probe events that get recorded by being more 
restrictive in what you ask iosnoop to do

2) Increase the size of the DTrace switch buffer using a D pragma.  
You''ll have to modify iosnoop.

3) Increase how often the switch buffer switches using a D pragma.  
Again, you''ll have to modify iosnoop.

Chip

Russ wrote:
> Hi;
>
> I''m trying to track I/O performance of an application accessing a
solid state storage device which should have sub 15us latencies...I was looking
at using iosnoop from the dtrace toolkit, as iostat is no use at this level of
granularity.
>
> Unfortunately, the app is very I/O heavy and after a couple of seconds the
I/O snoop craps out, giving me thousands of dynamic variable drops.
>
> Is there any way to fix this, or am I simply using iosnoop for something it
can''t handle?
>
> rgds,
>
> Russell
>
>
> --
> This message posted from opensolaris.org
> _______________________________________________
> dtrace-discuss mailing list
> dtrace-discuss at opensolaris.org
>
>

Russell.Rimmer at ubs.com wrote:
> Chip;
>
> 1) Is hard as all I/O is actually from one process, to one disk, so
> little to filter on
> 2&3) I will try playing with bufsize and switchrate (is cleanrate
> relevant here?)
>
>   
Not from what I can see.

Chip

G''Day Russ,

On Tue, Feb 05, 2008 at 07:51:38AM -0800, Russ wrote:
> Hi;
> 
> I''m trying to track I/O performance of an application accessing a
solid state storage device which should have sub 15us latencies...I was looking
at using iosnoop from the dtrace toolkit, as iostat is no use at this level of
granularity.
> 
> Unfortunately, the app is very I/O heavy and after a couple of seconds the
I/O snoop craps out, giving me thousands of dynamic variable drops.
> 
> Is there any way to fix this, or am I simply using iosnoop for something it
can''t handle?
Sub 15us is starting to get pretty fast (until CPUs become faster, I
wouldn''t
trust measuring something sub 1us).  You can try tuning DTrace to avoid the
dynamic variable drops (dynvarsize is the tunable), however there might
be an easier way:

You said that you started with iostat?  Stats such as those that provide
"asvc_t" are stored in a higher resolution than iostat prints (stored
as
hrtime_t).  If they were the metrics of interest, then you can dig the
original values from kstat and produce high resolution versions of wsvc_t,
asvc_t, %b, etc.  We discussed iostat calculations, kstat, and scripting
with kstat in the "Solaris Performance and Tools" book.

Brendan

-- 
Brendan
[CA, USA]