Hi,
Can DTrace assist me in finding out the disk access latencies for various
read write requests given by various processes. I am basically interested with
rotational latency.
--
Libin Varghese
This message posted from opensolaris.org
Libin Varghese wrote:> Hi, Can DTrace assist me in finding out the disk access latencies for > various read write requests given by various processes. I am > basically interested with rotational latency.Hi Libin, I am fairly certain that the answer to your question is no. What I am more interested in is why you think you might get an answer which even vaguely approaches physical reality, given that kernel caching and IO algorithms, and let alone the caching and other magic that disk drive manufacturers implement which you cannot turn off. best regards, James C. McPherson -- Solaris Datapath Engineering Data Management Group Sun Microsystems
James C. McPherson wrote:> Libin Varghese wrote: >> Hi, Can DTrace assist me in finding out the disk access latencies for >> various read write requests given by various processes. I am >> basically interested with rotational latency. > > Hi Libin, > I am fairly certain that the answer to your question is no. > > What I am more interested in is why you think you might get > an answer which even vaguely approaches physical reality, > given that kernel caching and IO algorithms, and let alone > the caching and other magic that disk drive manufacturers > implement which you cannot turn off. >Outside of the seek and rotational latencies, which as James states aren''t as big of a deal as they used to, we do have some tools inside of Sun that measure various i/o statistics at the process level. However, they aren''t publicly available due to...well...I''m not sure. Might be legal or such things but if you contact a knowledgeable Sun sales engineer you should be able to get access to said tools. (Or if they don''t know point them at me and I''ll buy them a clue.)
G''Day, On Sun, 29 Jan 2006, Libin Varghese wrote:> Hi, > Can DTrace assist me in finding out the disk access latencies for > various read write requests given by various processes. I am basically > interested with rotational latency.With various mindbending assumptions you may estimate what is going on. The disk doesn''t send signals to say "I have began rotating for this request; I have finished rotating for this request". We see requests (strategys) and completions (biodones) from the block I/O driver. Understanding these can be quite challenging - the disk internally may map sector and cylinder values to it''s own optimised layout - you can''t trust what the disk interface tells you. I think at best you''ll get a good estimate for access latency, and a vague estimate for rotational delay. See "Disk Utilisation by Process" for a discussion on using DTrace to analyse disk behaviour, http://www.brendangregg.com/Perf/solarismetrics.html ... Why is rotational latency interesting? Maybe you looking to buy faster disks?. If so, writing some test C programs may help you analyse latency issues. (and there is a utility called "filebench" over in the Performance community that may help you out). Often sites use storage arrays that have large front-end caches, eliminating rotational latency for most requests. hope this helps, Brendan [Sydney, Australia]
On Sun, 29 Jan 2006, James C. McPherson wrote:> Libin Varghese wrote: > > Hi, Can DTrace assist me in finding out the disk access latencies for > > various read write requests given by various processes. I am > > basically interested with rotational latency. > > Hi Libin, > I am fairly certain that the answer to your question is no. > > What I am more interested in is why you think you might get > an answer which even vaguely approaches physical reality, > given that kernel caching and IO algorithms, and let alone > the caching and other magic that disk drive manufacturers > implement which you cannot turn off.Yep - "caching and other magic" sums it up. :) It''s possible to work with it and get some useful metrics out, so long as you keep tabs on all the assumptions you''ve made... Brendan
G''Day Torrey, On Sun, 29 Jan 2006, Torrey McMahon wrote:> James C. McPherson wrote: > > Libin Varghese wrote: > >> Hi, Can DTrace assist me in finding out the disk access latencies for > >> various read write requests given by various processes. I am > >> basically interested with rotational latency. > > > > Hi Libin, > > I am fairly certain that the answer to your question is no. > > > > What I am more interested in is why you think you might get > > an answer which even vaguely approaches physical reality, > > given that kernel caching and IO algorithms, and let alone > > the caching and other magic that disk drive manufacturers > > implement which you cannot turn off. > > > Outside of the seek and rotational latencies, which as James states > aren''t as big of a deal as they used to, we do have some tools inside of > Sun that measure various i/o statistics at the process level. However, > they aren''t publicly available due to...well...I''m not sure. Might be > legal or such things but if you contact a knowledgeable Sun sales > engineer you should be able to get access to said tools. (Or if they > don''t know point them at me and I''ll buy them a clue.)Other tools? Please forward along any info you have - I''d rather not reinvent tools that already live within Sun somewhere. And while such tools aren''t public, people can always analyse I/O behaviour using the disk tools in the DTraceToolkit, http://www.opensolaris.org/os/community/dtrace/dtracetoolkit In particular, iosnoop snoop I/O events as they occur iotop display top disk I/O events by process Disk/bitesize.d print disk event size report diskhits disk access by file offset hotspot.d print disk event by location iofile.d I/O wait time by filename and process iopattern print disk I/O pattern iopending plot number of pending disk events pathopens.d pathnames successfully opened count seeksize.d print disk seek size report cheers, Brendan
Brendan Gregg wrote:>G''Day, > >On Sun, 29 Jan 2006, Libin Varghese wrote: > > > >>Hi, >> Can DTrace assist me in finding out the disk access latencies for >>various read write requests given by various processes. I am basically >>interested with rotational latency. >> >> > >With various mindbending assumptions you may estimate what is going on. > >The disk doesn''t send signals to say "I have began rotating for this >request; I have finished rotating for this request". We see requests >(strategys) and completions (biodones) from the block I/O driver. > >Understanding these can be quite challenging - the disk internally may map >sector and cylinder values to it''s own optimised layout - you can''t trust >what the disk interface tells you. I think at best you''ll get a good >estimate for access latency, and a vague estimate for rotational delay. > >See "Disk Utilisation by Process" for a discussion on using DTrace to >analyse disk behaviour, > > http://www.brendangregg.com/Perf/solarismetrics.html > > >... > >Why is rotational latency interesting? Maybe you looking to buy faster >disks?. If so, writing some test C programs may help you analyse latency >issues. (and there is a utility called "filebench" over in the Performance >community that may help you out). >James C. McPherson wrote:>What I am more interested in is why you think you might get >an answer which even vaguely approaches physical reality, >given that kernel caching and IO algorithms, and let alone >the caching and other magic that disk drive manufacturers >implement which you cannot turn off. >I am totally new to DTrace, actually someone guided me to DTrace with the information that you can track page fault of individual processes. Impressed by this, thought it could also serve my purpose. I am on a process to understand I/O scheduling, and in collecting some statistics as how various I/O scheduling techniques try to reduce seek latency and rotational latency of disks. Any idea as to how I can track these latencies. -- Libin Varghese -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20060129/e290b76f/attachment.html>
Brendan Gregg wrote:> http://www.opensolaris.org/os/community/dtrace/dtracetoolkit > >In particular, > > iosnoop snoop I/O events as they occur > iotop display top disk I/O events by process > Disk/bitesize.d print disk event size report > diskhits disk access by file offset > hotspot.d print disk event by location > iofile.d I/O wait time by filename and process > iopattern print disk I/O pattern > iopending plot number of pending disk events > pathopens.d pathnames successfully opened count > seeksize.d print disk seek size report > >Some of these tools will surely help me, in my study. Though not the ultimate goal of how various I/O scheduling algo''s reduce seek and rotational latencies. -- Libin Varghese -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20060129/0e724e87/attachment.html>
Libin Varghese wrote: ...> I am totally new to DTrace, actually someone guided me to DTrace with > the information that you can track page fault of individual processes.You can, it''s really really neat that one can do this.> Impressed by this, thought it could also serve my purpose. I am on a > process to understand I/O scheduling, and in collecting some statistics > as how various I/O scheduling techniques try to reduce seek latency and > rotational latency of disks. Any idea as to how I can track these latencies.I really doubt that you''ll be able to reduce the rotational latency of a disk -- there''s this slight problem with the laws of physics. Have you read the literature on IO scheduling algorithms? If you haven''t, please do so before you continue with your DTrace investigations. There are some handy references reported in this gooooooogle search: http://www.google.com.au/search?q=io+scheduling+algorithm+cite If, otoh, you are already up to date on the literature, are there particular techniques that you are interested in? regrads, James C. McPherson -- Solaris Datapath Engineering Data Management Group Sun Microsystems
James C. McPherson wrote:> >> Impressed by this, thought it could also serve my purpose. I am on a >> process to understand I/O scheduling, and in collecting some >> statistics as how various I/O scheduling techniques try to reduce >> seek latency and rotational latency of disks. Any idea as to how I >> can track these latencies. > > > I really doubt that you''ll be able to reduce the rotational > latency of a disk -- there''s this slight problem with the > laws of physics. > > Have you read the literature on IO scheduling algorithms? > If you haven''t, please do so before you continue with your > DTrace investigations. There are some handy references reported > in this gooooooogle search: > > http://www.google.com.au/search?q=io+scheduling+algorithm+cite > > > If, otoh, you are already up to date on the literature, are > there particular techniques that you are interested in?Yes, reduction in rotational latency is a difficult to achieve :) , but there are few algo''s that may reduce it such as Shortest Latency Time First, Shortest Access Time First take care of both seek and rotational latencies. - Libin Varghese
Libin Varghese wrote:> > Yes, reduction in rotational latency is a difficult to achieve :) , > but there are few algo''s that may reduce it such as Shortest Latency > Time First, Shortest Access Time First take care of both seek and > rotational latencies.www.hpl.hp.com/research/ssp/papers/HPL-CSP-91-7rev1.pdf -- Libin Varghese -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/dtrace-discuss/attachments/20060130/4db0af7a/attachment.html>
Brendan Gregg wrote:> > > Other tools? Please forward along any info you have - I''d rather not > reinvent tools that already live within Sun somewhere. >We have some internal tools that look at TNF data. Any of the storage related sales folks should know about it. (It''s called "SWAT".)> And while such tools aren''t public, people can always analyse I/O > behaviour using the disk tools in the DTraceToolkit, > > http://www.opensolaris.org/os/community/dtrace/dtracetoolkit > > In particular, > > iosnoop snoop I/O events as they occur > iotop display top disk I/O events by process > Disk/bitesize.d print disk event size report > diskhits disk access by file offset > hotspot.d print disk event by location > iofile.d I/O wait time by filename and process > iopattern print disk I/O pattern > iopending plot number of pending disk events > pathopens.d pathnames successfully opened count > seeksize.d print disk seek size report > >I believe the author of SWAT, who I''ve cc''d, has been looking at dtrace for awhile now and still has an open RFE or two...but I''ll let him expand on that if he so wishes. -- Torrey McMahon Sun Microsystems Inc.