Hi all, I''d like to ask whether there is a way to monitor disk seeks. I have an application where many concurrent readers (>50) sequentially read a large dataset (>10T) at a fairly low speed (8-10 Mbit/s). I can monitor read/write ops using iostat, but that doesn''t tell me how contiguous the data is, i.e. when iostat reports "500" read ops, does that translate to 500 seeks + 1 read per seek, or 50 seeks + 10 reads, etc? Thanks! Regards, -- Saso
I am not sure you can monitor actual mechanical seeks short of debugging and interrogating the HDD firmware - because it is the last responsible logic in the chain of caching, queuing and issuing actual commands to the disk heads. For example, a long logical IO spanning several cylinders would probably be seen by the OS as one IO with many MBs of transfer, but would break down into track-to-track seeks in disk''s reality. Nevertheless, as far as the OS commands are concerned, you can try to interpolate outputs of iostat and iopattern: http://www.richardelling.com/Home/scripts-and-programs-1/iopattern The latter tries to estimate the amounts of SEQuential and RNDom reads and writes in your workload. HTH, //Jim 2011-05-19 16:35, Sa?o Kiselkov ?????:> Hi all, > > I''d like to ask whether there is a way to monitor disk seeks. I have an > application where many concurrent readers (>50) sequentially read a > large dataset (>10T) at a fairly low speed (8-10 Mbit/s). I can monitor > read/write ops using iostat, but that doesn''t tell me how contiguous the > data is, i.e. when iostat reports "500" read ops, does that translate to > 500 seeks + 1 read per seek, or 50 seeks + 10 reads, etc? Thanks! > > Regards, > -- > Saso > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss-- +============================================================+ | | | ?????? ???????, Jim Klimov | | ??????????? ???????? CTO | | ??? "??? ? ??" JSC COS&HT | | | | +7-903-7705859 (cellular) mailto:jimklimov at cos.ru | | CC:admin at cos.ru,jimklimov at mail.ru | +============================================================+ | () ascii ribbon campaign - against html mail | | /\ - against microsoft attachments | +============================================================+
2011-05-19 17:00, Jim Klimov ?????:> I am not sure you can monitor actual mechanical seeks short > of debugging and interrogating the HDD firmware - because > it is the last responsible logic in the chain of caching, > queuing and issuing actual commands to the disk heads. > > For example, a long logical IO spanning several cylinders > would probably be seen by the OS as one IO with many MBs > of transfer, but would break down into track-to-track seeks > in disk''s reality. >> i.e. when iostat reports "500" read ops, does that translate to >> 500 seeks + 1 read per seek, or 50 seeks + 10 reads, etc? Thanks!First I wanted to assume that this would map to "at least 500 seeks and 1 read per seek". But... Actually, I am not sure there is any one good translation of logical IOPS as seen by ZFS, storage driver or iostat, into mechanical IOPS. Logical IOs issued by ZFS may be grouped in one mechanical IO by the HDD''s firmware, if these IOs happen to be backed by blocks on the same track (and they are issued in a rapid enough succession as to be natively/taggedly queued as one IO). I wrote of an opposite situation above. -- +============================================================+ | | | ?????? ???????, Jim Klimov | | ??????????? ???????? CTO | | ??? "??? ? ??" JSC COS&HT | | | | +7-903-7705859 (cellular) mailto:jimklimov at cos.ru | | CC:admin at cos.ru,jimklimov at mail.ru | +============================================================+ | () ascii ribbon campaign - against html mail | | /\ - against microsoft attachments | +============================================================+
On 19 May, 2011 - Sa??o Kiselkov sent me these 0,6K bytes:> Hi all, > > I''d like to ask whether there is a way to monitor disk seeks. I have an > application where many concurrent readers (>50) sequentially read a > large dataset (>10T) at a fairly low speed (8-10 Mbit/s). I can monitor > read/write ops using iostat, but that doesn''t tell me how contiguous the > data is, i.e. when iostat reports "500" read ops, does that translate to > 500 seeks + 1 read per seek, or 50 seeks + 10 reads, etc? Thanks!Get DTraceToolkit and check out the various things under Disk and FS, might help. /Tomas -- Tomas ?gren, stric at acc.umu.se, http://www.acc.umu.se/~stric/ |- Student at Computing Science, University of Ume? `- Sysadmin at {cs,acc}.umu.se
On 05/19/2011 03:35 PM, Tomas ?gren wrote:> On 19 May, 2011 - Sa??o Kiselkov sent me these 0,6K bytes: > >> Hi all, >> >> I''d like to ask whether there is a way to monitor disk seeks. I have an >> application where many concurrent readers (>50) sequentially read a >> large dataset (>10T) at a fairly low speed (8-10 Mbit/s). I can monitor >> read/write ops using iostat, but that doesn''t tell me how contiguous the >> data is, i.e. when iostat reports "500" read ops, does that translate to >> 500 seeks + 1 read per seek, or 50 seeks + 10 reads, etc? Thanks! > > Get DTraceToolkit and check out the various things under Disk and FS, > might help. > > /TomasThank you all for the tips, I''ll try to poke around using the DTrackToolkit. Regards, -- Saso
On Thu, May 19, 2011 at 5:35 AM, Sa?o Kiselkov <skiselkov.ml at gmail.com> wrote:> I''d like to ask whether there is a way to monitor disk seeks. I have an > application where many concurrent readers (>50) sequentially read a > large dataset (>10T) at a fairly low speed (8-10 Mbit/s). I can monitor > read/write ops using iostat, but that doesn''t tell me how contiguous the > data is, i.e. when iostat reports "500" read ops, does that translate to > 500 seeks + 1 read per seek, or 50 seeks + 10 reads, etc? Thanks!You can sort of do this with a DTrace script. Something like: (forgive my crappy script, I''ve only poked at DTrace a few times) #pragma D option quiet io:::done / args[1]->dev_name == "sd" && args[1]->dev_instance < 11 / { printf("%d.%03d,%s,%i,%s,%i\n", (timestamp/1000000), (timestamp / 1000) % 1000, args[1]->dev_statname, args[0]->b_lblkno, (args[0]->b_flags & B_WRITE ? "W" : "R"), args[0]->b_bcount ); } For every completed IO, this should give you the timestamp, device name, start LBA, "R"ead or "W"rite and length of the IO. -B -- Brandon High : bhigh at freaks.com
On May 19, 2011, at 5:35 AM, Sa?o Kiselkov wrote:> Hi all, > > I''d like to ask whether there is a way to monitor disk seeks. I have an > application where many concurrent readers (>50) sequentially read a > large dataset (>10T) at a fairly low speed (8-10 Mbit/s). I can monitor > read/write ops using iostat, but that doesn''t tell me how contiguous the > data is, i.e. when iostat reports "500" read ops, does that translate to > 500 seeks + 1 read per seek, or 50 seeks + 10 reads, etc? Thanks!In general, this is hard to see from the OS. In Solaris, the default I/O flowing through sd gets sorted based on LBA before being sent to the disk. If the disks gets more than 1 concurrent I/O request (10 is the default for Solaris-based ZFS) then the disk can resort or otherwise try to optimize the media accesses. As others have mentioned, iopattern is useful for looking a sequential patterns. I''ve made some adjustments for the version at http://www.richardelling.com/Home/scripts-and-programs-1/iopattern You can see low-level SCSI activity using scsi.d, but I usually uplevel that to using "iosnoop -Dast" which shows each I/O and its response time. Note that the I/Os can complete out-of-order on many devices. The only device I know that is so fast and elegant that it always completes in-order is the DDRdrive. For detailed analysis of iosnoop data, you will appreciate a real statistics package. I use JMP, but others have good luck with R. -- richard -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20110519/13d86812/attachment-0001.html>
On 05/19/2011 07:47 PM, Richard Elling wrote:> On May 19, 2011, at 5:35 AM, Sa?o Kiselkov wrote: > >> Hi all, >> >> I''d like to ask whether there is a way to monitor disk seeks. I have an >> application where many concurrent readers (>50) sequentially read a >> large dataset (>10T) at a fairly low speed (8-10 Mbit/s). I can monitor >> read/write ops using iostat, but that doesn''t tell me how contiguous the >> data is, i.e. when iostat reports "500" read ops, does that translate to >> 500 seeks + 1 read per seek, or 50 seeks + 10 reads, etc? Thanks! > > In general, this is hard to see from the OS. In Solaris, the default I/O > flowing through sd gets sorted based on LBA before being sent to the > disk. If the disks gets more than 1 concurrent I/O request (10 is the default > for Solaris-based ZFS) then the disk can resort or otherwise try to optimize > the media accesses. > > As others have mentioned, iopattern is useful for looking a sequential > patterns. I''ve made some adjustments for the version at > http://www.richardelling.com/Home/scripts-and-programs-1/iopattern > > You can see low-level SCSI activity using scsi.d, but I usually uplevel that > to using "iosnoop -Dast" which shows each I/O and its response time. > Note that the I/Os can complete out-of-order on many devices. The only > device I know that is so fast and elegant that it always completes in-order > is the DDRdrive. > > For detailed analysis of iosnoop data, you will appreciate a real statistics > package. I use JMP, but others have good luck with R. > -- richardThank you, the iopattern script seems to be quite close to what I wanted. The percentage split between random and sequential I/O is pretty much what I needed to know. Regards, -- Saso
Hi, see the seeksize script on this URL: http://prefetch.net/articles/solaris.dtracetopten.html Not used it but looks neat! cheers Andy.
On 05/24/2011 03:08 PM, a.smith at ukgrid.net wrote:> Hi, > > see the seeksize script on this URL: > > http://prefetch.net/articles/solaris.dtracetopten.html > > Not used it but looks neat! > > cheers Andy.I already did and it does the job just fine. Thank you for your kind suggestion. BR, -- Saso