CentOS - Dec 2009 - measuring iops on linux - numbers make sense?

Hello,

When approaching hosting providers for services, the first question
many of them asked us was about the amount of IOPS the disk system
should support.

While we stress-tested our service, we recorded between 4000 and 6000
"merged io operations per second" as seen in "iostat -x" and
collectd
(varies between the different components of the system, we have a few
such servers).

A couple of hosting providers told us that this (iostat and collectd
"merged operations per second") is a not so bad way to get IOPS.

A partner of ours doubts that this is possible with the current
hardware - a 3ware 9690SA-4I4E
(http://www.3ware.com/products/sas-9690SA.asp) with 512Mb battery
backed up cache and 8 SAS 15k rpm disks (SEAGATE ST3300656SS) in RAID
1+0. They calculate 750 IOPS per spindle and say that the maximum they
ever saw from any 15k disk was 350 iops on RAID 0.

Am I measuring the numbers correctly? Is there a better way to measure
IOPS on CentOS?

The OS is CentOS 5.3 x86_64, the rest of the hardware is 64Gb RAM, 2
quad-core 3GHz Intel Xeon CPU's.

Thanks,

--Amos

On Sat, Dec 05, 2009 at 08:57:55AM +1100, Amos Shapira
wrote:
> Hello,
> 
> When approaching hosting providers for services, the first question
> many of them asked us was about the amount of IOPS the disk system
> should support.
> 
> While we stress-tested our service, we recorded between 4000 and 6000
> "merged io operations per second" as seen in "iostat
-x" and collectd
> (varies between the different components of the system, we have a few
> such servers).
> 
> A couple of hosting providers told us that this (iostat and collectd
> "merged operations per second") is a not so bad way to get IOPS.
> 
> A partner of ours doubts that this is possible with the current
> hardware - a 3ware 9690SA-4I4E
> (http://www.3ware.com/products/sas-9690SA.asp) with 512Mb battery
> backed up cache and 8 SAS 15k rpm disks (SEAGATE ST3300656SS) in RAID
> 1+0. They calculate 750 IOPS per spindle and say that the maximum they
> ever saw from any 15k disk was 350 iops on RAID 0.
> 
> Am I measuring the numbers correctly? Is there a better way to measure
> IOPS on CentOS?
> 
> The OS is CentOS 5.3 x86_64, the rest of the hardware is 64Gb RAM, 2
> quad-core 3GHz Intel Xeon CPU's.
I typically would look at the tps numbers (either in iostat or sar -b).
You might also be able to use blktrace output (combined with
seekwatcher) to generate IOPS data.

Someone out there may have some better suggestions.

Ray

Amos Shapira wrote:
> Hello,
>
> When approaching hosting providers for services, the first question
> many of them asked us was about the amount of IOPS the disk system
> should support.
Really depends on what level your measuring IOPS. Most frequently
IOPS means disk IOPS not cache IOPS.

Your typical enterprise SATA disk is built to handle roughly 90
IOPS, your typical enterprise 10k SAS/FC disk 180 IOPS, your
typical enterprise 15k SAS/FC disk 250 IOPS.

The actual number varies depending on the I/O size, for example
if your workload(or controller) sends large I/Os to the disk the
amount of IOPS the disk will do will drop(I've seen SATA disks
on my enterprise storage system peg out at 35 IOPS for example)

Amount of IOPS also is greatly influenced by RAID type and
% of read/write workload.

If your application is pretty simple then go build a system
config that can run it, and take the raw IOPS of the disks
and send it to the hosting provider.

Myself I measure IOPS on my storage array, I trust the numbers
there much more than the OS. I can get per volume, per disk,
and per-host statistics.

nate