CentOS - Jun 2014 - iostat results for multi path disks

Here is a sample of running iostat on a server that has a LUN from a SAN with
multiple paths.  I am specifying a device list that just grabs the bits related
to the multi path device:

$ iostat -dxkt 1 2 sdf sdg sdh sdi dm-7 dm-8 dm-9
Linux 2.6.18-371.8.1.el5 (db21b.den.sans.org) 	06/20/2014

Time: 02:30:23 PM
Device:         rrqm/s   wrqm/s   r/s   w/s    rkB/s    wkB/s avgrq-sz avgqu-sz 
await  svctm  %util
sdf               0.66    52.32  3.57 34.54   188.38   347.52    28.13     0.14 
3.62   0.87   3.31
sdg               0.66    52.29  3.57 34.56   189.79   347.48    28.18     0.14 
3.72   0.87   3.32
sdh               0.00     0.00  0.00  0.00     0.00     0.00    14.19     0.00 
2.90   2.90   0.00
sdi               0.00     0.00  0.00  0.00     0.00     0.00    14.19     0.00 
2.87   2.87   0.00
dm-7              0.00     0.00  8.46 173.75   378.17   695.00    11.78     3.41
18.68   0.35   6.46
dm-8              0.00     0.00  8.46 173.75   378.17   695.00    11.78     3.41
18.68   0.36   6.47
dm-9              0.00     0.00  8.46 173.75   378.17   695.00    11.78     3.41
18.68   0.36   6.48

Time: 02:30:24 PM
Device:         rrqm/s   wrqm/s   r/s   w/s    rkB/s    wkB/s avgrq-sz avgqu-sz 
await  svctm  %util
sdf               0.00    54.00  7.00 48.00    88.00   408.00    18.04     0.12 
2.11   1.20   6.60
sdg               0.00    13.00  1.00 26.00     4.00   156.00    11.85     0.01 
0.52   0.48   1.30
sdh               0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00 
0.00   0.00   0.00
sdi               0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00 
0.00   0.00   0.00
dm-7              0.00     0.00  8.00 141.00    92.00   564.00     8.81     0.25
1.69   0.53   7.90
dm-8              0.00     0.00  8.00 141.00    92.00   564.00     8.81     0.25
1.70   0.54   8.00
dm-9              0.00     0.00  8.00 141.00    92.00   564.00     8.81     0.25
1.70   0.54   8.00


sdf,sdg,sdh,sdi - four paths for LUN (sdf and sdg are the active paths)
dm-7 - device-mapper pseudo device for the mpath device
dm-8 - device-mapper pseudo-device for the partition spanning the entire mpath
device
dm-9 - device-mapper pseudo-device for the LVM LV created on the mpath device

The first sample from iostat is the historical data so lets ignore it.  The
second sample are the stats for a 1 second interval.

I see the stats for sdf and sdg are roughly equal but they differ ? they are the
two active paths that are both being used.

I see the stats for dm-7, dm-8 and dm-9 are almost completely identical - makes
sense as they really represent the same ?disk?.

What confuses me is the fact that all the stats for sdf/sdg don?t add up to be
equivalent to the dm-[789] devices

For the rkB/s and wkB/s columns, the numbers for sdf and sdg add up to equal the
numbers for dm-9.

But for the first four columns:

Column	sdf/sdg			dm-9
======	=============	===rrqm/s	0.0 + 0.0 = 0.0		0		
wrqm/s	54.0 +13.0 = 67.0	0		Very different
r/s		7.0 + 1.0 = 8.0		8.0
w/s		48.0 + 26.0 = 74.0	141.0	Very different

So read data matches and write data diverges


Which numbers should I go with?  The physical devices or the logical device?

Thanks

David Goldsmith
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 204 bytes
Desc: Message signed with OpenPGP using GPGMail
URL:
<http://lists.centos.org/pipermail/centos/attachments/20140620/592906ce/attachment.sig>

On Jun 20, 2014, at 9:47 AM, David Goldsmith <dgoldsmith at sans.org>
wrote:
> Here is a sample of running iostat on a server that has a LUN from a SAN
with multiple paths.  I am specifying a device list that just grabs the bits
related to the multi path device:
> 
> $ iostat -dxkt 1 2 sdf sdg sdh sdi dm-7 dm-8 dm-9
[?]
> For the rkB/s and wkB/s columns, the numbers for sdf and sdg add up to
equal the numbers for dm-9.
> 
> But for the first four columns:
> 
> Column	sdf/sdg			dm-9
> ======	=============	===> rrqm/s	0.0 + 0.0 = 0.0		0		
> wrqm/s	54.0 +13.0 = 67.0	0		Very different
> r/s		7.0 + 1.0 = 8.0		8.0
> w/s		48.0 + 26.0 = 74.0	141.0	Very different
> 
> So read data matches and write data diverges
> 
> 
> Which numbers should I go with?  The physical devices or the logical
device?

I may be wrong but I believe the difference you are seeing is due to write
coalescing happening between the DM layer and the underlying SCSI layer reducing
the number of write requests even though the same amount of data is read/written

? 
Mark Tinberg
mtinberg at wisc.edu