zfs discuss - Jun 2007 - Re: [storage-discuss] Performance expectations of iscsi targets?

Paul,
> While testing iscsi targets exported from thumpers via 10GbE and  
> imported 10GbE on T2000s I am not seeing the throughput I expect,  
> and more importantly there is a tremendous amount of read IO  
> happending on a purely sequential write workload. (Note all systems  
> have Sun 10GbE cards and are running Nevada b65.)
The read IO activity you are seeing is a direct result of re-writes  
on the ZFS storage pool. If you were to recreate the test from  
scratch, you would notice that on the very first pass of write I/Os  
from ''dd'', there would be no reads. This is an artifact of
using
zvols as backing store for iSCSI Targets.

The iSCSI Target software supports raw SCSI disks, Solaris raw  
devices (/dev/rdsk/....), Solaris block devices (/dev/dsk/...),  
zvols, SVM volumes, files in file systems, including temps.

>
> Simple write workload (from T2000):
>
> # time dd if=/dev/zero of=/dev/rdsk/ 
> c6t010000144F210ECC00002A004675E957d0 \
>   bs=64k count=1000000
A couple of things, maybe missing here, or the commands are not true  
cut-n-paste of what is being tested.

1). From the iSCSI initiator, there is no device at /dev/rdsk/ 
c6t010000144F210ECC00002A004675E957d0, note the missing slice. (s0,  
s1, s2, etc....).

2). Even if one was to specify a slice, as in /dev/rdsk/ 
c6t010000144F210ECC00002A004675E957d0s2, it is unlikely that the LUN  
has been formatted. When I run format the first time, I get the error  
message of "Please run fdisk first".

Of course this does not have to be the case, because if the ZFS  
storage pool that backed up this LUN had previously been formatted  
with either a Solaris VTOC or Intel EFI label, then the disk would  
show up correctly.
>
> Performance of iscsi target pool on new blocks:
>
> bash-3.00# zpool iostat thumper1-vdev0 1
> thumper1-vdev0  17.4G  2.70T      0    526      0  63.6M
> thumper1-vdev0  17.5G  2.70T      0    564      0  60.5M
> thumper1-vdev0  17.5G  2.70T      0      0      0      0
> thumper1-vdev0  17.5G  2.70T      0      0      0      0
> thumper1-vdev0  17.5G  2.70T      0      0      0      0
>
> Configuration of zpool/iscsi target:
>
> # zpool status thumper1-vdev0
>   pool: thumper1-vdev0
>  state: ONLINE
>  scrub: none requested
> config:
>
>         NAME        STATE     READ WRITE CKSUM
>         thumper1-vdev0  ONLINE       0     0     0
>           c0t7d0    ONLINE       0     0     0
>           c1t7d0    ONLINE       0     0     0
>           c5t7d0    ONLINE       0     0     0
>           c6t7d0    ONLINE       0     0     0
>           c7t7d0    ONLINE       0     0     0
>           c8t7d0    ONLINE       0     0     0
>
> errors: No known data errors
>
> The first thing is that for this pool I was expecting 200-300MB/s  
> throughput, since it is a simple stripe across 6, 500G disks.  In  
> fact, a direct local workload (directly on thumper1) of the same  
> type confirms what I expected:
>
> bash-3.00# dd if=/dev/zero of=/dev/zvol/rdsk/thumper1-vdev0/iscsi  
> bs=64k count=1000000 &
>
> bash-3.00# zpool iostat thumper1-vdev0 1
> thumper1-vdev0  20.4G  2.70T      0  2.71K      0   335M
> thumper1-vdev0  20.4G  2.70T      0  2.92K      0   374M
> thumper1-vdev0  20.4G  2.70T      0  2.88K      0   368M
> thumper1-vdev0  20.4G  2.70T      0  2.84K      0   363M
> thumper1-vdev0  20.4G  2.70T      0  2.57K      0   327M
>
> The second thing, is that when overwriting already written blocks  
> via the iscsi target (from the T2000) I see a lot of read bandwidth  
> for blocks that are being completely overwritten.  This does not  
> seem to slow down the write performance, but 1) it is not seem in  
> the direct case; and 2) it consumes channel bandwidth unnecessarily.
>
> bash-3.00# zpool iostat thumper1-vdev0 1
> thumper1-vdev0  8.90G  2.71T    279    783  31.7M  95.9M
> thumper1-vdev0  8.90G  2.71T    281    318  31.7M  29.1M
> thumper1-vdev0  8.90G  2.71T    139      0  15.8M      0
> thumper1-vdev0  8.90G  2.71T    279      0  31.7M      0
> thumper1-vdev0  8.90G  2.71T    139      0  15.8M      0
>
> Can anyone help to explain what I am seeing, or give me some  
> guidance on diagnosing the cause of the following:
> - The bottleneck in accessing the iscsi target from the T2000
 From the iSCSI Initiator''s point of view, there are various  
(Negotiated) Login Parameters, which may have a direct effect on  
performance. Take a look at "iscsiadm list target --verbose", then  
consult the iSCSI man pages, or documentation online at docs.sun.com.

Remember to keep track of what you change on a per-target basis, and  
only change one parameter at a time, and measure your results.
> - The cause of the extra read bandwidth when overwriting blocks on  
> the iscsi target from the T2000.
ZFS as the backing store, and it COW (Copy-on-write) in maintaining  
the ZFS zvols within the storage pool.

>
>
> Any help is much appreciated,
> paul
> _______________________________________________
> storage-discuss mailing list
> storage-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/storage-discuss
>
>
Jim Dunham
Solaris, Storage Software Group

Sun Microsystems, Inc.
1617 Southwood Drive
Nashua, NH 03063
Email: James.Dunham at Sun.COM
http://blogs.sun.com/avs



-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20070619/22f08e1e/attachment.html>