Brian H. Nelson
2009-Jan-06 18:36 UTC
[zfs-discuss] Performance issue with zfs send of a zvol
I noticed this issue yesterday when I first started playing around with
zfs send/recv. This is on Solaris 10U6.
It seems that a zfs send of a zvol issues ''volblocksize'' reads
to the
physical devices. This doesn''t make any sense to me, as zfs generally
consolidates read/write requests to improve performance. Even the dd
case with the same snapshot does not exhibit this behavior. It seems to
be specific to zfs send.
I checked with 8k, 64k, and 128k volblocksize, and the reads generated
by zfs send always seem to follow that size, while the reads with dd do not.
The small reads seems to hurt performance of zfs send. I tested with a
mirror, but on another machine with a 7 disk raidz, the performance is
MUCH worse because the 8k reads get broken up into even smaller reads
and spread across the raidz.
Is this a bug, or can someone explain why this is happening?
Thanks
-Brian
Using 8k volblocksize:
-bash-3.00# zfs send pool1/vol8k at now > /dev/null
capacity operations bandwidth
pool used avail read write read write
----------- ----- ----- ----- ----- ----- -----
pool1 4.01G 274G 1.88K 0 15.0M 0
mirror 4.01G 274G 1.88K 0 15.0M 0
c0t9d0 - - 961 0 7.46M 0
c0t11d0 - - 968 0 7.53M 0
----------- ----- ----- ----- ----- ----- -----
== ~8k reads to pool and drives
-bash-3.00# dd if=/dev/zvol/dsk/pool1/vol8k at now of=/dev/null bs=8k
capacity operations bandwidth
pool used avail read write read write
----------- ----- ----- ----- ----- ----- -----
pool1 4.01G 274G 2.25K 0 17.9M 0
mirror 4.01G 274G 2.25K 0 17.9M 0
c0t9d0 - - 108 0 9.00M 0
c0t11d0 - - 109 0 8.92M 0
----------- ----- ----- ----- ----- ----- -----
== ~8k reads to pool, ~85k reads to drives
Using volblocksize of 64k:
-bash-3.00# zfs send pool1/vol64k at now > /dev/null
capacity operations bandwidth
pool used avail read write read write
----------- ----- ----- ----- ----- ----- -----
pool1 6.01G 272G 378 0 23.5M 0
mirror 6.01G 272G 378 0 23.5M 0
c0t9d0 - - 189 0 11.8M 0
c0t11d0 - - 189 0 11.7M 0
----------- ----- ----- ----- ----- ----- -----
== ~64k reads to pool and drives
-bash-3.00# dd if=/dev/zvol/dsk/pool1/vol64k at now of=/dev/null bs=64k
capacity operations bandwidth
pool used avail read write read write
----------- ----- ----- ----- ----- ----- -----
pool1 6.01G 272G 414 0 25.7M 0
mirror 6.01G 272G 414 0 25.7M 0
c0t9d0 - - 107 0 12.9M 0
c0t11d0 - - 106 0 12.8M 0
----------- ----- ----- ----- ----- ----- -----
== ~64k reads to pool, ~124k reads to drives
Using volblocksize of 128k:
-bash-3.00# zfs send pool1/vol128k at now > /dev/null
capacity operations bandwidth
pool used avail read write read write
----------- ----- ----- ----- ----- ----- -----
pool1 4.01G 274G 188 0 23.3M 0
mirror 4.01G 274G 188 0 23.3M 0
c0t9d0 - - 94 0 11.7M 0
c0t11d0 - - 93 0 11.7M 0
----------- ----- ----- ----- ----- ----- -----
== ~128k reads to pool and drives
-bash-3.00# dd if=/dev/zvol/dsk/pool1/vol128k at now of=/dev/null bs=128k
capacity operations bandwidth
pool used avail read write read write
----------- ----- ----- ----- ----- ----- -----
pool1 4.01G 274G 247 0 30.8M 0
mirror 4.01G 274G 247 0 30.8M 0
c0t9d0 - - 122 0 15.3M 0
c0t11d0 - - 123 0 15.5M 0
----------- ----- ----- ----- ----- ----- -----
== ~128k reads to pool and drives
--
---------------------------------------------------
Brian H. Nelson Youngstown State University
System Administrator Media and Academic Computing
bnelson[at]cis.ysu.edu
---------------------------------------------------
Brian H. Nelson
2009-Jan-20 16:39 UTC
[zfs-discuss] Performance issue with zfs send of a zvol (Again)
Nobody can comment on this? -Brian Brian H. Nelson wrote:> I noticed this issue yesterday when I first started playing around with > zfs send/recv. This is on Solaris 10U6. > > It seems that a zfs send of a zvol issues ''volblocksize'' reads to the > physical devices. This doesn''t make any sense to me, as zfs generally > consolidates read/write requests to improve performance. Even the dd > case with the same snapshot does not exhibit this behavior. It seems to > be specific to zfs send. > > I checked with 8k, 64k, and 128k volblocksize, and the reads generated > by zfs send always seem to follow that size, while the reads with dd do not. > > The small reads seems to hurt performance of zfs send. I tested with a > mirror, but on another machine with a 7 disk raidz, the performance is > MUCH worse because the 8k reads get broken up into even smaller reads > and spread across the raidz. > > Is this a bug, or can someone explain why this is happening? > > Thanks > -Brian > > Using 8k volblocksize: > > -bash-3.00# zfs send pool1/vol8k at now > /dev/null > > capacity operations bandwidth > pool used avail read write read write > ----------- ----- ----- ----- ----- ----- ----- > pool1 4.01G 274G 1.88K 0 15.0M 0 > mirror 4.01G 274G 1.88K 0 15.0M 0 > c0t9d0 - - 961 0 7.46M 0 > c0t11d0 - - 968 0 7.53M 0 > ----------- ----- ----- ----- ----- ----- ----- > == ~8k reads to pool and drives > > -bash-3.00# dd if=/dev/zvol/dsk/pool1/vol8k at now of=/dev/null bs=8k > > capacity operations bandwidth > pool used avail read write read write > ----------- ----- ----- ----- ----- ----- ----- > pool1 4.01G 274G 2.25K 0 17.9M 0 > mirror 4.01G 274G 2.25K 0 17.9M 0 > c0t9d0 - - 108 0 9.00M 0 > c0t11d0 - - 109 0 8.92M 0 > ----------- ----- ----- ----- ----- ----- ----- > == ~8k reads to pool, ~85k reads to drives > > > Using volblocksize of 64k: > > -bash-3.00# zfs send pool1/vol64k at now > /dev/null > > capacity operations bandwidth > pool used avail read write read write > ----------- ----- ----- ----- ----- ----- ----- > pool1 6.01G 272G 378 0 23.5M 0 > mirror 6.01G 272G 378 0 23.5M 0 > c0t9d0 - - 189 0 11.8M 0 > c0t11d0 - - 189 0 11.7M 0 > ----------- ----- ----- ----- ----- ----- ----- > == ~64k reads to pool and drives > > -bash-3.00# dd if=/dev/zvol/dsk/pool1/vol64k at now of=/dev/null bs=64k > > capacity operations bandwidth > pool used avail read write read write > ----------- ----- ----- ----- ----- ----- ----- > pool1 6.01G 272G 414 0 25.7M 0 > mirror 6.01G 272G 414 0 25.7M 0 > c0t9d0 - - 107 0 12.9M 0 > c0t11d0 - - 106 0 12.8M 0 > ----------- ----- ----- ----- ----- ----- ----- > == ~64k reads to pool, ~124k reads to drives > > > Using volblocksize of 128k: > > -bash-3.00# zfs send pool1/vol128k at now > /dev/null > > capacity operations bandwidth > pool used avail read write read write > ----------- ----- ----- ----- ----- ----- ----- > pool1 4.01G 274G 188 0 23.3M 0 > mirror 4.01G 274G 188 0 23.3M 0 > c0t9d0 - - 94 0 11.7M 0 > c0t11d0 - - 93 0 11.7M 0 > ----------- ----- ----- ----- ----- ----- ----- > == ~128k reads to pool and drives > > -bash-3.00# dd if=/dev/zvol/dsk/pool1/vol128k at now of=/dev/null bs=128k > > capacity operations bandwidth > pool used avail read write read write > ----------- ----- ----- ----- ----- ----- ----- > pool1 4.01G 274G 247 0 30.8M 0 > mirror 4.01G 274G 247 0 30.8M 0 > c0t9d0 - - 122 0 15.3M 0 > c0t11d0 - - 123 0 15.5M 0 > ----------- ----- ----- ----- ----- ----- ----- > == ~128k reads to pool and drives > >-- --------------------------------------------------- Brian H. Nelson Youngstown State University System Administrator Media and Academic Computing bnelson[at]cis.ysu.edu ---------------------------------------------------
Richard Elling
2009-Jan-20 17:20 UTC
[zfs-discuss] Performance issue with zfs send of a zvol (Again)
Brian H. Nelson wrote:> Nobody can comment on this? > > -Brian > > > Brian H. Nelson wrote: >> I noticed this issue yesterday when I first started playing around with >> zfs send/recv. This is on Solaris 10U6. >> >> It seems that a zfs send of a zvol issues ''volblocksize'' reads to the >> physical devices. This doesn''t make any sense to me, as zfs generally >> consolidates read/write requests to improve performance. Even the dd >> case with the same snapshot does not exhibit this behavior. It seems to >> be specific to zfs send. >> >> I checked with 8k, 64k, and 128k volblocksize, and the reads generated >> by zfs send always seem to follow that size, while the reads with dd do not.This is what I would expect, though there seems to be an optimization opportunity. When doing a send, the things sent are records which are of the size volblocksize. When doing a dd, the minimum size read will be a record, but there may also be prefetching involved. The optimization opportunity would be to enable prefetching when needed. However, this may not be readily apparent on devices which allow multiple outstanding I/Os.>> The small reads seems to hurt performance of zfs send. I tested with a >> mirror, but on another machine with a 7 disk raidz, the performance is >> MUCH worse because the 8k reads get broken up into even smaller reads >> and spread across the raidz. >> >> Is this a bug, or can someone explain why this is happening?This is how raidz works, today. A record is split across the columns of the raidz set. -- richard>> Thanks >> -Brian >> >> Using 8k volblocksize: >> >> -bash-3.00# zfs send pool1/vol8k at now > /dev/null >> >> capacity operations bandwidth >> pool used avail read write read write >> ----------- ----- ----- ----- ----- ----- ----- >> pool1 4.01G 274G 1.88K 0 15.0M 0 >> mirror 4.01G 274G 1.88K 0 15.0M 0 >> c0t9d0 - - 961 0 7.46M 0 >> c0t11d0 - - 968 0 7.53M 0 >> ----------- ----- ----- ----- ----- ----- ----- >> == ~8k reads to pool and drives >> >> -bash-3.00# dd if=/dev/zvol/dsk/pool1/vol8k at now of=/dev/null bs=8k >> >> capacity operations bandwidth >> pool used avail read write read write >> ----------- ----- ----- ----- ----- ----- ----- >> pool1 4.01G 274G 2.25K 0 17.9M 0 >> mirror 4.01G 274G 2.25K 0 17.9M 0 >> c0t9d0 - - 108 0 9.00M 0 >> c0t11d0 - - 109 0 8.92M 0 >> ----------- ----- ----- ----- ----- ----- ----- >> == ~8k reads to pool, ~85k reads to drives >> >> >> Using volblocksize of 64k: >> >> -bash-3.00# zfs send pool1/vol64k at now > /dev/null >> >> capacity operations bandwidth >> pool used avail read write read write >> ----------- ----- ----- ----- ----- ----- ----- >> pool1 6.01G 272G 378 0 23.5M 0 >> mirror 6.01G 272G 378 0 23.5M 0 >> c0t9d0 - - 189 0 11.8M 0 >> c0t11d0 - - 189 0 11.7M 0 >> ----------- ----- ----- ----- ----- ----- ----- >> == ~64k reads to pool and drives >> >> -bash-3.00# dd if=/dev/zvol/dsk/pool1/vol64k at now of=/dev/null bs=64k >> >> capacity operations bandwidth >> pool used avail read write read write >> ----------- ----- ----- ----- ----- ----- ----- >> pool1 6.01G 272G 414 0 25.7M 0 >> mirror 6.01G 272G 414 0 25.7M 0 >> c0t9d0 - - 107 0 12.9M 0 >> c0t11d0 - - 106 0 12.8M 0 >> ----------- ----- ----- ----- ----- ----- ----- >> == ~64k reads to pool, ~124k reads to drives >> >> >> Using volblocksize of 128k: >> >> -bash-3.00# zfs send pool1/vol128k at now > /dev/null >> >> capacity operations bandwidth >> pool used avail read write read write >> ----------- ----- ----- ----- ----- ----- ----- >> pool1 4.01G 274G 188 0 23.3M 0 >> mirror 4.01G 274G 188 0 23.3M 0 >> c0t9d0 - - 94 0 11.7M 0 >> c0t11d0 - - 93 0 11.7M 0 >> ----------- ----- ----- ----- ----- ----- ----- >> == ~128k reads to pool and drives >> >> -bash-3.00# dd if=/dev/zvol/dsk/pool1/vol128k at now of=/dev/null bs=128k >> >> capacity operations bandwidth >> pool used avail read write read write >> ----------- ----- ----- ----- ----- ----- ----- >> pool1 4.01G 274G 247 0 30.8M 0 >> mirror 4.01G 274G 247 0 30.8M 0 >> c0t9d0 - - 122 0 15.3M 0 >> c0t11d0 - - 123 0 15.5M 0 >> ----------- ----- ----- ----- ----- ----- ----- >> == ~128k reads to pool and drives >> >> >