zfs discuss - May 2011 - Quick zfs send -i performance questions

Hi all,

I''m playing around with nearline backups using zfs send | zfs recv.
A full backup made this way takes quite a lot of time, so I was
wondering: after the initial copy, would using an incremental send
(zfs send -i) make the process much quick because only the stuff that
had changed between the previous snapshot and the current one be
copied?  Is my understanding of incremental zfs send correct?

Also related to this is a performance question.  My initial test involved
copying a 50 MB zfs file system to a new disk, which took 2.5 minutes
to complete.  The strikes me as being a bit high for a mere 50 MB;
are my expectation realistic or is it just because of my very budget
concious set up?  If so, where''s the bottleneck?

The source pool is on a pair of 146 GB 10K RPM disks on separate
busses in a D1000 (split bus arrangement) and the destination pool
is on a IOMega 1 GB USB attached disk.  The machine to which both
pools are connected is a Sun Blade 1000 with a pair of 900 MHz US-III
CPUs and 2 GB of RAM.  The HBA is Sun''s dual differential UltraSCSI
PCI card.  The machine was relatively quiescent apart from doing the
local zfs send | zfs recv.

-- 
Rich Teer, Publisher
Vinylphile Magazine

www.vinylphilemag.com

On Tue, May  3 at 17:39, Rich Teer wrote:
>Hi all,
>
>I''m playing around with nearline backups using zfs send | zfs recv.
>A full backup made this way takes quite a lot of time, so I was
>wondering: after the initial copy, would using an incremental send
>(zfs send -i) make the process much quick because only the stuff that
>had changed between the previous snapshot and the current one be
>copied?  Is my understanding of incremental zfs send correct?
Your understanding is correct.  We use -I, not -i, since it can send
multiple snapshots with a single command.  Only the amount of changed
data is sent with an incremental ''zfs send''.
>Also related to this is a performance question.  My initial test involved
>copying a 50 MB zfs file system to a new disk, which took 2.5 minutes
>to complete.  The strikes me as being a bit high for a mere 50 MB;
>are my expectation realistic or is it just because of my very budget
>concious set up?  If so, where''s the bottleneck?
Our setup does a send/recv at roughly 40MB/s over ssh connected to a
1gbit/s ethernet connection.  There are ways to make this faster by
not using an encrypted transport, but setup is a bit more advanced
than just an ssh ''zfs recv'' command line.
>The source pool is on a pair of 146 GB 10K RPM disks on separate
>busses in a D1000 (split bus arrangement) and the destination pool
>is on a IOMega 1 GB USB attached disk.  The machine to which both
>pools are connected is a Sun Blade 1000 with a pair of 900 MHz US-III
>CPUs and 2 GB of RAM.  The HBA is Sun''s dual differential UltraSCSI
>PCI card.  The machine was relatively quiescent apart from doing the
>local zfs send | zfs recv.
I''m guessing that the USB bus and/or the USB disk is part of your
bottleneck.  UltraSCSI should be plenty fast and your CPU should be
fine too.

--eric


-- 
Eric D. Mudama
edmudama at bounceswoosh.org

On 2011-May-04 08:39:39 +0800, Rich Teer <rich.teer at rite-group.com>
wrote:
>Also related to this is a performance question.  My initial test involved
>copying a 50 MB zfs file system to a new disk, which took 2.5 minutes
>to complete.  The strikes me as being a bit high for a mere 50 MB;
>are my expectation realistic or is it just because of my very budget
>concious set up?  If so, where''s the bottleneck?
Possibilities I can think of:
- Do you have lots of snapshots?  There''s an overhead of a second or so
  for each snapshot to be sent.
- Is the source pool heavily fragmented with lots of small files?
>The source pool is on a pair of 146 GB 10K RPM disks on separate
>busses in a D1000 (split bus arrangement) and the destination pool
>is on a IOMega 1 GB USB attached disk.  The machine to which both
>pools are connected is a Sun Blade 1000 with a pair of 900 MHz US-III
>CPUs and 2 GB of RAM.
Hopefully a silly question but does the SB1000 support USB2?  All of
the Sun hardware I''ve dealt with only has USB1 ports.

And, BTW, 2GB RAM is very light on for ZFS (though I note you only
have a very small amount of data).

-- 
Peter Jeremy
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 196 bytes
Desc: not available
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20110504/5cba9453/attachment.bin>

On Wed, 4 May 2011, Peter Jeremy wrote:
> Possibilities I can think of:
> - Do you have lots of snapshots?  There''s an overhead of a second
or so
>   for each snapshot to be sent.
> - Is the source pool heavily fragmented with lots of small files?
Nope, and I don''t think so.
> Hopefully a silly question but does the SB1000 support USB2?  All of
> the Sun hardware I''ve dealt with only has USB1 ports.
Not such a silly question.  :-)  The USB1 port was indeed the source of
much of the bottleneck.  The same 50 MB file system took only 8 seconds
to copy when I plugged the drive into a USB 2.0 card I had in the machine!

An 80 GB file system took 2 hours with the USB 2 port in use, with
compression off.  I''m tryin git again right now with compression 
turned on in the receiving pool.  Should be interesting...
> And, BTW, 2GB RAM is very light on for ZFS (though I note you only
> have a very small amount of data).
True, but the SB1000 only supports 2GB of RAM IIRC!  I''ll soon be
migrating this machine''s duties to an Ultra 20 M2.  A faster CPU
and 4 GB should make an noticable improvement (not to mention, on
board USB 2.0 ports).

Thanks for you ideas!

-- 
Rich Teer, Publisher
Vinylphile Magazine

www.vinylphilemag.com

On Tue, May 3, 2011 19:39, Rich Teer wrote:
> I''m playing around with nearline backups using zfs send | zfs
recv.
> A full backup made this way takes quite a lot of time, so I was
> wondering: after the initial copy, would using an incremental send
> (zfs send -i) make the process much quick because only the stuff that
> had changed between the previous snapshot and the current one be
> copied?  Is my understanding of incremental zfs send correct?
Yes, that works.  In my setup, a full backup takes 6 hours (about 800GB of
data to an external USB 2 drive), the incremental maybe 20 minutes even if
I''ve added several gigabytes of images.
> Also related to this is a performance question.  My initial test involved
> copying a 50 MB zfs file system to a new disk, which took 2.5 minutes
> to complete.  The strikes me as being a bit high for a mere 50 MB;
> are my expectation realistic or is it just because of my very budget
> concious set up?  If so, where''s the bottleneck?
In addition to issues others have mentiond, the way incremental send
works, it follows the order the blocks were written in rather than disk
order, so that can sometimes be bad.
-- 
David Dyer-Bennet, dd-b at dd-b.net; http://dd-b.net/
Snapshots: http://dd-b.net/dd-b/SnapshotAlbum/data/
Photos: http://dd-b.net/photography/gallery/
Dragaera: http://dragaera.info

> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-
> bounces at opensolaris.org] On Behalf Of Rich Teer
> 
> Also related to this is a performance question.  My initial test involved
> copying a 50 MB zfs file system to a new disk, which took 2.5 minutes
> to complete.  The strikes me as being a bit high for a mere 50 MB;
> 
> The source pool is on a pair of 146 GB 10K RPM disks on separate
> busses in a D1000 (split bus arrangement) and the destination pool
> is on a IOMega 1 GB USB attached disk.  
Even the fastest USB3 thumb drive is slower than the slowest cheapest hard
drive.  Whatever specs are published on the supposed speed of the flash
drive, don''t believe them.  I''m not saying they lie - just
that they publish
the fastest conceivable speed under ideal situations, which are totally
unrealistic and meaningless in the real world.

I suspect you''re using a junky 1G slow-as-dirt usb thumb drive.

> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-
> bounces at opensolaris.org] On Behalf Of Rich Teer
> 
> Not such a silly question.  :-)  The USB1 port was indeed the source of
> much of the bottleneck.  The same 50 MB file system took only 8 seconds
> to copy when I plugged the drive into a USB 2.0 card I had in the machine!
50 Mbit/sec

> An 80 GB file system took 2 hours with the USB 2 port in use, with
88 Mbit/sec.

> True, but the SB1000 only supports 2GB of RAM IIRC!  I''ll soon be
> migrating this machine''s duties to an Ultra 20 M2.  A faster CPU
> and 4 GB should make an noticable improvement (not to mention, on
> board USB 2.0 ports).
4G is also lightweight, unless you''re not doing much of anything.  No
dedup,
no L2ARC, just simple pushing bits around.  No services running...  Just ssh

I don''t understand why so many people are building systems with
insufficient
ram these days.  I don''t put less than 8G into a personal laptop
anymore...

On Wed, 4 May 2011, Edward Ned Harvey wrote:
> I suspect you''re using a junky 1G slow-as-dirt usb thumb drive.
Nope--unless an IOMega Prestige Desktop Hard Drive (containing an
Hitachi 7200K RPM hard drive with 32MB of cache) counts as a slow
as dirt USB thumb drive!  

-- 
Rich Teer, Publisher
Vinylphile Magazine

www.vinylphilemag.com

On Wed, 4 May 2011, Edward Ned Harvey wrote:
> 4G is also lightweight, unless you''re not doing much of anything. 
No dedup,
> no L2ARC, just simple pushing bits around.  No services running...  Just
ssh
Yep, that''s right.  This is a repurposed workstation for use in my home
network.
> I don''t understand why so many people are building systems with
insufficient
> ram these days.  I don''t put less than 8G into a personal laptop
anymore...
The Ultra 20 only supports 4 GB of RAM, and I''ve installed that much. 
It
can''t hold any more!  I have to make do with the resources I have here,
with my next to $0 budget...

-- 
Rich Teer, Publisher
Vinylphile Magazine

www.vinylphilemag.com

On Tue, May 3, 2011 at 11:42 PM, Peter Jeremy <
peter.jeremy at alcatel-lucent.com> wrote:
> - Is the source pool heavily fragmented with lots of small files?
>
Peter,

  We''ve some servers holding Xen VMs and the setup was create to have a
default VM from where others would be cloned so the space saving are quite
good.

  The problem we''ve started seeing is that a zfs send -i is taking
hours to
send a very small amount of data (eg. 20GB in 6 hours) while a zfs send full
transfer everything faster than the incremental (40-70MB/s). Sometimes we
just give up on sending the incremental and send a full altogether.

  I''m wondering if it has to do with fragmentation too. Has anyone
experience this? OpenSolaris b111. As a data point, we also have servers
holding Vmware VMs (not cloned) and there is no problem. Anyone know
what''s
special about Xen''s cloned VMs? Sparse files maybe?

Thanks,

-- 
Giovanni Tirloni
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20110504/db5b6bfb/attachment.html>

On Wed, May 4, 2011 at 2:25 PM, Giovanni Tirloni <gtirloni at
sysdroid.com> wrote:
> ? The problem we''ve started seeing is that a zfs send -i is taking
hours to
> send a very small amount of data (eg. 20GB in 6 hours) while a zfs send
full
> transfer everything faster than the incremental (40-70MB/s). Sometimes we
> just give up on sending the incremental and send a full altogether.
Does the send complete faster if you just pipe to /dev/null? I''ve
observed that if recv stalls, it''ll pause the send, and they two go
back and forth stepping on each other''s toes. Unfortunately, send and
recv tend to pause with each individual snapshot they are working on.

Putting something like mbuffer
(http://www.maier-komor.de/mbuffer.html) in the middle can help smooth
it out and speed things up tremendously. It prevents the send from
pausing when the recv stalls, and allows the recv to continue working
when the send is stalled. You will have to fiddle with the buffer size
and other options to tune it for your use.

-B

-- 
Brandon High : bhigh at freaks.com

On 05/03/11 22:45, Rich Teer wrote:
>
> True, but the SB1000 only supports 2GB of RAM IIRC!  I''ll soon be
Actually you can get up to 16GB ram in a SB1000 (or SB2000). The 4GB
dimms are most likely not too common however the 1GB and 2GB dimms seem
to be common. At one time Dataram and maybe Kingston made 4GB dimms for
the SB1000 and SB2000. And don''t forget can also put the 1.2GHz
processors
in it.

Even with all that it is still not even close to the speed of the U20 M2
you are mentioning below. At least as a workstation...
> migrating this machine''s duties to an Ultra 20 M2.  A faster CPU
> and 4 GB should make an noticable improvement (not to mention, on
You can also get up to 8GB ram in the U20 M2.
> board USB 2.0 ports).
>
> Thanks for you ideas!
>

-- 
----------------------------------
Randy Jones
E-Mail: randy at jones.tri.net
----------------------------------

On Wed, May 4, 2011 at 9:04 PM, Brandon High <bhigh at freaks.com> wrote:
> On Wed, May 4, 2011 at 2:25 PM, Giovanni Tirloni <gtirloni at
sysdroid.com>
> wrote:
> >   The problem we''ve started seeing is that a zfs send -i is
taking hours
> to
> > send a very small amount of data (eg. 20GB in 6 hours) while a zfs
send
> full
> > transfer everything faster than the incremental (40-70MB/s). Sometimes
we
> > just give up on sending the incremental and send a full altogether.
>
> Does the send complete faster if you just pipe to /dev/null? I''ve
> observed that if recv stalls, it''ll pause the send, and they two
go
> back and forth stepping on each other''s toes. Unfortunately, send
and
> recv tend to pause with each individual snapshot they are working on.
>
> Putting something like mbuffer
> (http://www.maier-komor.de/mbuffer.html) in the middle can help smooth
> it out and speed things up tremendously. It prevents the send from
> pausing when the recv stalls, and allows the recv to continue working
> when the send is stalled. You will have to fiddle with the buffer size
> and other options to tune it for your use.
>

We''ve done various tests piping it to /dev/null and then transferring
the
files to the destination. What seems to stall is the recv because it
doesn''t
complete (through mbuffer, ssh, locally, etc). The zfs send always complete
at the same rate.

Mbuffer is being used but doesn''t seem to help. When things start to
stall,
the in / out buffers will quickly fill up and nothing will be sent. Probably
because the mbuffer on the other side can''t receive any more data until
the
zfs recv gives it some air to breath.

What I find it curious is that it only happens with incrementals. Full
send''s go as fast as possible (monitored with mbuffer). I was just
wondering
if other people have seen it, if there is a bug (b111 is quite old), etc.

-- 
Giovanni Tirloni
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20110505/2cb636f7/attachment.html>

On Thu, May 5, 2011 at 2:17 PM, Giovanni Tirloni <gtirloni at
sysdroid.com> wrote:
> What I find it curious is that it only happens with incrementals. Full
> send''s go as fast as possible (monitored with mbuffer). I was just
wondering
> if other people have seen it, if there is a bug (b111 is quite old), etc.
    I have been using zfs send / recv via ssh and a WAN connection to
replicate about 20 TB of data. One initial Full followed by an
Incremental every 4 hours. This has been going on for over a year and
I have not had any reliability issues. I started at Solaris 10U6, then
10U8, and now 10U9.

    I did run into a bug early on that if the ssh failed, then the zfs
recv would hang, but that was fixed ages ago.

-- 
{--------1---------2---------3---------4---------5---------6---------7---------}
Paul Kraus
-> Senior Systems Architect, Garnet River ( http://www.garnetriver.com/ )
-> Sound Coordinator, Schenectady Light Opera Company (
http://www.sloctheater.org/ )
-> Technical Advisor, RPI Players

On Thu, May 5, 2011 at 11:17 AM, Giovanni Tirloni <gtirloni at
sysdroid.com> wrote:
> What I find it curious is that it only happens with incrementals. Full
> send''s go as fast as possible (monitored with mbuffer). I was just
wondering
> if other people have seen it, if there is a bug (b111 is quite old), etc.
I missed that you were using b111 earlier. That''s probably a large
part of the problem. There were a lot of performance and reliability
improvements between b111 and b134, and there have been more between
b134 and b148 (OI) or b151 (S11 Express).

Updating the host you''re receiving on to something more recent may fix
the performance problem you''re seeing.

Fragmentation shouldn''t be to great of an issue if the pool
you''re
writing to is relatively empty. There were changes made to zpool
metaslab allocation post-b111 that might improve performance for pools
between 70% and 96% full. This could also be why the full sends
perform better than incremental sends.

-B

-- 
Brandon High : bhigh at freaks.com