zfs discuss - Dec 2012 - any more efficient way to transfer snapshot between two hosts than ssh tunnel?

Assuming in a secure and trusted env, we want to get the maximum transfer speed
without the overhead from ssh.

Thanks.

Fred
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20121213/654f543f/attachment-0001.html>

Hi Fred,

Try mbuffer (http://www.maier-komor.de/mbuffer.html)


On 14 December 2012 15:01, Fred Liu <Fred_Liu at issi.com> wrote:
>  Assuming in a secure and trusted env, we want to get the maximum
> transfer speed without the overhead from ssh.****
>
> ** **
>
> Thanks.****
>
> ** **
>
> Fred****
>
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss at opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
>
>

-- 
Adrian Smith (ISUnix), Ext: 55070
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20121214/c50df41d/attachment.html>

Adrian,

That is cool!

Thank you so much!

BTW, anyone played NDMP in solaris? Or is it feasible to transfer snapshot via
NDMP protocol?
Before the acquisition, SUN advocated the NDMP backup feature in the
openstorage/fishwork.
I am sorry if it is the wrong place to ask this question.

Thanks.

Fred

From: Adrian Smith [mailto:adrian.smith at rmit.edu.au]
Sent: ???, ??? 14, 2012 12:08
To: Fred Liu
Cc: zfs-discuss at opensolaris.org
Subject: Re: [zfs-discuss] any more efficient way to transfer snapshot between
two hosts than ssh tunnel?

Hi Fred,

Try mbuffer (http://www.maier-komor.de/mbuffer.html)

On 14 December 2012 15:01, Fred Liu <Fred_Liu at issi.com<mailto:Fred_Liu
at issi.com>> wrote:
Assuming in a secure and trusted env, we want to get the maximum transfer speed
without the overhead from ssh.

Thanks.

Fred

_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris.org<mailto:zfs-discuss at opensolaris.org>
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss



--
Adrian Smith (ISUnix), Ext: 55070




-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20121213/60d6806b/attachment.html>

On Dec 13, 2012 8:02 PM, "Fred Liu" <Fred_Liu at issi.com>
wrote:
>
> Assuming in a secure and trusted env, we want to get the maximum transfer
speed without the overhead from ssh.

Add the HPN patches to OpenSSH and enable the NONE cipher.  We can saturate
a gigabits link (980 mbps) between two FreeBSD hosts using that.

Without it, we were only able to hit ~480 mbps on a good day.

If you want 0 overhead, there''s always netcat. :)
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20121213/20fe1649/attachment.html>

>Add the HPN patches to OpenSSH and enable the NONE cipher.? We can saturate
a gigabits link (980 mbps) between two FreeBSD hosts using that.
>Without it, we were only able to hit ~480 mbps on a good day.
>If you want 0 overhead, there''s always netcat. :)
980mbps is awesome! I am thinking running two ssh services -- one normal and one
with HPN patches only for backup job.
But now sure they can work before I try them. I will also try netcat.

Many thanks.

Fred

> From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-
> bounces at opensolaris.org] On Behalf Of Fred Liu
> 
> BTW, anyone played NDMP in solaris? Or is it feasible to transfer snapshot
via
> NDMP protocol?
I''ve heard you could, but I''ve never done it.  Sorry
I''m not much help, except as a cheer leader.  You can do it!  I think
you can!  Don''t give up! heheheheh
Please post back whatever you find, or if you have to figure it out for
yourself, then blog about it and post that.

We have found mbuffer to be the fastest solution.   Our rates for large
transfers on 10GbE are:

280MB/s    mbuffer
220MB/s    rsh
180MB/s    HPN-ssh unencrypted
 60MB/s     standard ssh

The tradeoff mbuffer is a little more complicated to script;   rsh is, well, you
know;  and hpn-ssh requires rebuilding ssh and (probably) maintaining a second
copy of it.

     -- Trey Palmer
________________________________________
From: zfs-discuss-bounces at opensolaris.org [zfs-discuss-bounces at
opensolaris.org] on behalf of Fred Liu [Fred_Liu at issi.com]
Sent: Thursday, December 13, 2012 11:23 PM
To: Freddie Cash
Cc: zfs-discuss
Subject: Re: [zfs-discuss] any more efficient way to transfer snapshot between
two hosts than ssh tunnel?
>Add the HPN patches to OpenSSH and enable the NONE cipher.  We can saturate
a gigabits link (980 mbps) between two FreeBSD hosts using that.
>Without it, we were only able to hit ~480 mbps on a good day.
>If you want 0 overhead, there''s always netcat. :)
980mbps is awesome! I am thinking running two ssh services -- one normal and one
with HPN patches only for backup job.
But now sure they can work before I try them. I will also try netcat.

Many thanks.

Fred
_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

> 
> I''ve heard you could, but I''ve never done it.  Sorry
I''m not much help,
> except as a cheer leader.  You can do it!  I think you can!  Don''t
give
> up! heheheheh
> Please post back whatever you find, or if you have to figure it out for
> yourself, then blog about it and post that.

Aha! Gotcha! I will give it a try.

Post in the list.
> -----Original Message-----
> From: Fred Liu
> Sent: ???, ??? 14, 2012 23:41
> To: ''real-men-dont-click at gmx.net''
> Subject: RE: [zfs-discuss] any more efficient way to transfer snapshot
> between two hosts than ssh tunnel?
> 
> 
> 
> >
> > Hi Fred,
> >
> > I played with zfs send/reveive some time ago. One important thing I
> > learned was that netcat is not the first choice to use.
> > There is a tool called mbuffer out there. mbuffer works similar to
> > netcat but allows a specific buffer size and block size.
> > From various resources I found out that the best buffer and block
> sizes
> > for zfs send/receive seem to be 1GB for the buffer with a block size
> of
> > 131073.
> > Replacing netcat by mubuffer dramatically increases the throughput.
> >
> >
> > The resulting commands are like:
> >
> > ssh -f $REMOTESRV "/opt/csw/bin/mbuffer -q -I $PORT -m 1G -s
131072 |
> > zfs receive -vFd $REMOTEPOOL"
> >
> > zfs send $CURRENTLOCAL | /opt/csw/bin/mbuffer -q -O $REMOTESRV:$PORT
> -m
> > 1G -s 131072 > /dev/null
> >
> >
> > cu
> >
> 
> Carsten,
> 
> Thank you so much for the sharing and I will try it.
> 
> Fred

> 
> We have found mbuffer to be the fastest solution.   Our rates for large
> transfers on 10GbE are:
> 
> 280MB/s    mbuffer
> 220MB/s    rsh
> 180MB/s    HPN-ssh unencrypted
>  60MB/s     standard ssh
> 
> The tradeoff mbuffer is a little more complicated to script;   rsh is,
> well, you know;  and hpn-ssh requires rebuilding ssh and (probably)
> maintaining a second copy of it.
> 
>      -- Trey Palmer
> ________________________________________
In 10GbE env, even 280MB/s is not a so decent result. Maybe the alternative
could
be a two-step way. Putting snapshots via NFS/iSCSI and receiving them locally.
But that is not perfect.

In my own experiments with my own equivalent of mbuffer, it''s well
worth
giving the receiving side a buffer which is sized to hold the amount of 
data in a transaction commit, which allows ZFS to be banging out one tx 
group to disk, whilst the network is bringing the next one across for 
it. This will be roughly the link speed in bytes/second x 5, plus a bit 
more for good measure, say 250-300Mbytes for a gigabit link. It seems to 
be most important when the disks and the network link have similar max 
theoretical bandwidths (100Mbytes/sec is what you might expect from both 
gigabit ethernet and reasonable disks), and it becomes less important as 
the difference in max performance between them increases. Without the 
buffer, you tend to see the network run flat out for 5 seconds, and then 
the receiving disks run flat out for 5 seconds, alternating back and 
forth, whereas with the buffer, both continue streaming at full gigabit 
speed without a break.

I have not seen any benefit of buffering on the sending side, although 
I''d still be inclined to include a small one.

YMMV...


Palmer, Trey wrote:
> We have found mbuffer to be the fastest solution.   Our rates for large
transfers on 10GbE are:
>
> 280MB/s    mbuffer
> 220MB/s    rsh
> 180MB/s    HPN-ssh unencrypted
>  60MB/s     standard ssh
>
> The tradeoff mbuffer is a little more complicated to script;   rsh is,
well, you know;  and hpn-ssh requires rebuilding ssh and (probably) maintaining
a second copy of it.
>   
-- 
Andrew Gabriel

On Fri, Dec 14 at  9:29, Fred Liu wrote:
>
>>
>> We have found mbuffer to be the fastest solution.   Our rates for large
>> transfers on 10GbE are:
>>
>> 280MB/s    mbuffer
>> 220MB/s    rsh
>> 180MB/s    HPN-ssh unencrypted
>>  60MB/s     standard ssh
>>
>> The tradeoff mbuffer is a little more complicated to script;   rsh is,
>> well, you know;  and hpn-ssh requires rebuilding ssh and (probably)
>> maintaining a second copy of it.
>>
>>      -- Trey Palmer
>> ________________________________________
>
>In 10GbE env, even 280MB/s is not a so decent result. Maybe the alternative
could
>be a two-step way. Putting snapshots via NFS/iSCSI and receiving them
locally.
>But that is not perfect.
Even with infinite wire speed, you''re bound by the ability of the
source server to generate the snapshot stream and the ability of the
destination server to write the snapshots to the media.

Our little servers in-house using ZFS don''t read/write that fast when
pulling snapshot contents off the disks, since they''re essentially
random access on a server that''s been creating/deleting snapshots for
a long time.

--eric


-- 
Eric D. Mudama
edmudama at bounceswoosh.org

>
>Even with infinite wire speed, you''re bound by the ability of the
source server
>to generate the snapshot stream and the ability of the destination server to
>write the snapshots to the media.
>
>Our little servers in-house using ZFS don''t read/write that fast
when pulling
>snapshot contents off the disks, since they''re essentially random
access on a
>server that''s been creating/deleting snapshots for a long time.
>
>--eric
>
That is true. This discussion is assumed under ideal condition. We want to
minimize the overhead in transportation layer only.


Thanks.

Fred