zfs discuss - Sep 2006 - Fastest way to send 100gb ( with ZFS send )

Hey,

so i created a snapshot, ( zfs snapshot blah/mail/01 at 2006-09-29 ) and
then i tried to send it accross the network, ( zfs send
blah/mail/01 at 2006-09-29 | ssh user at 192.168.0.101 zfs recv
blah/mail/01 at 2006-09-29 ) and i only seem to be getting 2 - 4 mb /
s... unless i''m missing somthing. ( it''s a gbe cross over from
one
v20z, to another )

I''m getting the figures from the zfs list i''m doing on the
destination...

so, is there a faster way ? am i missing somthing ?

Patrick

-- 
Patrick
----------------------------------------
patrick <at> eefy <dot> net

On 9/29/06, Patrick <patrick at xsinet.co.za>
wrote:
> Hey,
>
> so i created a snapshot, ( zfs snapshot blah/mail/01 at 2006-09-29 ) and
> then i tried to send it accross the network, ( zfs send
> blah/mail/01 at 2006-09-29 | ssh user at 192.168.0.101 zfs recv
> blah/mail/01 at 2006-09-29 ) and i only seem to be getting 2 - 4 mb /
> s... unless i''m missing somthing. ( it''s a gbe cross over
from one
> v20z, to another )
>
> I''m getting the figures from the zfs list i''m doing on
the destination...
>
> so, is there a faster way ? am i missing somthing ?
That''s almost precisely the performance I got sending a bunch of data
across gbe via SSH myself -- but I was just doing an scp.  And it was
a slow processor, which probably was the actual limiting factor.  So
my suspicion is that ssh encryption is your limiting factor in that
test.

An easy test -- try rsh instead of ssh if you can.  Remember ssh is
encrypting everything it carries.  That''ll tell you if the problem is
network performance, encryption overhead, or ZFS behavior.

For that matter, do the zfs send to /dev/null; that''ll let you measure
the inescapable minimum ZFS part of the time.
-- 
David Dyer-Bennet, <mailto:dd-b at dd-b.net>,
<http://www.dd-b.net/dd-b/>
RKBA: <http://www.dd-b.net/carry/>
Pics: <http://www.dd-b.net/dd-b/SnapshotAlbum/>
Dragaera/Steven Brust: <http://dragaera.info/>

You''re most certainly are hitting the SSH limitation. Note that SSH/SCP
sessions are single threaded and won''t utilize all of the system
resources even if they are available.

Around 4 months back I was doing some testing between 2 fully configured T2000s
connected using crossover cables and figured the maximum I was able to push
across the wire was around <10MB/S. To further ensure that I didn''t
have other network problems I ran 10-15 simultaneous SCP sessions and was able
to push the network utilization up linearly. This told me that SSH (v2)
transfers cannot saturate a GigE channel.
 
 
This message posted from opensolaris.org

Anantha N. Srirama wrote:
> You''re most certainly are hitting the SSH limitation. Note that
> SSH/SCP sessions are single threaded and won''t utilize all of the
> system resources even if they are available.
You may want to try ''ssh -c blowfish'' to use the (faster)
blowfish
encryption algorithm rather than default of triple-DES.

--matt

Opteron 280 or 275 with blowfish cipher does 33MB/s, default (DES?) cipher does
25MB/s.

- Luke

Msg is shrt cuz m on ma treo

 -----Original Message-----
From: 	Anantha N. Srirama [mailto:anantha.srirama at cdc.hhs.gov]
Sent:	Saturday, September 30, 2006 12:34 PM Eastern Standard Time
To:	zfs-discuss at opensolaris.org
Subject:	[zfs-discuss] Re: Fastest way to send 100gb ( with ZFS send )

You''re most certainly are hitting the SSH limitation. Note that SSH/SCP
sessions are single threaded and won''t utilize all of the system
resources even if they are available.

Around 4 months back I was doing some testing between 2 fully configured T2000s
connected using crossover cables and figured the maximum I was able to push
across the wire was around <10MB/S. To further ensure that I didn''t
have other network problems I ran 10-15 simultaneous SCP sessions and was able
to push the network utilization up linearly. This told me that SSH (v2)
transfers cannot saturate a GigE channel.
 
 
This message posted from opensolaris.org
_______________________________________________
zfs-discuss mailing list
zfs-discuss at opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Matthew Ahrens wrote:
> Anantha N. Srirama wrote:
>> You''re most certainly are hitting the SSH limitation. Note
that
>> SSH/SCP sessions are single threaded and won''t utilize all of
the
>> system resources even if they are available.
> 
> You may want to try ''ssh -c blowfish'' to use the (faster)
blowfish
> encryption algorithm rather than default of triple-DES.
AES 128 in Counter Mode should be the default - aes-128-ctr.

The fastest for "streaming" data is usually arcfour.

-- 
Darren J Moffat

On Sun, 1 Oct 2006, Darren J Moffat wrote:
> Matthew Ahrens wrote:
> > Anantha N. Srirama wrote:
> >> You''re most certainly are hitting the SSH limitation.
Note that
> >> SSH/SCP sessions are single threaded and won''t utilize
all of the
> >> system resources even if they are available.
> >
> > You may want to try ''ssh -c blowfish'' to use the
(faster) blowfish
> > encryption algorithm rather than default of triple-DES.
>
> AES 128 in Counter Mode should be the default - aes-128-ctr.
>
> The fastest for "streaming" data is usually arcfour.
Agreed.  After some experimenting, here is a cut/paste from my post to
solarisx86 at yahoogroups.com.

For each host/zone you want to ssh efficiently to, add the following to
your $HOME/.ssh/config file:

Host 10.200.20.10
Ciphers=arcfour,aes128-cbc
Compression=yes
HostKeyAlgorithms=ssh-dss
Protocol=2
ForwardX11=yes

Regards,

Al Hopper  Logical Approach Inc, Plano, TX.  al at logical-approach.com
           Voice: 972.379.2133 Fax: 972.379.2134  Timezone: US CDT
OpenSolaris.Org Community Advisory Board (CAB) Member - Apr 2005
                OpenSolaris Governing Board (OGB) Member - Feb 2006