CentOS - Jan 2015 - network copy performance is poor (rsync) - debugging suggestions?

Hi,

I do have two centos 6.6 servers. With a "performance optimized" rsync
I
get an speed of 15 - 20 MB/s

The options I use are:

rsync -aHAXxv --numeric-ids --progress -e "ssh -T -c arcfour -o
Compression=no -x"

If I copy files by smb to/from the servers I do get 60 - 80 MB/s, a dd
(r/w) on the storages attached gives 90 MB/s on the 1Gbit ISCSI (Source
Server) and up to 600MB/s on the 10Gbit ISCSI (Destination Server) storage.

Both servers have plenty of memory and cpu usage looks low.

Currently we dont use jumbo frames. Network over all usage is moderate
to low. There are no special sysctl tweeks yet in use.

As mentioned, I'm confused that even with SMB I do get 3 to 4 times
better performance.

Any hint and suggestion to track that problem down is welcome!

	Thanks and best regards . G?tz

-- 
G?tz Reinicke
IT-Koordinator

Tel. +49 7141 969 82 420
E-Mail goetz.reinicke at filmakademie.de

Filmakademie Baden-W?rttemberg GmbH
Akademiehof 10
71638 Ludwigsburg
www.filmakademie.de

Eintragung Amtsgericht Stuttgart HRB 205016

Vorsitzender des Aufsichtsrats: J?rgen Walter MdL
Staatssekret?r im Ministerium f?r Wissenschaft,
Forschung und Kunst Baden-W?rttemberg

Gesch?ftsf?hrer: Prof. Thomas Schadt

>
> I do have two centos 6.6 servers. With a "performance optimized"
rsync I
> get an speed of 15 - 20 MB/s
>
> The options I use are:
>
> rsync -aHAXxv --numeric-ids --progress -e "ssh -T -c arcfour -o
> Compression=no -x"
>
> If I copy files by smb to/from the servers I do get 60 - 80 MB/s, a dd
> (r/w) on the storages attached gives 90 MB/s on the 1Gbit ISCSI (Source
> Server) and up to 600MB/s on the 10Gbit ISCSI (Destination Server) storage.
>
> Both servers have plenty of memory and cpu usage looks low.
>
> Currently we dont use jumbo frames. Network over all usage is moderate
> to low. There are no special sysctl tweeks yet in use.
>
> As mentioned, I'm confused that even with SMB I do get 3 to 4 times
> better performance.
>
> Any hint and suggestion to track that problem down is welcome!
>
 Not an expert in rsync/ssh, but I'm pretty sure it's ssh's tcp
window size
that is the slowness. ssh is trying to leak the minimal amount of
information to anyone eavesdropping. If speed is your main concern,
http://psc.edu/index.php/hpn-ssh, or rsync to an nfs mount.

On Fri, January 23, 2015 12:54 pm, Patrick Flaherty
wrote:
>>
>> I do have two centos 6.6 servers. With a "performance
optimized" rsync I
>> get an speed of 15 - 20 MB/s
>>
>> The options I use are:
>>
>> rsync -aHAXxv --numeric-ids --progress -e "ssh -T -c arcfour -o
>> Compression=no -x"
>>
>> If I copy files by smb to/from the servers I do get 60 - 80 MB/s, a dd
>> (r/w) on the storages attached gives 90 MB/s on the 1Gbit ISCSI (Source
>> Server) and up to 600MB/s on the 10Gbit ISCSI (Destination Server)
>> storage.
>>
>> Both servers have plenty of memory and cpu usage looks low.
>>
>> Currently we dont use jumbo frames. Network over all usage is moderate
>> to low. There are no special sysctl tweeks yet in use.
>>
>> As mentioned, I'm confused that even with SMB I do get 3 to 4 times
>> better performance.
>>
>> Any hint and suggestion to track that problem down is welcome!
>>
>
>  Not an expert in rsync/ssh, but I'm pretty sure it's ssh's tcp
window
> size
> that is the slowness. ssh is trying to leak the minimal amount of
> information to anyone eavesdropping. If speed is your main concern,
> http://psc.edu/index.php/hpn-ssh, or rsync to an nfs mount.
I'm not certain what the problem could be. But enabling jumbo packets
would be the fist thing I would try, after turning off firewall to test if
it isn't involved. Joining "other advises" move: we usually use
bbftp and
gridftp for massive data transfers. Bbftp:

http://doc.in2p3.fr/bbftp/

gridftp is available with globus installation.

Just my $0.02

Valeri

++++++++++++++++++++++++++++++++++++++++
Valeri Galtsev
Sr System Administrator
Department of Astronomy and Astrophysics
Kavli Institute for Cosmological Physics
University of Chicago
Phone: 773-702-4247
++++++++++++++++++++++++++++++++++++++++

On 01/23/2015 01:44 AM, G?tz Reinicke - IT Koordinator
wrote:
> I do have two centos 6.6 servers. With a "performance optimized"
rsync I
> get an speed of 15 - 20 MB/s
That *is* pretty slow for sustained writes.  Does the same rate hold 
true for individual large files as it does for lots of small ones?  What 
filesystem are you using on each side?
> rsync -aHAXxv --numeric-ids --progress -e "ssh -T -c arcfour -o
> Compression=no -x"
It's worth noting that -X and -A are going to perform filesystem IO that 
you don't see on SMB, because it isn't going to preserve/set ACLs and 
extended attributes (IIRC).  So, one possibility is that you're seeing a 
difference in rate because you're doing lots of small files and 
filesystem operations are relatively slow.

You might drop those two options and see how that affects the rate.  If 
you determine that those are the cause of the performance difference, 
you can turn them back on, understanding that there's a cost associated 
with preserving that data.
> Both servers have plenty of memory and cpu usage looks low.
Define low.  If you're using top and press '1' to expand the CPU
lines,
you'll probably see one cpu with higher "us" percentage, which is
SSH
encrypting the data.  What percentage is that?  Is there a large value 
in "sy" or "hi" on any CPU?  Probably not since you see good
rates using
'dd' and smb copies, but I've seen systems where interrupt
processing
was a major bottleneck, so I make it a standard check.

On Wed, Jan 28, 2015 at 03:39:50PM -0800, Gordon Messmer
wrote:
> On 01/23/2015 01:44 AM, G?tz Reinicke - IT Koordinator wrote:
> >I do have two centos 6.6 servers. With a "performance
optimized" rsync I
> >get an speed of 15 - 20 MB/s
Add this parameter:   --bwlimit=0

We routinely have to sync 4TB, which is about 2M files...



Rsync never does well for us - it just cant push the line at all



So, this may or may not work for you - but this is a huge problem - so we tried 
whole excel spreadsheet worth of combinations, every protocol imaginable to make
this happen



In the end, after a year of constant work on this -



We found if we map a network share from Server Source to Server Destination, and
use CIFS protocol to "map a drive" then sync say



/srv/www -> to /mnt/shadow-www



It worked at 99% of line rate ONLY if we used the cp command to sync the source
and destination



Cd /srv/www



root at pas01#cp -R -u * /mnt/shadow-www/





something to consider if you find yourself not getting "line rate"



our investigation showed the rsync process even with all switches we found has
to "open" the file a bit before it copies it... so rsync sucks for
this kind of stuff with 2 MILLION small files - it never gets going moving
millions of small files it has to keep reading. There a switch that says
don't do that - but never really helped :)



Cheers



-----Original Message-----
From: centos-bounces at centos.org [mailto:centos-bounces at centos.org] On
Behalf Of Gordon Messmer
Sent: Wednesday, January 28, 2015 06:40 PM
To: CentOS mailing list
Subject: Re: [CentOS] network copy performance is poor (rsync) - debugging
suggestions?



On 01/23/2015 01:44 AM, G?tz Reinicke - IT Koordinator wrote:
> I do have two centos 6.6 servers. With a "performance optimized"
rsync
> I get an speed of 15 - 20 MB/s


That *is* pretty slow for sustained writes.  Does the same rate hold true for
individual large files as it does for lots of small ones?  What filesystem are
you using on each side?


> rsync -aHAXxv --numeric-ids --progress -e "ssh -T -c arcfour -o
> Compression=no -x"


It's worth noting that -X and -A are going to perform filesystem IO that you
don't see on SMB, because it isn't going to preserve/set ACLs and
extended attributes (IIRC).  So, one possibility is that you're seeing a
difference in rate because you're doing lots of small files and filesystem
operations are relatively slow.



You might drop those two options and see how that affects the rate.  If you
determine that those are the cause of the performance difference, you can turn
them back on, understanding that there's a cost associated with preserving
that data.


> Both servers have plenty of memory and cpu usage looks low.


Define low.  If you're using top and press '1' to expand the CPU
lines, you'll probably see one cpu with higher "us" percentage,
which is SSH encrypting the data.  What percentage is that?  Is there a large
value in "sy" or "hi" on any CPU?  Probably not since you
see good rates using 'dd' and smb copies, but I've seen systems
where interrupt processing was a major bottleneck, so I make it a standard
check.



_______________________________________________

CentOS mailing list

CentOS at centos.org<mailto:CentOS at centos.org>

http://lists.centos.org/mailman/listinfo/centos

2015-01-23 11:44 GMT+02:00 G?tz Reinicke - IT Koordinator <
goetz.reinicke at filmakademie.de>:
> Hi,
>
> I do have two centos 6.6 servers. With a "performance optimized"
rsync I
> get an speed of 15 - 20 MB/s
>
> The options I use are:
>
> rsync -aHAXxv --numeric-ids --progress -e "ssh -T -c arcfour -o
> Compression=no -x"
>
> If I copy files by smb to/from the servers I do get 60 - 80 MB/s, a dd
> (r/w) on the storages attached gives 90 MB/s on the 1Gbit ISCSI (Source
> Server) and up to 600MB/s on the 10Gbit ISCSI (Destination Server) storage.
>
> Both servers have plenty of memory and cpu usage looks low.
>
> Currently we dont use jumbo frames. Network over all usage is moderate
> to low. There are no special sysctl tweeks yet in use.
>
> As mentioned, I'm confused that even with SMB I do get 3 to 4 times
> better performance.
>
> Any hint and suggestion to track that problem down is welcome!
>
Problem is ssh, do not use it or apply hpnssh-patches:
http://sourceforge.net/projects/hpnssh/ to improve network performance

--
Eero

On 28-01-2015 21:39, Gordon Messmer wrote:
> On 01/23/2015 01:44 AM, G?tz Reinicke - IT Koordinator wrote:
>> I do have two centos 6.6 servers. With a "performance
optimized" rsync I
>> get an speed of 15 - 20 MB/s
>
> That *is* pretty slow for sustained writes.  Does the same rate hold
> true for individual large files as it does for lots of small ones?  What
> filesystem are you using on each side?
>
>> rsync -aHAXxv --numeric-ids --progress -e "ssh -T -c arcfour -o
>> Compression=no -x"
>
> It's worth noting that -X and -A are going to perform filesystem IO
that
> you don't see on SMB, because it isn't going to preserve/set ACLs
and
> extended attributes (IIRC).  So, one possibility is that you're seeing
a
> difference in rate because you're doing lots of small files and
> filesystem operations are relatively slow.
>
> You might drop those two options and see how that affects the rate.  If
> you determine that those are the cause of the performance difference,
> you can turn them back on, understanding that there's a cost associated
> with preserving that data.
>
>> Both servers have plenty of memory and cpu usage looks low.
>
> Define low.  If you're using top and press '1' to expand the
CPU lines,
> you'll probably see one cpu with higher "us" percentage,
which is SSH
> encrypting the data.  What percentage is that?  Is there a large value
> in "sy" or "hi" on any CPU?  Probably not since you see
good rates using
> 'dd' and smb copies, but I've seen systems where interrupt
processing
> was a major bottleneck, so I make it a standard check.
+1 on all above.

Also, it's likely that your ssh process is going to limit the transfer. 
Yet, if you remotely mount the share (cifs/nfs) and do rsync on top of 
it, it may give you line rate but also may end up transferring data that 
wouldn't have to be transferred (specially if you use rsync -c option). 
It will also transfer over network millions syscalls for reading mtime's 
and if your sync was going to transfer just 5% of total payload, it may 
take longer then do it via ssh.

If that's the case (ssh limiting), I would simply consider splitting 
this process into several rsyncs. Spawn one for each subdir, for example 
(and maybe limit to 4, 8 simultaneous processes). That should scale well 
if your storage doesn't complain. It's an easy shell script to write and
that's what rsync ends up doing anyway.

Never used, just found it when searching for "parallel rsync": 
http://moo.nac.uci.edu/~hjm/parsync/  may be useful.

   Marcelo

On 01/23/2015 10:54 AM, Patrick Flaherty wrote:
>   Not an expert in rsync/ssh, but I'm pretty sure it's ssh's
tcp window size
> that is the slowness.
Probably not?  TCP window size is usually only a problem if the sender 
is capable of sending more data than the window size before the receiver 
can ACK.  On low latency networks, it usually isn't a problem.