rsync - Mar 2006 - Help -- rsync Causing High Load Averages

This is my situation and I am running into dead ends.

We have a server with about 400GB of data that we are trying to backup
with rsync. On the content1 server we had rsyncd.conf as:

[content1]
   path = /
   comment = Backup
   list = no
   read only = yes
   hosts allow = 192.168.22.181
   hosts deny = *
   uid = root
   gid = root

and on the backup server we had a crontab entry as follows:

0 7 * * * /usr/bin/rsync -av --stats --delete 192.168.22.182::content1
/var1/content1

This will pull everything over to the backup server. When it runs,
however, the load averages on the content1 server continue to grow to
the 100s, bringing the server to a practical standstill. After a day of
tweaking, and killing the content1 server periodically, I have not been
able to reduce the loads much.

Both servers are dual xeon 3.0 with about 1TB of storage configured in a
RAID5 with SATA drives, and a separate OS drive. Running CentOS version
2.6.15.6. Rsync version 2.6.7  protocol version 29.

I tried tweaking the rsync commands and this is where I ended up, but
still it eventually brings the load average up and the sever to it knees.

So now, rsyncd.conf on the content1 server is:

max connections = 1

[content1]
   path = /
   comment = Backup
   list = no
   read only = yes
   hosts allow = 192.168.22.181
   hosts deny = *
   uid = root
   gid = root

I also created a shell script to start rsync in daemon mode:

#!/bin/sh
exec nice -n 19 /usr/bin/rsync --daemon $*

That was to try to get the load down.

Here is a snapshot of the load now and it is pushing 50.7 Mbits/sec:

top - 17:42:50 up 14 days, 21:53,  2 users,  load average: 0.23, 0.33, 1.06
Tasks: 467 total,   1 running, 466 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.4% us,  0.8% sy,  0.0% ni, 88.5% id, 10.2% wa,  0.1% hi,  0.0% si
Mem:   4148912k total,  4131716k used,    17196k free,    10112k buffers
Swap:  1020116k total,     1000k used,  1019116k free,  3783936k cached

On the backup server we tried:

/bin/nice -n 19 /usr/bin/rsync -rlptgoWv --stats --delete-after
192.168.22.182::content1 /var1/content1

We are not worried about network traffic, as these 2 servers are
connected via a LAN. Most of the files are images and media files.

I have not had problems with rsync in the past, and 3 other servers,
much smaller, are being backed up just fine. I am not sure what other
tweaks I can use, so any suggestions on how to keep the load averages
rising to the point where the server is useless would be appreciated.

Thanks
Ron

On Tue, 2006-03-28 at 16:58 -0800, Plugger wrote:
> We have a server with about 400GB of data that we are trying to backup
> with rsync. [...] When it runs,
> however, the load averages on the content1 server continue to grow to
> the 100s, bringing the server to a practical standstill.
If your individual files are larger than a gigabyte or so, Shachar
Shemesh's dynamic hash patch may improve performance significantly.  I
recommend you try an rsync with that patch.  To build one, extract the
rsync source package, run "patch -p1 <patches/dynamic_hash.diff",
configure, and make.

-- 
Matt McCutchen
hashproduct@verizon.net
http://hashproduct.metaesthetics.net/

On Tue, Mar 28, 2006 at 04:58:42PM -0800, Plugger wrote:
> When it runs, however, the load averages on the content1 server
> continue to grow to the 100s, bringing the server to a practical
> standstill. After a day of tweaking, and killing the content1 server
> periodically, I have not been able to reduce the loads much.
You should first determine what lacking resource is causing the high
load average.  If the CPU is maxed out, the decision to "nice" rsync
is
a good one, but also consider using the --whole-file option (it will use
less CPU on the sender, and is a good choice for those with a fast
network connection and too little CPU).  If it is not CPU, it is
probably disk I/O, which rsync doesn't have a very good way to ration --
you can try using --bwlimit (which doesn't really help much during the
filesystem scan) or try the slow-down.diff in the patches dir (which is
an overly-simple sleep idiom that only affects the filesystem scan), but
neither one is really the right solution when rsync is using too much
I/O (and the OS doesn't have a "nice" equivalent for prioritizing
multiple process's access to I/O).

..wayne..