rsync - Feb 2006 - Problems with high load

Hi,
Specs: rsync is running as root with prio: 0
Linux: Debian Kernel: 2.4.30
Rsync: rsync  version 2.6.4  protocol version 29


We are having problems with a very high load at the time of an rsync is running.
Sometimes if 2 or 3 Rsyncs are running simultanously on to the same machine the
load goes up on
that machine to 10 or more. the datas will be synced in about 2 or 3 minutes
with about:
sent 3277383 bytes  received 3719902 bytes  31590.45 bytes/sec
total size is 391067345  speedup is 55.89

But the Load is so high that all the Webserveres are stopping its work.

We have no bandwith throttling because it is important to get its changes fast.

Any suggestions or answers??

Greetings
Matthias

On Thu 09 Feb 2006, Matthias Blohm wrote:
> We are having problems with a very high load at the time of an rsync is 
> running.
> Sometimes if 2 or 3 Rsyncs are running simultanously on to the same machine
> the load goes up on that machine to 10 or more. the datas will be synced in
> about 2 or 3 minutes with about:
> sent 3277383 bytes  received 3719902 bytes  31590.45 bytes/sec
> total size is 391067345  speedup is 55.89
> 
> But the Load is so high that all the Webserveres are stopping its work.
There are a couple of things you could try:

- try to limit the number of rsyncs to 1. You'll have a lot of disk
  thrashing if 3 rsyncs are competing for disk IO.

- Try to reduce the set of files being rsynced, if possible. E.g. run
  rsync multiple times, once for each subdir instead of running it over
  one larger higher directory. (You don't mention how many files are
  involved; output from --stats could help.)

- If files are generally replaced (and not updated in a smallish way),
  try --whole-file, which will reduce IO, as the old data isn't read
  from disk, it's just replaced.
> We have no bandwith throttling because it is important to get its changes 
> fast.
If you know what files are changed, --files-from might be useful.


Paul Slootman