rsync - Apr 2006 - too many connections on one module?

I have ran into an odd problem with rsync and modules. Here are the details:

There are about 2500 clients (of various rsync and OS versions beyond my
control) syncing every hour to one server running rsync as a daemon.

There are 4 modules. 3 are different branches of the same code. These 3
modules each contain about 50k files. Most of the files are <1k or even
empty, with a few files being above 10M and the total size being about
100M. The other module contains about 180 files, 30M total size.

About 1500 clients rsync from the 4th module and have no problem. About 50
clients rsync from the 2nd and 3rd module and also have no problem.

The remaining 1000 rsync from the first module doing two passes. The first
pass is the top level dir excluding the dir containing about 45k of the
files. This is relatively fast and no problems are seen.

The second pass is the problem. On this pass the clients are syncing about
10 files from the previously excluded dir only. For some reason rsync
sessions take an excessively long time to complete (hours as opposed to
minutes). Eventually the connections pile up and load on the server has
gotten as high as 900!

The clients that rsync to the 2nd and 3rd module are doing the same thing
except they don't take a long time (even when the load is high). They take
a few minutes at most on the second pass.

Have I reached some per module limit?

Thanks
Kimo

On Thu, 2006-04-27 at 12:13 -0700, kimor79@sonic.net
wrote:
> The second pass is the problem. On this pass the clients are syncing about
> 10 files from the previously excluded dir only. For some reason rsync
> sessions take an excessively long time to complete (hours as opposed to
> minutes). Eventually the connections pile up and load on the server has
> gotten as high as 900!
This previously excluded directory has a lot of big files, right?  In
rsync's incremental transfer protocol, the sender uses a considerable
amount of CPU time: it scans each file byte by byte, looking for regions
matching the block hashes it got from the receiver.  This usually isn't
a problem, but with 1000 clients at a time, I can see why the
connections back up.

Having the clients disable incremental transfer on the second pass by
passing --whole-file will remove the CPU load but increase network
traffic.

Notes for rsync developers:

This problem highlights the need for several changes to rsync's
incremental transfer protocol.  I believe the sender should compute and
send the block hashes.  Then the receiver should scan byte-by-byte,
transferring matched data to the temporary file and informing the sender
of what literal data it needs.  Then the sender should send the
necessary literal data.  The cost is a third stage to the pipeline:
"send sums, request blocks, send blocks" instead of "send sums,
send
blocks and matching information".  There are several benefits:

(1) Less information about the soon-to-be-overwritten receiver version
of the file leaks from receiver to sender.  The sender only finds out
what parts of its file the receiver needs, not the hash of every block
the receiver currently has.  This is important when the receiver doesn't
trust the sender.

(2) The receiver does the hard work of the byte-by-byte scan.  This
distributes load when there is one sender and many receivers.  This case
is much more common than the other way around.

(3) The sender, if it wishes to further reduce its load, could compute
the block hashes of its files once and send them in bulk to receivers.
I suggest storing block hashes in cache files kept alongside the data
files or in a separate hierarchy.  The sender could decide whether to
trust a block hash file by storing the size and mtime of the original
file in the data of the block hash file and using the quick check.

-- 
Matt McCutchen
hashproduct@verizon.net
http://hashproduct.metaesthetics.net/