sync performance falls off a cliff

Mike Connell
Thu, 16 Jul 2009 04:09:50 -0000

Hi,

Here is an update. I haven't deployed a new version of rsync into
production.
Instead I split my current rsync up into 10 independent sub directories of
the
main directory. I run them serially one after the other.

I'm up to 404,000 files and the total sync time doesn't seem to be
falling
off
a cliff (yet).

In my case, only about .1% of my files change, so I'm sure it isn't a
rsync
memory
issue. But I strongly suspect with the results I'm getting so far, that it
is a
matter of how many directories and inodes can be kept cached in memory. The
largest of the 10 sub directory rsyncs is about 75,000 files. So this would
seem
to put less pressure on this cache.

Thanks,

Mike