rsync - Mar 2005 - Premature optimization in f_name_cmp()?

I appear to be hitting a case of premature optimization at my site -
with rsync 2.6.4pre2:

I backup a number of very large systems using rsync, and have been
updating as I try and fix various weird problems.  It appears that rsync
2.6.4pre2 fixes some of my other issues (timeouts in particular), but
now I'm being bitten by very high loads and slow transfers.

My current theory is that the offending code is from this checkin:

http://cvs.samba.org/cgi-bin/cvsweb/rsync/flist.c.diff?r1=1.272&r2=1.273
> If f_name_cmp() discovers that two directory strings compare to an 
> equal value without being equal pointers, substitute one of the
> pointers for the other in the file list.  This optimizes future name
> comparisons.  Note also that this optimization won't be triggered
> very often (because rsync tends to send the names grouped by dir-
> name at transmission time), but it's nice to be able to assume that
> all files in the same dir have identical dir-name pointers after the
> qsort is finished.
What seems to be happening for me is that we then walk, rather often,
the half-million files in the directory tree I am syncing to my backup
server.  This causes the server to spend a lot of time in a CPU loop,
and slows down the process just a little...

Is there a way to make this only look at the 'right' part of sorted-
flist, given it's sorted, and is this really needed at all?

Thanks,

Andrew Bartlett

-- 
Andrew Bartlett                                http://samba.org/~abartlet/
Authentication Developer, Samba Team           http://samba.org
Student Network Administrator, Hawker College  http://hawkerc.net
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
Url :
http://lists.samba.org/archive/rsync/attachments/20050314/b6b091d5/attachment.bin

On Mon, Mar 14, 2005 at 10:40:00AM +1100, Andrew Bartlett
wrote:
> Is there a way to make this only look at the 'right' part of
sorted-
> flist, given it's sorted, and is this really needed at all?
Yeah, that code is really pretty silly.  I neglected to test my
assumption about how often it would get triggered, and it is actually
getting run much more often than expected.  I'm just removing that bit
of code for the next release (and tweaking some code in generator for
the --fuzzy option handling so that it doesn't depend on the dirnames
having identical string pointers).

Thanks for the detective work!

..wayne..

On Mon, Mar 14, 2005 at 02:43:05PM -0800, Wayne Davison
wrote:
> Another note:  if you want --fuzzy to help you with compressed files
> that have differing contents, you should use the --rsyncable option
> to gzip to compress the files.
Looks like you already thought of that.  Let me further state that
using --delete-after instead of --delete lets --fuzzy find renamed
files (which would otherwise be removed before they could be used)
but it is not absolutely required if all you need is fuzzy matching
for updated versions.

..wayne..