rsync - Sep 2006 - Rsync order of maginitude slower with twice as large directory

I am in the habit of syncing my /usr/local tree over several computers 
(all Mac OS X). Recently my /usr/local tree has doubled in size, from 
300,000 files to 600,000 files because I have added a large subversion 
checkout (TeX Live). As a result, rsync has become a time hog. Instead 
of a minute or two or so to calculate what needs to be sent over, 
suddenly this process takes 18-20 minutes, 10-20 times as long for 
twice the tree. If I use --exclude to skip the subversion tree, the 
process goes back to a minute or so.

How can I find out what causes this?

I am using rsync 2.6.0 with Kevin Boyd's HFS+ extensions, because one 
of the systems runs OS X 10.3.9 and does not have a recent 
Apple-provided HFS+-aware rsync.

Thanks,

G

Check swap usage during rsync as well as rsync memory usage; perhaps rsync 
dataset no longer fits into core, so swap slows it all down.

Wojtek

On Wed, 20 Sep 2006, Gerben Wierda wrote:
> Date: Wed, 20 Sep 2006 23:16:39 +0200
> From: Gerben Wierda <Gerben.Wierda@rna.nl>
> To: rsync@lists.samba.org
> Subject: Rsync order of maginitude slower with twice as large directory
> 
> I am in the habit of syncing my /usr/local tree over several computers 
> (all Mac OS X). Recently my /usr/local tree has doubled in size, from 
> 300,000 files to 600,000 files because I have added a large subversion 
> checkout (TeX Live). As a result, rsync has become a time hog. Instead 
> of a minute or two or so to calculate what needs to be sent over, 
> suddenly this process takes 18-20 minutes, 10-20 times as long for 
> twice the tree. If I use --exclude to skip the subversion tree, the 
> process goes back to a minute or so.
> 
> How can I find out what causes this?
> 
> I am using rsync 2.6.0 with Kevin Boyd's HFS+ extensions, because one 
> of the systems runs OS X 10.3.9 and does not have a recent 
> Apple-provided HFS+-aware rsync.
> 
> Thanks,
> 
> G
> 
>

On Sep 20, 2006, at 23:51, Wojtek.Pilorz wrote:
> Check swap usage during rsync as well as rsync memory usage; perhaps 
> rsync
> dataset no longer fits into core, so swap slows it all down.
I have checked this, but that is not the problem.

What I did notice is something else. Part of a smaller tree I am 
syncing also has moved o being from a svn repository. Suddenly this one 
also slows down enormously.

Looking on the system I see 2 rsync processes. It also happens when 
rsyncing 1 file:
> /usr/local/bin/rsync --rsync-path=/usr/local/bin/rsync --eahfs -e ssh 
> -avu  --progress \
>         gerben@hedwig:Library/Safari/Bookmarks.plist 
> /Users/gerben/Library/Safari/Bookmarks.plist
> receiving file list ...
> 1 file to consider
>
> wrote 16 bytes  read 111 bytes  28.22 bytes/sec
> total size is 35075  speedup is 276.18
And it hangs. PS shows:

   501  3847  3831   0  31  0    97800   5004 -      S+    p0    0:00.14 
/usr/local/bin/rsync --rsync-path=/usr/local/bin/rsync --eahfs -e ssh 
-avu --progress gerben@mail.rna.nl:Library/Safari/Bookmarks.plist 
/Users/gerben/Library/Safari/Bookmarks.plist
   501  3860  3847   0  31  0    97864   1984 -      R+    p0    1:46.07 
/usr/local/bin/rsync --rsync-path=/usr/local/bin/rsync --eahfs -e ssh 
-avu --progress gerben@mail.rna.nl:Library/Safari/Bookmarks.plist 
/Users/gerben/Library/Safari/Bookmarks.plist

97864 is taking up 100% CPU almost.

G

On Wed 20 Sep 2006, Gerben Wierda wrote:
> I am in the habit of syncing my /usr/local tree over several computers 
> (all Mac OS X). Recently my /usr/local tree has doubled in size, from 
> 300,000 files to 600,000 files because I have added a large subversion 
> checkout (TeX Live). As a result, rsync has become a time hog. Instead 
> of a minute or two or so to calculate what needs to be sent over, 
> suddenly this process takes 18-20 minutes, 10-20 times as long for 
> twice the tree. If I use --exclude to skip the subversion tree, the 
> process goes back to a minute or so.
> 
> How can I find out what causes this?
> 
> I am using rsync 2.6.0 with Kevin Boyd's HFS+ extensions, because one 
> of the systems runs OS X 10.3.9 and does not have a recent 
> Apple-provided HFS+-aware rsync.
If you can't run a different version of rsync, what does it matter what
causes it... Say it's been fixed in recent versions, would you be happy
knowing that, while you can't actually use the fixed version?


Paul Slootman