rsync - Apr 2006 - Using --link-dest works on small directory tree but not large one

I'm using version 2.6.3 on OS X. I'm trying to use --link-dest to keep
some
rolling backups of production websites. There is about 2GB of data in 60,000
total files. I do an initial rsync to get an initial copy of the data, and
each day thereafter I run rsync again with --link-dest to create a new daily
backup. For some reason, in the subsequent runs, rsync updates a whole lot
of files that haven't changed at all. The files in the source and in the
link-dest location have identical contents, are the same size, and have the
same modified date. I can't seem to find any patterns in the files it
decides to bring back down the subsequent times. Sometimes it will bring
down the entire directory tree again, and sometimes it will bring down
slightly less than the whole tree.

I've tried running the same commands on subsets of the directory tree, and
everything works perfectly. The problem only appears when I try to sync my
entire directory tree.

This is what I'm doing:

$ rsync -vv --progress --archive --compress --numeric-ids --delete -e ssh
blah@blah.com:/usr/home/blah/websites/ /backup/websites/1
$ rsync -vv --progress --archive --compress --numeric-ids --delete
--link-dest=../1 -e ssh blah@blah.com:/usr/home/blah/websites/
/backup/websites/2
$ du -hsc /backup/websites/*
2.1G    /backup/websites/1
2.1G    /backup/websites/2
4.2G    total

If, however, I run the following, it all seems to work perfectly:

$ rsync -vv --progress --archive --compress --numeric-ids --delete -e ssh
blah@blah.com:/usr/home/blah/websites/site1.com /backup/websites/1
$ rsync -vv --progress --archive --compress --numeric-ids --delete
--link-dest=../1 -e ssh blah@blah.com:/usr/home/blah/websites/site1.com
/backup/websites/2
$ du -hsc /backup/websites/*
35M    /backup/websites/1
0    /backup/websites/2
35M    total

This is frustrating to debug because there's a lot of data being
transferred, and it only occurs when I try to sync the full tree. Is this a
bug? Am I doing something completely wrong and stupid? I'm happy to provide
more detailed info if you need it. Thanks!!
-------------- next part --------------
HTML attachment scrubbed and removed

On Thursday 06 Apr 2006 11:19, Robot Robot wrote:
> I'm using version 2.6.3 on OS X. I'm trying to use --link-dest to
keep some
> rolling backups of production websites. There is about 2GB of data in
> 60,000 total files. I do an initial rsync to get an initial copy of the
> data, and each day thereafter I run rsync again with --link-dest to create
> a new daily backup. For some reason, in the subsequent runs, rsync updates
> a whole lot of files that haven't changed at all. The files in the
source
> and in the link-dest location have identical contents, are the same size,
> and have the same modified date. I can't seem to find any patterns in
the
> files it decides to bring back down the subsequent times. Sometimes it will
> bring down the entire directory tree again, and sometimes it will bring
> down slightly less than the whole tree.
Sounds like you want to try rsnapshot
http://rsnapshot.org/
>
> I've tried running the same commands on subsets of the directory tree,
and
> everything works perfectly. The problem only appears when I try to sync my
> entire directory tree.
>
> This is what I'm doing:
>
> $ rsync -vv --progress --archive --compress --numeric-ids --delete -e ssh
> blah@blah.com:/usr/home/blah/websites/ /backup/websites/1
> $ rsync -vv --progress --archive --compress --numeric-ids --delete
> --link-dest=../1 -e ssh blah@blah.com:/usr/home/blah/websites/
> /backup/websites/2
> $ du -hsc /backup/websites/*
> 2.1G    /backup/websites/1
> 2.1G    /backup/websites/2
> 4.2G    total
>
> If, however, I run the following, it all seems to work perfectly:
>
> $ rsync -vv --progress --archive --compress --numeric-ids --delete -e ssh
> blah@blah.com:/usr/home/blah/websites/site1.com /backup/websites/1
> $ rsync -vv --progress --archive --compress --numeric-ids --delete
> --link-dest=../1 -e ssh blah@blah.com:/usr/home/blah/websites/site1.com
> /backup/websites/2
> $ du -hsc /backup/websites/*
> 35M    /backup/websites/1
> 0    /backup/websites/2
> 35M    total
>
> This is frustrating to debug because there's a lot of data being
> transferred, and it only occurs when I try to sync the full tree. Is this a
> bug? Am I doing something completely wrong and stupid? I'm happy to
provide
> more detailed info if you need it. Thanks!!
-- 
-----------------
Bob Hutchinson
Midwales dot com
-----------------

On Thu, Apr 06, 2006 at 12:19:44AM -1000, Robot Robot
wrote:
> $ du -hsc /backup/websites/*
> 2.1G    /backup/websites/1
> 2.1G    /backup/websites/2
> 4.2G    total
You're asking du to report each directory separately, which is deceiving
you.  Run "du -hsc /backup/websites" instead.

..wayne..