rsync - Oct 2011 - rsync compares all files again and again

Hi,

we do a 1:1 backup from our main raid to a backup raid every night with

rsync -a --delete /mnt/raid1/ /mnt/raid2

rsync is 3.09, filesystems are ext3, OS is SLES 11 SP1.

The rsync process takes several hours, even if no file has changed at all.
Using -vv I see that rsync compares all files every time and that takes
long for some hundreds of millions of small files.

Can I tell rsync it should not compare files in a directory if the 
directory has not changed, i.e., ctime is the same in source and target?

I.e., when I have a dir x with a subdir y and a file y1 in it, I see:

rsync -avv x z
created directory z
delta-transmission disabled for local transfer or --whole-file
x/
x/y/
x/y/y1

on the first run and

rsync -avv x z
x/y/y1 is uptodate

on every following run. Why does rsync compare y1 again? 

I couldn't find a solution in the mailing list or FAQ and when googling.
I found a mail in this list telling that -a was the culprit and one
should add -O to fix it, but it doesn't make any difference.

cu,
Frank

-- 
Dipl.-Inform. Frank Steiner   Web:  http://www.bio.ifi.lmu.de/~steiner/
Lehrstuhl f. Bioinformatik    Mail: http://www.bio.ifi.lmu.de/~steiner/m/
LMU, Amalienstr. 17           Phone: +49 89 2180-4049
80333 Muenchen, Germany       Fax:   +49 89 2180-99-4049
* Rekursion kann man erst verstehen, wenn man Rekursion verstanden hat. *

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Use --itemize-changes to find out why.  In fact, --verbose is almost
entirely worthless without --itemize-changes.

On 10/14/11 05:13, Frank Steiner wrote:
> Hi,
> 
> we do a 1:1 backup from our main raid to a backup raid every night with
> 
> rsync -a --delete /mnt/raid1/ /mnt/raid2
> 
> rsync is 3.09, filesystems are ext3, OS is SLES 11 SP1.
> 
> The rsync process takes several hours, even if no file has changed at all.
> Using -vv I see that rsync compares all files every time and that takes
> long for some hundreds of millions of small files.
> 
> Can I tell rsync it should not compare files in a directory if the 
> directory has not changed, i.e., ctime is the same in source and target?
> 
> I.e., when I have a dir x with a subdir y and a file y1 in it, I see:
> 
> rsync -avv x z
> created directory z
> delta-transmission disabled for local transfer or --whole-file
> x/
> x/y/
> x/y/y1
> 
> on the first run and
> 
> rsync -avv x z
> x/y/y1 is uptodate
> 
> on every following run. Why does rsync compare y1 again? 
> 
> I couldn't find a solution in the mailing list or FAQ and when
googling.
> I found a mail in this list telling that -a was the culprit and one
> should add -O to fix it, but it doesn't make any difference.
> 
> cu,
> Frank
> 
- -- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
	Kevin Korb			Phone:    (407) 252-6853
	Systems Administrator		Internet:
	FutureQuest, Inc.		Kevin at FutureQuest.net  (work)
	Orlando, Florida		kmk at sanitarium.net (personal)
	Web page:			http://www.sanitarium.net/
	PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk6YWCgACgkQVKC1jlbQAQeySgCg1+5T+XDdW+y+CQTsytIAgw3G
WRUAnRY1gDFkL6MwGmWGpvPcsaCb8Y3K
=oXQk
-----END PGP SIGNATURE-----

On 10/14/2011 02:13 AM, Frank Steiner wrote:
> Hi,
>
> we do a 1:1 backup from our main raid to a backup raid every night with
>
> rsync -a --delete /mnt/raid1/ /mnt/raid2
>
> rsync is 3.09, filesystems are ext3, OS is SLES 11 SP1.
>
> The rsync process takes several hours, even if no file has changed at all.
> Using -vv I see that rsync compares all files every time and that takes
> long for some hundreds of millions of small files.
>
> Can I tell rsync it should not compare files in a directory if the
> directory has not changed, i.e., ctime is the same in source and target?
>
> I.e., when I have a dir x with a subdir y and a file y1 in it, I see:
>
> rsync -avv x z
> created directory z
> delta-transmission disabled for local transfer or --whole-file
> x/
> x/y/
> x/y/y1
>
> on the first run and
>
> rsync -avv x z
> x/y/y1 is uptodate
>
> on every following run. Why does rsync compare y1 again?
>
> I couldn't find a solution in the mailing list or FAQ and when
googling.
> I found a mail in this list telling that -a was the culprit and one
> should add -O to fix it, but it doesn't make any difference.
>
> cu,
> Frank
>
Perhaps using the ext3 dirindex option would help?

-- 
-Eric 'shubes'