rsync - Feb 2009 - Macintosh rsync: Inconsistent Behaviour: Hints Appreciated

I have a problem with rsync on macintosh copying files that have not been
updated. Despite careful reading of the archives, I can't find an
explanation (or a fix), but I strongly suspect I'm just missing something
that an expert will recognise instantly... Your assistance would be greatly
appreciated.

I'm running a nightly backup of my photo/home video directory from a
partition BIGPIX on a firewire drive on one machine (G3 running OSX 10.3,
imaginatively named g3) to another (Pix) on a firewire drive on my laptop
(G4 running 10.4, even more imaginatively bobsmac). I'll include the script
below. All are HFS+ partitions.

The problem is, that every night, something like 3/4 of the whole directory
gets updated. This doesn't reflect reality - in fact, I only add stuff to
the directory once a week or so. If the whole of the directory was getting
updated, I'd understand that I just had the options wrong, but it's
completely inconsistent - some files get re-synched night after night, while
others are just synched once and then rsync does what I expected and leaves
them alone. I can't see any consistency in this. There is some correlation
with age (the older directories are much more likely to be re-synched than
newer ones). But this correlation is not reliable. I'll use an example of a
particular directory from May 2008. Of five files in the directory, two
(creation dates Jan 7 2008) are synched every night. The other three are
omitted. One has creation date Nov. 15 2007, the other two April 14 2008.
Other than age, I can't see any difference between the files, either from
unix shell (see below) or from finder info (I'd include this if I could
figure a way to cut and paste, but please accept that it is completely
consistent with the shell results). The rsync options I'm using are
'-vpcrutxzE --ignore-existing'; I've read the man pages carefully,
and I
can't understand why they would produce this behaviour.

If you can see any way I can further diagnose this, any hints would be
appreciated - I'm completely stuck. Of course, if you know why it's
happening - or know a fix - that would be even better!

     Thanks and Best Wishes
     Bob

*********************** directory listings of example source and target
directories  ********************

G3:/Volumes/BIGPIX/PIXREADONLY/200805-FriendsMisc080502 rim$ ls -l
total 12472
-rwxr-xr-x   1 root  admin  1827015 Apr 14  2008 HannahP1060453.jpg
-rwxr-xr-x   1 root  admin  2219607 Apr 14  2008 HannahP1060578.jpg
-rwxr-xr-x   1 root  admin   192966 Jan  7  2008 VikyP1030042.jpg
-rwxr-xr-x   1 root  admin   206962 Jan  7  2008 VivP1030020.jpg
-rwxr-xr-x   1 root  admin  1900455 Nov 15  2007 WOOK-DSCN3830.jpg

bobsmac:/Volumes/Pix/PIXBACKUP/PIXREADONLY/200805-FriendsMisc080502 rim$ ls
-l
total 12472
-rwxr-xr-x   1 rim  rim  1827015 Apr 14  2008 HannahP1060453.jpg
-rwxr-xr-x   1 rim  rim  2219607 Apr 14  2008 HannahP1060578.jpg
-rwxr-xr-x   1 rim  rim   192966 Jan  7  2008 VikyP1030042.jpg
-rwxr-xr-x   1 rim  rim   206962 Jan  7  2008 VivP1030020.jpg
-rwxr-xr-x   1 rim  rim  1900455 Nov 15  2007 WOOK-DSCN3830.jpg

(Viky and Viv get synched repeatedly, the rest do not)


*********************** abbreviated transcript showing relevant stages of
copying  ********************

Copying g3.local:/Volumes/BIGPIX/PIXREADONLY /Volumes/Pix/PIXBACKUP
receiving file list ... done
PIXREADONLY/._.DS_Store
PIXREADONLY/200202-China/Wuhan Early Pics/._PIC_6250.JPG
.
.
.
PIXREADONLY/200805-FriendsMisc080502/._.DS_Store
PIXREADONLY/200805-FriendsMisc080502/._VikyP1030042.jpg
PIXREADONLY/200805-FriendsMisc080502/._VivP1030020.jpg
PIXREADONLY/200805-KaesongNK080511/._.DS_Store
.
.
.

sent 59156 bytes  received 115825818 bytes  48599.28 bytes/sec
total size is 22910676123  speedup is 197.70


*********************** shell script  ********************

#!/bin/sh

SRCDIR="g3.local:/Volumes/BIGPIX/PIXREADONLY"
DESTDIR="/Volumes/Pix/PIXBACKUP"
echo "Copying "$SRCDIR" "$DESTDIR
rsync -vpcrutxzE --ignore-existing\
        --rsync-path="/usr/bin/rsync" root@$SRCDIR $DESTDIR
-------------- next part --------------
HTML attachment scrubbed and removed

You're using the version of rsync that is built-in to Mac OS X.  Don't  
do that.  It's a fairly old version (though it is maintained by  
Apple), and it doesn't handle extended attribute modifications (or  
lack thereof), thus it transfers them every time.  When you have many  
files with resource forks or other extended attributes, you'll see  
files getting copied every time regardless of whether they've been  
modified.  rsync 3.0.x handles these much more efficiently.

I posted some tips for compiling and running rsync on Mac OS X here:

http://www.bombich.com/mactips/rsync.html

Note that the arguments will be slightly different, so be sure to  
review the man page and modify your script accordingly.  Also be sure  
to call rsync from /usr/local/bin/rsync (or wherever you choose to  
install it), otherwise you'll still be using the built-in version.

Mike

On Feb 11, 2009, at 6:59 PM, Bob McKay wrote:
> I have a problem with rsync on macintosh copying files that have not  
> been updated. Despite careful reading of the archives, I can't find  
> an explanation (or a fix), but I strongly suspect I'm just missing  
> something that an expert will recognise instantly... Your assistance  
> would be greatly appreciated.
>
> I'm running a nightly backup of my photo/home video directory from a  
> partition BIGPIX on a firewire drive on one machine (G3 running OSX  
> 10.3, imaginatively named g3) to another (Pix) on a firewire drive  
> on my laptop (G4 running 10.4, even more imaginatively bobsmac).  
> I'll include the script below. All are HFS+ partitions.
>
> The problem is, that every night, something like 3/4 of the whole  
> directory gets updated. This doesn't reflect reality - in fact, I  
> only add stuff to the directory once a week or so. If the whole of  
> the directory was getting updated, I'd understand that I just had  
> the options wrong, but it's completely inconsistent - some files get  
> re-synched night after night, while others are just synched once and  
> then rsync does what I expected and leaves them alone. I can't see  
> any consistency in this. There is some correlation with age (the  
> older directories are much more likely to be re-synched than newer  
> ones). But this correlation is not reliable. I'll use an example of  
> a particular directory from May 2008. Of five files in the  
> directory, two (creation dates Jan 7 2008) are synched every night.  
> The other three are omitted. One has creation date Nov. 15 2007, the  
> other two April 14 2008. Other than age, I can't see any difference  
> between the files, either from unix shell (see below) or from finder  
> info (I'd include this if I could figure a way to cut and paste, but  
> please accept that it is completely consistent with the shell  
> results). The rsync options I'm using are '-vpcrutxzE --ignore- 
> existing'; I've read the man pages carefully, and I can't
understand
> why they would produce this behaviour.
>
> If you can see any way I can further diagnose this, any hints would  
> be appreciated - I'm completely stuck. Of course, if you know why  
> it's happening - or know a fix - that would be even better!
>
>      Thanks and Best Wishes
>      Bob
>
> *********************** directory listings of example source and  
> target directories  ********************
>
> G3:/Volumes/BIGPIX/PIXREADONLY/200805-FriendsMisc080502 rim$ ls -l
> total 12472
> -rwxr-xr-x   1 root  admin  1827015 Apr 14  2008 HannahP1060453.jpg
> -rwxr-xr-x   1 root  admin  2219607 Apr 14  2008 HannahP1060578.jpg
> -rwxr-xr-x   1 root  admin   192966 Jan  7  2008 VikyP1030042.jpg
> -rwxr-xr-x   1 root  admin   206962 Jan  7  2008 VivP1030020.jpg
> -rwxr-xr-x   1 root  admin  1900455 Nov 15  2007 WOOK-DSCN3830.jpg
>
> bobsmac:/Volumes/Pix/PIXBACKUP/PIXREADONLY/200805-FriendsMisc080502  
> rim$ ls -l
> total 12472
> -rwxr-xr-x   1 rim  rim  1827015 Apr 14  2008 HannahP1060453.jpg
> -rwxr-xr-x   1 rim  rim  2219607 Apr 14  2008 HannahP1060578.jpg
> -rwxr-xr-x   1 rim  rim   192966 Jan  7  2008 VikyP1030042.jpg
> -rwxr-xr-x   1 rim  rim   206962 Jan  7  2008 VivP1030020.jpg
> -rwxr-xr-x   1 rim  rim  1900455 Nov 15  2007 WOOK-DSCN3830.jpg
>
> (Viky and Viv get synched repeatedly, the rest do not)
>
>
> *********************** abbreviated transcript showing relevant  
> stages of copying  ********************
>
> Copying g3.local:/Volumes/BIGPIX/PIXREADONLY /Volumes/Pix/PIXBACKUP
> receiving file list ... done
> PIXREADONLY/._.DS_Store
> PIXREADONLY/200202-China/Wuhan Early Pics/._PIC_6250.JPG
> .
> .
> .
> PIXREADONLY/200805-FriendsMisc080502/._.DS_Store
> PIXREADONLY/200805-FriendsMisc080502/._VikyP1030042.jpg
> PIXREADONLY/200805-FriendsMisc080502/._VivP1030020.jpg
> PIXREADONLY/200805-KaesongNK080511/._.DS_Store
> .
> .
> .
>
> sent 59156 bytes  received 115825818 bytes  48599.28 bytes/sec
> total size is 22910676123  speedup is 197.70
>
>
> *********************** shell script  ********************
>
> #!/bin/sh
>
> SRCDIR="g3.local:/Volumes/BIGPIX/PIXREADONLY"
> DESTDIR="/Volumes/Pix/PIXBACKUP"
> echo "Copying "$SRCDIR" "$DESTDIR
> rsync -vpcrutxzE --ignore-existing\
>         --rsync-path="/usr/bin/rsync" root@$SRCDIR $DESTDIR
> -- 
> Please use reply-all for most replies to avoid omitting the mailing  
> list.
> To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

I have also used Mike Bombich' very clear instructions foud at

http://www.bombich.com/mactips/rsync.html
  in order  to compile the folllowing  version of rsync on OS10.4.11:
rsync  version 3.0.4  protocol version 30

I could not easily however find the associated manual to better  
understand some of Mike's suggested arguments such as:
-aNHAXx --protect-args --fileflags --force-change
Where can I find the meaning of them?

Thanks

rgds