Bob McKay
2009-Feb-12 00:59 UTC
Macintosh rsync: Inconsistent Behaviour: Hints Appreciated
I have a problem with rsync on macintosh copying files that have not been
updated. Despite careful reading of the archives, I can't find an
explanation (or a fix), but I strongly suspect I'm just missing something
that an expert will recognise instantly... Your assistance would be greatly
appreciated.
I'm running a nightly backup of my photo/home video directory from a
partition BIGPIX on a firewire drive on one machine (G3 running OSX 10.3,
imaginatively named g3) to another (Pix) on a firewire drive on my laptop
(G4 running 10.4, even more imaginatively bobsmac). I'll include the script
below. All are HFS+ partitions.
The problem is, that every night, something like 3/4 of the whole directory
gets updated. This doesn't reflect reality - in fact, I only add stuff to
the directory once a week or so. If the whole of the directory was getting
updated, I'd understand that I just had the options wrong, but it's
completely inconsistent - some files get re-synched night after night, while
others are just synched once and then rsync does what I expected and leaves
them alone. I can't see any consistency in this. There is some correlation
with age (the older directories are much more likely to be re-synched than
newer ones). But this correlation is not reliable. I'll use an example of a
particular directory from May 2008. Of five files in the directory, two
(creation dates Jan 7 2008) are synched every night. The other three are
omitted. One has creation date Nov. 15 2007, the other two April 14 2008.
Other than age, I can't see any difference between the files, either from
unix shell (see below) or from finder info (I'd include this if I could
figure a way to cut and paste, but please accept that it is completely
consistent with the shell results). The rsync options I'm using are
'-vpcrutxzE --ignore-existing'; I've read the man pages carefully,
and I
can't understand why they would produce this behaviour.
If you can see any way I can further diagnose this, any hints would be
appreciated - I'm completely stuck. Of course, if you know why it's
happening - or know a fix - that would be even better!
Thanks and Best Wishes
Bob
*********************** directory listings of example source and target
directories ********************
G3:/Volumes/BIGPIX/PIXREADONLY/200805-FriendsMisc080502 rim$ ls -l
total 12472
-rwxr-xr-x 1 root admin 1827015 Apr 14 2008 HannahP1060453.jpg
-rwxr-xr-x 1 root admin 2219607 Apr 14 2008 HannahP1060578.jpg
-rwxr-xr-x 1 root admin 192966 Jan 7 2008 VikyP1030042.jpg
-rwxr-xr-x 1 root admin 206962 Jan 7 2008 VivP1030020.jpg
-rwxr-xr-x 1 root admin 1900455 Nov 15 2007 WOOK-DSCN3830.jpg
bobsmac:/Volumes/Pix/PIXBACKUP/PIXREADONLY/200805-FriendsMisc080502 rim$ ls
-l
total 12472
-rwxr-xr-x 1 rim rim 1827015 Apr 14 2008 HannahP1060453.jpg
-rwxr-xr-x 1 rim rim 2219607 Apr 14 2008 HannahP1060578.jpg
-rwxr-xr-x 1 rim rim 192966 Jan 7 2008 VikyP1030042.jpg
-rwxr-xr-x 1 rim rim 206962 Jan 7 2008 VivP1030020.jpg
-rwxr-xr-x 1 rim rim 1900455 Nov 15 2007 WOOK-DSCN3830.jpg
(Viky and Viv get synched repeatedly, the rest do not)
*********************** abbreviated transcript showing relevant stages of
copying ********************
Copying g3.local:/Volumes/BIGPIX/PIXREADONLY /Volumes/Pix/PIXBACKUP
receiving file list ... done
PIXREADONLY/._.DS_Store
PIXREADONLY/200202-China/Wuhan Early Pics/._PIC_6250.JPG
.
.
.
PIXREADONLY/200805-FriendsMisc080502/._.DS_Store
PIXREADONLY/200805-FriendsMisc080502/._VikyP1030042.jpg
PIXREADONLY/200805-FriendsMisc080502/._VivP1030020.jpg
PIXREADONLY/200805-KaesongNK080511/._.DS_Store
.
.
.
sent 59156 bytes received 115825818 bytes 48599.28 bytes/sec
total size is 22910676123 speedup is 197.70
*********************** shell script ********************
#!/bin/sh
SRCDIR="g3.local:/Volumes/BIGPIX/PIXREADONLY"
DESTDIR="/Volumes/Pix/PIXBACKUP"
echo "Copying "$SRCDIR" "$DESTDIR
rsync -vpcrutxzE --ignore-existing\
--rsync-path="/usr/bin/rsync" root@$SRCDIR $DESTDIR
-------------- next part --------------
HTML attachment scrubbed and removed
Mike Bombich
2009-Feb-12 02:50 UTC
Macintosh rsync: Inconsistent Behaviour: Hints Appreciated
You're using the version of rsync that is built-in to Mac OS X. Don't do that. It's a fairly old version (though it is maintained by Apple), and it doesn't handle extended attribute modifications (or lack thereof), thus it transfers them every time. When you have many files with resource forks or other extended attributes, you'll see files getting copied every time regardless of whether they've been modified. rsync 3.0.x handles these much more efficiently. I posted some tips for compiling and running rsync on Mac OS X here: http://www.bombich.com/mactips/rsync.html Note that the arguments will be slightly different, so be sure to review the man page and modify your script accordingly. Also be sure to call rsync from /usr/local/bin/rsync (or wherever you choose to install it), otherwise you'll still be using the built-in version. Mike On Feb 11, 2009, at 6:59 PM, Bob McKay wrote:> I have a problem with rsync on macintosh copying files that have not > been updated. Despite careful reading of the archives, I can't find > an explanation (or a fix), but I strongly suspect I'm just missing > something that an expert will recognise instantly... Your assistance > would be greatly appreciated. > > I'm running a nightly backup of my photo/home video directory from a > partition BIGPIX on a firewire drive on one machine (G3 running OSX > 10.3, imaginatively named g3) to another (Pix) on a firewire drive > on my laptop (G4 running 10.4, even more imaginatively bobsmac). > I'll include the script below. All are HFS+ partitions. > > The problem is, that every night, something like 3/4 of the whole > directory gets updated. This doesn't reflect reality - in fact, I > only add stuff to the directory once a week or so. If the whole of > the directory was getting updated, I'd understand that I just had > the options wrong, but it's completely inconsistent - some files get > re-synched night after night, while others are just synched once and > then rsync does what I expected and leaves them alone. I can't see > any consistency in this. There is some correlation with age (the > older directories are much more likely to be re-synched than newer > ones). But this correlation is not reliable. I'll use an example of > a particular directory from May 2008. Of five files in the > directory, two (creation dates Jan 7 2008) are synched every night. > The other three are omitted. One has creation date Nov. 15 2007, the > other two April 14 2008. Other than age, I can't see any difference > between the files, either from unix shell (see below) or from finder > info (I'd include this if I could figure a way to cut and paste, but > please accept that it is completely consistent with the shell > results). The rsync options I'm using are '-vpcrutxzE --ignore- > existing'; I've read the man pages carefully, and I can't understand > why they would produce this behaviour. > > If you can see any way I can further diagnose this, any hints would > be appreciated - I'm completely stuck. Of course, if you know why > it's happening - or know a fix - that would be even better! > > Thanks and Best Wishes > Bob > > *********************** directory listings of example source and > target directories ******************** > > G3:/Volumes/BIGPIX/PIXREADONLY/200805-FriendsMisc080502 rim$ ls -l > total 12472 > -rwxr-xr-x 1 root admin 1827015 Apr 14 2008 HannahP1060453.jpg > -rwxr-xr-x 1 root admin 2219607 Apr 14 2008 HannahP1060578.jpg > -rwxr-xr-x 1 root admin 192966 Jan 7 2008 VikyP1030042.jpg > -rwxr-xr-x 1 root admin 206962 Jan 7 2008 VivP1030020.jpg > -rwxr-xr-x 1 root admin 1900455 Nov 15 2007 WOOK-DSCN3830.jpg > > bobsmac:/Volumes/Pix/PIXBACKUP/PIXREADONLY/200805-FriendsMisc080502 > rim$ ls -l > total 12472 > -rwxr-xr-x 1 rim rim 1827015 Apr 14 2008 HannahP1060453.jpg > -rwxr-xr-x 1 rim rim 2219607 Apr 14 2008 HannahP1060578.jpg > -rwxr-xr-x 1 rim rim 192966 Jan 7 2008 VikyP1030042.jpg > -rwxr-xr-x 1 rim rim 206962 Jan 7 2008 VivP1030020.jpg > -rwxr-xr-x 1 rim rim 1900455 Nov 15 2007 WOOK-DSCN3830.jpg > > (Viky and Viv get synched repeatedly, the rest do not) > > > *********************** abbreviated transcript showing relevant > stages of copying ******************** > > Copying g3.local:/Volumes/BIGPIX/PIXREADONLY /Volumes/Pix/PIXBACKUP > receiving file list ... done > PIXREADONLY/._.DS_Store > PIXREADONLY/200202-China/Wuhan Early Pics/._PIC_6250.JPG > . > . > . > PIXREADONLY/200805-FriendsMisc080502/._.DS_Store > PIXREADONLY/200805-FriendsMisc080502/._VikyP1030042.jpg > PIXREADONLY/200805-FriendsMisc080502/._VivP1030020.jpg > PIXREADONLY/200805-KaesongNK080511/._.DS_Store > . > . > . > > sent 59156 bytes received 115825818 bytes 48599.28 bytes/sec > total size is 22910676123 speedup is 197.70 > > > *********************** shell script ******************** > > #!/bin/sh > > SRCDIR="g3.local:/Volumes/BIGPIX/PIXREADONLY" > DESTDIR="/Volumes/Pix/PIXBACKUP" > echo "Copying "$SRCDIR" "$DESTDIR > rsync -vpcrutxzE --ignore-existing\ > --rsync-path="/usr/bin/rsync" root@$SRCDIR $DESTDIR > -- > Please use reply-all for most replies to avoid omitting the mailing > list. > To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Yves Petronin
2009-Feb-13 18:08 UTC
Macintosh rsync: Inconsistent Behaviour: Hints Appreciated
I have also used Mike Bombich' very clear instructions foud at http://www.bombich.com/mactips/rsync.html in order to compile the folllowing version of rsync on OS10.4.11: rsync version 3.0.4 protocol version 30 I could not easily however find the associated manual to better understand some of Mike's suggested arguments such as: -aNHAXx --protect-args --fileflags --force-change Where can I find the meaning of them? Thanks rgds