rsync - Dec 2008 - weird interaction between --delete-delay and --partial-dir when transfer is interrupted

Hi,

I'm not sure if this is a bug, but after reading the manual, this does 
not seem like expected behavior.  I'm using the following rsync command 
to approximate an atomic update (I can't use --link-dest as hard links 
in hfsplus filesystems are fubar under linux as of 2.6.27):

rsync -a --delete-excluded --backup --backup-dir=../bar.archive 
--backup-suffix=.2008-12-15-0115 --delete-delay --delay-updates 
--partial-dir=.rsync-partial --verbose --human-readable 
--log-file=/bar.rsync-log host:/foo/ /bar

If a transfer is interrupted, .rsync-partial/ directories are left over 
in the backup directory, as expected. The next rsync run exits with 
non-zero exit status even though all files are transferred (verified 
this using md5sum). A third rsync run transfers nothing (going by the 
log file) but exits with success.  I'm guessing this is because: (1) 
--delete-excluded causes the .rsync-partial/ directories to be marked 
for deletion during file list generation time of the second rsync run 
(there are no .rsync-partial/ directories in host:/foo/), (2) successful 
transfer of all files during the second run results in rsync removing 
the .rsync-partial/ directories as they become empty, and (3) 
--delete-delay which I understand is necessary to allow rsync to use the 
files in .rsync-partial/ to speed up the second transfer, fails to find 
the .rsync-partial/ directories at the end of the transfer and 
complains.  The third transfer sees no .rsync-partial/ directories in 
/bar and so returns success.

I would like the second run to return success if all files are 
transferred.  Is there some option I'm supposed to use (e.g. --filter) 
or is this a bug?

Shiv

Shivkumar Venkatasubrahmanyam wrote:
> Hi,
> 
> I'm not sure if this is a bug, but after reading the manual, this does 
> not seem like expected behavior.  I'm using the following rsync command
> to approximate an atomic update (I can't use --link-dest as hard links 
> in hfsplus filesystems are fubar under linux as of 2.6.27):
> 
> rsync -a --delete-excluded --backup --backup-dir=../bar.archive 
> --backup-suffix=.2008-12-15-0115 --delete-delay --delay-updates 
> --partial-dir=.rsync-partial --verbose --human-readable 
> --log-file=/bar.rsync-log host:/foo/ /bar
> 
> If a transfer is interrupted, .rsync-partial/ directories are left over 
> in the backup directory, as expected. The next rsync run exits with 
> non-zero exit status even though all files are transferred (verified 
> this using md5sum). A third rsync run transfers nothing (going by the 
> log file) but exits with success.  I'm guessing this is because: (1) 
> --delete-excluded causes the .rsync-partial/ directories to be marked 
> for deletion during file list generation time of the second rsync run 
> (there are no .rsync-partial/ directories in host:/foo/), (2) successful 
> transfer of all files during the second run results in rsync removing 
> the .rsync-partial/ directories as they become empty, and (3) 
> --delete-delay which I understand is necessary to allow rsync to use the 
> files in .rsync-partial/ to speed up the second transfer, fails to find 
> the .rsync-partial/ directories at the end of the transfer and 
> complains.  The third transfer sees no .rsync-partial/ directories in 
> /bar and so returns success.
> 
> I would like the second run to return success if all files are 
> transferred.  Is there some option I'm supposed to use (e.g. --filter) 
> or is this a bug?
> 
> Shiv
What is the exact exit value after the 2nd run - 23 perhaps ?

You can do a simple test:

touch /test/src/a
mkdir /test/dst/.rp
rsync -a -vvvv --delete-excluded --delay-updates --delete-delay 
--partial-dir=.rp /test/src/ /test/dst/ | less

In this case, rsync will exit with value 23. It looks like rsync is 
deleting .rp before any file transfer starts.

If you put any bogus file in .rp before rsync call, it will exit with 0 
though. Looks like it prevents it from a deletion.

If you remove delete-excluded, it will work as expected with or w/o 
bogus file (with bogus file, .rp dir will not get deleted, otherwise it 
will), exit value is 0 in both cases.


It does look like sort of a tiny bug. Or maybe there's something subtle 
we're missing.


Either way - as a workaround, you could drop delete-excluded, and then 
just finalize the thing with something like: find /bar/ -depth -name 
".rsync-partial" -exec rm -rf "{}" \;

Rsync will usually clean up after itself - only bogus files (not being 
part of the transfer) would prevent it from doing so, as far as I can see.