rsync - Aug 2013 - Potential incompatibilities between '--delete' and --copy-unsafe-symlinks' ???

Hi,

New to this list, but long-time (appreciative) user of rsync.
Grateful for any help with my problem here...

In particular, I've been having long-standing issues (just now getting
around to trying to resolve them) when I use rsync with
'--copy-unsafe-links' alongside the '--delete' parameter.  If I
use
either of these two parameters in isolation (along with other shared
parameters), I get the expected behavior.  However, when I use
'--copy-unsafe-links', rsync no longer properly deletes files that are
present in the destination, but not in the source.

The usage scenario is rsync (v 3.0.9, stock from Debian Wheezy)
getting called by rsnapshot (v 1.3.1, also Wheezy).  Rsync source and
destination are both on the same host (nothing remote at all in this
entire scenario), though they happen to be on different partitions.

As typical of rsnapshot, the destination directory for rsync is also
hard-linked to other directories (past daily/etc snapshots), so a
typical "new" rsnapshot invocation on my system (to create the more
recent 'daily.0' snapshot) has the following two key steps:

  /bin/cp -al /backup/daily.0 /backup/daily.1 
  /usr/bin/rsync -avvv --delete --numeric-ids --relative --delete-excluded \
    --copy-unsafe-links --exclude=*/.*~ --exclude=*/*~ \
    --exclude=/home/lost+found  <... more excludes ...> \
    /home /backup/daily.0/<hostname>/ 

In this above case, files that were deleted on '/home' between the
'daily.1' backup and the 'daily.0' backup (the following day)
are
still incorrectly present in the new 'daily.0'.  However, if I remove
'--copy-unsafe-links' (through rsnapshot's appropriate configuration
file for passing parameters to rsync), then all functions as expected,
and file deletions on '/home' are properly mirrored in the updated
'daily.0' snapshot.

I've attempted to reproduce this in two ways:

  (1) I've called the above two commands (cp, then rsync) directly, on
      an appropriately prepared identical baseline of past rsnapshot
      'daily.<n>' directories.  In other words, this totally
removes
      rsnapshot from the test setup.  I see the same behavior as
      described above, so it doesn't appear to be an rsnapshot bug.

  (2) I also fashioned a vastly smaller dataset (only a few
      directories / files, rather than all of '/home'), again calling
      rsync directly, rather than through rsnapshot.  Alas, in this
      case, I can't seem to trigger the problem.  <sigh> This might be
      because my simpler tests didn't involve multiple partitions, etc.

At this point, my lack of deeper familiarity with rsync has become
apparent to me, so I thought I'd turn to this mailing list.

Any suggestions / pointers / answers would be wonderful...  Happy to
provide more info as necessary, but I hope I touched on the big points
here.

Thanks so much.. And thanks also for providing such a great tool!

Best,
Daniel

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

How about if you use rsync's --link-dest instead of cp -al?  Then no
- --delete is needed as superfluous hard links aren't created in the
first place.

IOW, instead of /bin/cp -al /backup/daily.0 /backup/daily.1
drop the --delete and --delete-exculded from the rsync param and add a
- --link-dest=/backup/daily.1

On 08/24/13 18:19, freedman at systems.cs.cornell.edu
wrote:
> Hi,
> 
> New to this list, but long-time (appreciative) user of rsync. 
> Grateful for any help with my problem here...
> 
> In particular, I've been having long-standing issues (just now
> getting around to trying to resolve them) when I use rsync with 
> '--copy-unsafe-links' alongside the '--delete' parameter. 
If I
> use either of these two parameters in isolation (along with other
> shared parameters), I get the expected behavior.  However, when I
> use '--copy-unsafe-links', rsync no longer properly deletes files
> that are present in the destination, but not in the source.
> 
> The usage scenario is rsync (v 3.0.9, stock from Debian Wheezy) 
> getting called by rsnapshot (v 1.3.1, also Wheezy).  Rsync source
> and destination are both on the same host (nothing remote at all in
> this entire scenario), though they happen to be on different
> partitions.
> 
> As typical of rsnapshot, the destination directory for rsync is
> also hard-linked to other directories (past daily/etc snapshots),
> so a typical "new" rsnapshot invocation on my system (to create
the
> more recent 'daily.0' snapshot) has the following two key steps:
> 
> /bin/cp -al /backup/daily.0 /backup/daily.1 /usr/bin/rsync -avvv
> --delete --numeric-ids --relative --delete-excluded \ 
> --copy-unsafe-links --exclude=*/.*~ --exclude=*/*~ \ 
> --exclude=/home/lost+found  <... more excludes ...> \ /home
> /backup/daily.0/<hostname>/
> 
> In this above case, files that were deleted on '/home' between the 
> 'daily.1' backup and the 'daily.0' backup (the following
day) are
> still incorrectly present in the new 'daily.0'.  However, if I
> remove '--copy-unsafe-links' (through rsnapshot's appropriate
> configuration file for passing parameters to rsync), then all
> functions as expected, and file deletions on '/home' are properly
> mirrored in the updated 'daily.0' snapshot.
> 
> I've attempted to reproduce this in two ways:
> 
> (1) I've called the above two commands (cp, then rsync) directly,
> on an appropriately prepared identical baseline of past rsnapshot 
> 'daily.<n>' directories.  In other words, this totally
removes
> rsnapshot from the test setup.  I see the same behavior as 
> described above, so it doesn't appear to be an rsnapshot bug.
> 
> (2) I also fashioned a vastly smaller dataset (only a few 
> directories / files, rather than all of '/home'), again calling 
> rsync directly, rather than through rsnapshot.  Alas, in this case,
> I can't seem to trigger the problem.  <sigh> This might be
because
> my simpler tests didn't involve multiple partitions, etc.
> 
> At this point, my lack of deeper familiarity with rsync has become 
> apparent to me, so I thought I'd turn to this mailing list.
> 
> Any suggestions / pointers / answers would be wonderful...  Happy
> to provide more info as necessary, but I hope I touched on the big
> points here.
> 
> Thanks so much.. And thanks also for providing such a great tool!
> 
> Best, Daniel
> 
- -- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
	Kevin Korb			Phone:    (407) 252-6853
	Systems Administrator		Internet:
	FutureQuest, Inc.		Kevin at FutureQuest.net  (work)
	Orlando, Florida		kmk at sanitarium.net (personal)
	Web page:			http://www.sanitarium.net/
	PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.20 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlIZRwwACgkQVKC1jlbQAQc+/wCbBM0+GJdVlJO/z3qKc9YcgQZ5
MLEAoNbbXR+eR2SE79tjusCgLjHY/d0d
=GmPO
-----END PGP SIGNATURE-----

On Sat, Aug 24, 2013 at 3:19 PM, <freedman at systems.cs.cornell.edu>
wrote:
> In particular, I've been having long-standing issues (just now
> getting around to trying to resolve them) when I use rsync
> with '--copy-unsafe-links' alongside the '--delete'
parameter.  If I
> use either of these two parameters in isolation (along with other
> shared parameters), I get the expected behavior.  However, when I
> use '--copy-unsafe-links', rsync no longer properly deletes files
that
> are present in the destination, but not in the source.
>
Be sure to check the errors from the rsync run.  I'd imagine that one of
your unsafe symlinks is bogus, and you will be getting a pair of errors:


symlink has no referent: "/some/path"
IO error encountered -- skipping file deletion

Because a bogus dereferenced symlink can't be replaced with a file, it is
left out of the transfer. This would cause the receiving side to delete
that missing file (symlink) in the destination directory if deletions were
allowed to continue.  Rsync currently prevents that from happening by
turning off file deletions.

..wayne..
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.samba.org/pipermail/rsync/attachments/20130825/344b4956/attachment.html>