rsync - Mar 2008 - rsync: Please add an option to ignore vanished files

Hi Everyone,

The following request was received from a Debian user; could you
comment on it?  I think this seems reasonable, although the cron
output can be fixed by writing to a temporary file and using grep -v.
(The temporary file is necessary to avoid discarding the exit code,
 although that may not be needed in the case of rsync, since it writes
 that to stderr).

On Mon, Feb 20, 2006 at 02:28:50PM +0300, Cyril Bouthors
wrote:
> Package: rsync
> Version: 2.6.6-1
> Severity: wishlist
> 
> I make nightly backups of 1TB+ of data that are continuously
> read-write accessed and I constantly have *huge* output of the cron
> job because of vanished files:
> 
> file has vanished:
"/drbd/var/www/tmp/sess_003323885e425f2a67e69e0ac4c24124"
> file has vanished:
"/drbd/var/www/tmp/sess_033cc2efa0b325a2d154c082995d216b"
> file has vanished:
"/drbd/var/www/tmp/sess_11460e1bd071bc56e385bd36d00b55c5"
> file has vanished:
"/drbd/var/www/tmp/sess_182e6dfcc0373bef667ed0f902aa93d5"
> file has vanished:
"/drbd/var/www/tmp/sess_1d274c8d694cbdd819c65ae1e5d15b91"
> file has vanished:
"/drbd/var/www/tmp/sess_1e77d2fd4d81da9a8614cd10347c08b5"
> file has vanished:
"/drbd/var/www/tmp/sess_1f23f36129d192a8522842aa33bca2eb"
> file has vanished:
"/drbd/var/www/tmp/sess_23c5d3f51ea5a02ee981c180c84d0bd0"
> file has vanished:
"/drbd/var/www/tmp/sess_245feab1eba25a857108e6571d3405e6"
> file has vanished:
"/drbd/var/www/tmp/sess_3172de12d16979a863d1fda04545ca89"
> file has vanished:
"/drbd/var/www/tmp/sess_405b9d38341b955afcbae9519678a319"
> file has vanished:
"/drbd/var/www/tmp/sess_418f14e27f10725506b3100ca0a8e643"
> file has vanished:
"/drbd/var/www/tmp/sess_4bf5cfe9dc12c7480f016dc1509fbfa7"
> file has vanished:
"/drbd/var/www/tmp/sess_4e0a051348d9f11608f6aafa07b2c401"
> file has vanished:
"/drbd/var/www/tmp/sess_4f070d18b9d7992f40d997a1e1a19bae"
> file has vanished:
"/drbd/var/www/tmp/sess_53a52b6406ccbf36d3abad6188bb3add"
> file has vanished:
"/drbd/var/www/tmp/sess_56a724e0d24e8a217acfc3bc0b039745"
> file has vanished:
"/drbd/var/www/tmp/sess_56e60ad1a0908df5617adf04147247e0"
> (...)
> [skipped tens of thousands of lines]
> 
> I'd like rsync to provide an option called '--ignore-vanished'
that
> would not display those warnings anymore and not exit with value 24
> ("Partial transfer due to vanished source files" according to the
> manpage).
> 
> Thanks.
> 
> -- System Information:
> Debian Release: testing/unstable
> Architecture: i386 (i686)
> Shell:  /bin/sh linked to /bin/bash
> Kernel: Linux 2.6.15-1-686
> Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
> 
> Versions of packages rsync depends on:
> ii  libc6                         2.3.6-1    GNU C Library: Shared
libraries an
> ii  libpopt0                      1.7-5      lib for parsing cmdline
parameters
> 
> rsync recommends no packages.
> 
> -- no debconf information
> 
> -- 
> Cyril Bouthors

Having not yet seen a response to this, here's my $.02...

On Thu, 6 Mar 2008, Justin Pryzby <justinpryzby@users.sourceforge.net>
wrote:
> Hi Everyone,
> 
> The following request was received from a Debian user; could you
> comment on it?  I think this seems reasonable, although the cron
> output can be fixed by writing to a temporary file and using grep -v.
> (The temporary file is necessary to avoid discarding the exit code,
>  although that may not be needed in the case of rsync, since it writes
>  that to stderr).
> 
> On Mon, Feb 20, 2006 at 02:28:50PM +0300, Cyril Bouthors wrote:
>> Package: rsync
>> Version: 2.6.6-1
>> Severity: wishlist
>> 
>> I make nightly backups of 1TB+ of data that are continuously
>> read-write accessed and I constantly have *huge* output of the cron
>> job because of vanished files:
>> 
>> file has vanished:
"/drbd/var/www/tmp/sess_003323885e425f2a67e69e0ac4c24124"
>> (...)
>> [skipped tens of thousands of lines]
>> 
>> I'd like rsync to provide an option called
'--ignore-vanished' that
>> would not display those warnings anymore and not exit with value 24
>> ("Partial transfer due to vanished source files" according to
the
>> manpage).

Instead of attacking the symptom, attack the problem.  You are backing
up temporary files that are not needed on a restore.  These "vanished"
messages actually help pinpoint directories whose contents should be
excluded, in this case via --exclude=/drbd/var/www/tmp/*

Also, if you want, you can also check the exit code within a script and
treat value 24 the same as a 0 (e.g. no extra warnings about failure).

    John Van Essen

On Thu, 2008-03-06 at 11:56 -0500, Justin Pryzby wrote:
> On Mon, Feb 20, 2006 at 02:28:50PM +0300, Cyril Bouthors wrote:
> > I'd like rsync to provide an option called
'--ignore-vanished' that
> > would not display those warnings anymore and not exit with value 24
> > ("Partial transfer due to vanished source files" according
to the
> > manpage).
See:

https://bugzilla.samba.org/show_bug.cgi?id=3653

Andreas Kotes had exactly the same proposal, but I'm thinking that
instead of adding an option to suppress a warning that in one case can
indicate an incorrect transfer, it would be better to improve the design
of rsync so the warning can no longer happen.

Matt