rsync 3.1.1: --ignore-missing-args / --delete-missing args problem

Georgy Fedorov
Sat, 08 Apr 2017 14:10:53 -0000

Hi Alex,

Thank you very much indeed for that link. I was actually considering
using find -0 together with --files-from, and making find to list only
files that were changed some time ago -- but first, this would probably
break hardlinks, and second, despite most of the files in question are
temporary files, a user can still delete something old enough right in
the middle of an rsync.

However, looking at rsync problems that we have more closely, I now
realize that most of them are related to time setting issues -- part of
these similar to bug https://bugzilla.samba.org/show_bug.cgi?id=4977 and
the like. But I think these problems can and shall be addressed )

I will write more on this when I will finish testing.

Kind regards, George


On 08/04/17 00:01, Axel Kittenberger wrote:
> With this two options on a very live system you may need to take into
> account this bug as well I reported a while ago:
>
> https://bugzilla.samba.org/show_bug.cgi?id=12569
>
<https://protect-au.mimecast.com/s/5vYEBwtZLRbJcm?domain=bugzilla.samba.org>
>
> Due to this I'm currently not using  --ignore-missing-args /
> --delete-missing but rather -exclude=* -include=- and a autocreated
> hierachical filter rule list on stdin.
>
> PS: Yes to simply ignore code 24 would be correct. However, it creates
> exit code 2 in these cases.
>
> On Fri, Apr 7, 2017 at 2:58 PM, Georgy Fedorov via rsync
> <rsync at lists.samba.org <mailto:rsync at lists.samba.org>>
wrote:
>
>     Dear All,
>
>     We sometimes have to replicate large "live" filesystems with
many
>     ( sometimes millions, up to few hundred millions ) files on them.
>     ( Copying actively used files is of course a bad idea, but it
>     really helps to keep the delta small, so one final transfer can
>     later save the day. )
>
>     The problem, as one may guess, is that some files may disappear
>     during the process, so rsync finishes with an error code 23.
>     The good news is that since version 3.1.0 rsync has two options to
>     address this problem: --ignore-missing-args and
>     --delete-missing-args .
>
>     The bad news, however, is that even with any of these, when the
>     file disappears, rsync handles the file transfer properly, but
>     then tries to set times or attributes on these files nevertheless.
>     That fails with errno 22 ( EINVAL ) and still leads to exit code
>     23, which is a bit of an annoyance.
>
>     I am currently trying to come up with a patch to address this
>     issue, since I'd like to have exit code 0 for either of
>     --ignore-missing-args or --delete-missing-args, when the files
>     are, well, missing.
>
>     Probably it is not the right way to address this problem, but in
>     the same vein as --ignore-missing-args are implemented, the patch
>     can go as follows:
>     https://gist.github.com/anonymous/96ba8bf10f864a93fd9203f75c43ffe9
>
<https://gist.github.com/anonymous/96ba8bf10f864a93fd9203f75c43ffe9> .
>
>     Basically we need to do two changes:
>
>     (1) in options.c, make sure that the "missing_args" value is
>     transferred when set ; and
>     (2) in rsync.c, in the "missing_args" case, replace
FERROR_XFER
>     with something else, since apparently any log message with logcode
>     FERROR_XFER sets the flag 'got_xfer_error' in log.c, and that
>     finally leads to exit code RERR_PARTIAL (23), which is what we are
>     trying to avoid.
>
>     I am currently testing this on fairly big datasets to see if
>     there's something missing, and will write more when I see how it
>     goes ( as I said, the data sets are fairly large ).
>
>     There shall be a better way to address that, but this is all I can
>     do with a very shallow acquaintance with rsync source code.
>     Nevertheless, it would be great if that could be fixed in the
>     trunk one way or another )
>
>     Kind regards, George
>
>     --
>     George Fedorov
>     Senior Systems Specialist
>     Melbourne School of Engineering
>     The University of Melbourne, Victoria 3010, Australia
>     phone: *****
>     email: *****
>     http://www.eng.unimelb.edu.au/
>
>
>     --
>     Please use reply-all for most replies to avoid omitting the
>     mailing list.
>     To unsubscribe or change options:
>     https://lists.samba.org/mailman/listinfo/rsync
>
<https://protect-au.mimecast.com/s/gn98BxSeXmW9fJ?domain=lists.samba.org>
>     Before posting, read:
>     http://www.catb.org/~esr/faqs/smart-questions.html
>
<https://protect-au.mimecast.com/s/oDLrBgcVgWZDt8?domain=catb.org>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.samba.org/pipermail/rsync/attachments/20170409/a2948a6a/attachment.html>