rsync - Apr 2017 - rsync 3.1.1: --ignore-missing-args / --delete-missing args problem

Dear All,

We sometimes have to replicate large "live" filesystems with many ( 
sometimes millions, up to few hundred millions ) files on them. ( 
Copying actively used files is of course a bad idea, but it really helps 
to keep the delta small, so one final transfer can later save the day. )

The problem, as one may guess, is that some files may disappear during 
the process, so rsync finishes with an error code 23.
The good news is that since version 3.1.0 rsync has two options to 
address this problem: --ignore-missing-args and --delete-missing-args .

The bad news, however, is that even with any of these, when the file 
disappears, rsync handles the file transfer properly, but then tries to 
set times or attributes on these files nevertheless. That fails with 
errno 22 ( EINVAL ) and still leads to exit code 23, which is a bit of 
an annoyance.

I am currently trying to come up with a patch to address this issue, 
since I'd like to have exit code 0 for either of --ignore-missing-args 
or --delete-missing-args, when the files are, well, missing.

Probably it is not the right way to address this problem, but in the 
same vein as --ignore-missing-args are implemented, the patch can go as 
follows:
https://gist.github.com/anonymous/96ba8bf10f864a93fd9203f75c43ffe9 .

Basically we need to do two changes:

(1) in options.c, make sure that the "missing_args" value is
transferred
when set ; and
(2) in rsync.c, in the "missing_args" case, replace FERROR_XFER with 
something else, since apparently any log message with logcode 
FERROR_XFER sets the flag 'got_xfer_error' in log.c, and that finally 
leads to exit code RERR_PARTIAL (23), which is what we are trying to avoid.

I am currently testing this on fairly big datasets to see if there's 
something missing, and will write more when I see how it goes ( as I 
said, the data sets are fairly large ).

There shall be a better way to address that, but this is all I can do 
with a very shallow acquaintance with rsync source code. Nevertheless, 
it would be great if that could be fixed in the trunk one way or another )

Kind regards, George

-- 
George Fedorov
Senior Systems Specialist
Melbourne School of Engineering
The University of Melbourne, Victoria 3010, Australia
phone: *****
email: *****
http://www.eng.unimelb.edu.au/

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.samba.org/pipermail/rsync/attachments/20170407/d9d70f05/attachment.html>

Those options are for handling files/dirs that are specified on the
actual command line they have nothing to do with files vanishing while
rsync is running.  The correct solution is to simply ignore exit code 24.

BTW, this is what filesystem snapshots are for.

On 04/07/2017 08:58 AM, Georgy Fedorov via rsync wrote:
> Dear All,
> 
> We sometimes have to replicate large "live" filesystems with many
(
> sometimes millions, up to few hundred millions ) files on them. (
> Copying actively used files is of course a bad idea, but it really helps
> to keep the delta small, so one final transfer can later save the day. )
> 
> The problem, as one may guess, is that some files may disappear during
> the process, so rsync finishes with an error code 23.
> The good news is that since version 3.1.0 rsync has two options to
> address this problem: --ignore-missing-args and --delete-missing-args .
> 
> The bad news, however, is that even with any of these, when the file
> disappears, rsync handles the file transfer properly, but then tries to
> set times or attributes on these files nevertheless. That fails with
> errno 22 ( EINVAL ) and still leads to exit code 23, which is a bit of
> an annoyance.
> 
> I am currently trying to come up with a patch to address this issue,
> since I'd like to have exit code 0 for either of --ignore-missing-args
> or --delete-missing-args, when the files are, well, missing.
> 
> Probably it is not the right way to address this problem, but in the
> same vein as --ignore-missing-args are implemented, the patch can go as
> follows:
> https://gist.github.com/anonymous/96ba8bf10f864a93fd9203f75c43ffe9 .
> 
> Basically we need to do two changes:
> 
> (1) in options.c, make sure that the "missing_args" value is
transferred
> when set ; and
> (2) in rsync.c, in the "missing_args" case, replace FERROR_XFER
with
> something else, since apparently any log message with logcode
> FERROR_XFER sets the flag 'got_xfer_error' in log.c, and that
finally
> leads to exit code RERR_PARTIAL (23), which is what we are trying to avoid.
> 
> I am currently testing this on fairly big datasets to see if there's
> something missing, and will write more when I see how it goes ( as I
> said, the data sets are fairly large ).
> 
> There shall be a better way to address that, but this is all I can do
> with a very shallow acquaintance with rsync source code. Nevertheless,
> it would be great if that could be fixed in the trunk one way or another )
> 
> Kind regards, George
> 
> -- 
> George Fedorov
> Senior Systems Specialist
> Melbourne School of Engineering
> The University of Melbourne, Victoria 3010, Australia
> phone: *****
> email: *****
> http://www.eng.unimelb.edu.au/
> 
> 
> 
-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
	Kevin Korb			Phone:    (407) 252-6853
	Systems Administrator		Internet:
	FutureQuest, Inc.		Kevin at FutureQuest.net  (work)
	Orlando, Florida		kmk at sanitarium.net (personal)
	Web page:			http://www.sanitarium.net/
	PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 224 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.samba.org/pipermail/rsync/attachments/20170407/702a2199/signature.sig>

With this two options on a very live system you may need to take into
account this bug as well I reported a while ago:

https://bugzilla.samba.org/show_bug.cgi?id=12569

Due to this I'm currently not using  --ignore-missing-args /
--delete-missing but rather -exclude=* -include=- and a autocreated
hierachical filter rule list on stdin.

PS: Yes to simply ignore code 24 would be correct. However, it creates exit
code 2 in these cases.

On Fri, Apr 7, 2017 at 2:58 PM, Georgy Fedorov via rsync <
rsync at lists.samba.org> wrote:
> Dear All,
>
> We sometimes have to replicate large "live" filesystems with many
(
> sometimes millions, up to few hundred millions ) files on them. ( Copying
> actively used files is of course a bad idea, but it really helps to keep
> the delta small, so one final transfer can later save the day. )
>
> The problem, as one may guess, is that some files may disappear during the
> process, so rsync finishes with an error code 23.
> The good news is that since version 3.1.0 rsync has two options to address
> this problem: --ignore-missing-args and --delete-missing-args .
>
> The bad news, however, is that even with any of these, when the file
> disappears, rsync handles the file transfer properly, but then tries to set
> times or attributes on these files nevertheless. That fails with errno 22 (
> EINVAL ) and still leads to exit code 23, which is a bit of an annoyance.
>
> I am currently trying to come up with a patch to address this issue, since
> I'd like to have exit code 0 for either of --ignore-missing-args or
> --delete-missing-args, when the files are, well, missing.
>
> Probably it is not the right way to address this problem, but in the same
> vein as --ignore-missing-args are implemented, the patch can go as follows:
> https://gist.github.com/anonymous/96ba8bf10f864a93fd9203f75c43ffe9 .
>
> Basically we need to do two changes:
>
> (1) in options.c, make sure that the "missing_args" value is
transferred
> when set ; and
> (2) in rsync.c, in the "missing_args" case, replace FERROR_XFER
with
> something else, since apparently any log message with logcode FERROR_XFER
> sets the flag 'got_xfer_error' in log.c, and that finally leads to
exit
> code RERR_PARTIAL (23), which is what we are trying to avoid.
>
> I am currently testing this on fairly big datasets to see if there's
> something missing, and will write more when I see how it goes ( as I said,
> the data sets are fairly large ).
>
> There shall be a better way to address that, but this is all I can do with
> a very shallow acquaintance with rsync source code. Nevertheless, it would
> be great if that could be fixed in the trunk one way or another )
>
> Kind regards, George
>
> --
> George Fedorov
> Senior Systems Specialist
> Melbourne School of Engineering
> The University of Melbourne, Victoria 3010, Australia
> phone: *****
> email: *****http://www.eng.unimelb.edu.au/
>
>
> --
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: https://lists.samba.org/
> mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.samba.org/pipermail/rsync/attachments/20170407/848a8e16/attachment.html>

Exit code 2 is "Protocol incompatibility".

Also, sounds like what you really want is --files-from

On 04/07/2017 10:01 AM, Axel Kittenberger via rsync
wrote:
> With this two options on a very live system you may need to take into
> account this bug as well I reported a while ago:
> 
> https://bugzilla.samba.org/show_bug.cgi?id=12569
> 
> Due to this I'm currently not using  --ignore-missing-args /
> --delete-missing but rather -exclude=* -include=- and a autocreated
> hierachical filter rule list on stdin.
> 
> PS: Yes to simply ignore code 24 would be correct. However, it creates
> exit code 2 in these cases.
> 
> On Fri, Apr 7, 2017 at 2:58 PM, Georgy Fedorov via rsync
> <rsync at lists.samba.org <mailto:rsync at lists.samba.org>>
wrote:
> 
>     Dear All,
> 
>     We sometimes have to replicate large "live" filesystems with
many (
>     sometimes millions, up to few hundred millions ) files on them. (
>     Copying actively used files is of course a bad idea, but it really
>     helps to keep the delta small, so one final transfer can later save
>     the day. )
> 
>     The problem, as one may guess, is that some files may disappear
>     during the process, so rsync finishes with an error code 23.
>     The good news is that since version 3.1.0 rsync has two options to
>     address this problem: --ignore-missing-args and --delete-missing-args .
> 
>     The bad news, however, is that even with any of these, when the file
>     disappears, rsync handles the file transfer properly, but then tries
>     to set times or attributes on these files nevertheless. That fails
>     with errno 22 ( EINVAL ) and still leads to exit code 23, which is a
>     bit of an annoyance.
> 
>     I am currently trying to come up with a patch to address this issue,
>     since I'd like to have exit code 0 for either of
>     --ignore-missing-args or --delete-missing-args, when the files are,
>     well, missing.
> 
>     Probably it is not the right way to address this problem, but in the
>     same vein as --ignore-missing-args are implemented, the patch can go
>     as follows:
>     https://gist.github.com/anonymous/96ba8bf10f864a93fd9203f75c43ffe9
>    
<https://gist.github.com/anonymous/96ba8bf10f864a93fd9203f75c43ffe9> .
> 
>     Basically we need to do two changes:
> 
>     (1) in options.c, make sure that the "missing_args" value is
>     transferred when set ; and
>     (2) in rsync.c, in the "missing_args" case, replace
FERROR_XFER with
>     something else, since apparently any log message with logcode
>     FERROR_XFER sets the flag 'got_xfer_error' in log.c, and that
>     finally leads to exit code RERR_PARTIAL (23), which is what we are
>     trying to avoid.
> 
>     I am currently testing this on fairly big datasets to see if
there's
>     something missing, and will write more when I see how it goes ( as I
>     said, the data sets are fairly large ).
> 
>     There shall be a better way to address that, but this is all I can
>     do with a very shallow acquaintance with rsync source code.
>     Nevertheless, it would be great if that could be fixed in the trunk
>     one way or another )
> 
>     Kind regards, George
> 
>     -- 
>     George Fedorov
>     Senior Systems Specialist
>     Melbourne School of Engineering
>     The University of Melbourne, Victoria 3010, Australia
>     phone: *****
>     email: *****
>     http://www.eng.unimelb.edu.au/
> 
> 
>     --
>     Please use reply-all for most replies to avoid omitting the mailing
>     list.
>     To unsubscribe or change options:
>     https://lists.samba.org/mailman/listinfo/rsync
>     <https://lists.samba.org/mailman/listinfo/rsync>
>     Before posting, read:
>     http://www.catb.org/~esr/faqs/smart-questions.html
>     <http://www.catb.org/~esr/faqs/smart-questions.html>
> 
> 
> 
> 
-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
	Kevin Korb			Phone:    (407) 252-6853
	Systems Administrator		Internet:
	FutureQuest, Inc.		Kevin at FutureQuest.net  (work)
	Orlando, Florida		kmk at sanitarium.net (personal)
	Web page:			http://www.sanitarium.net/
	PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 224 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.samba.org/pipermail/rsync/attachments/20170407/422998da/signature.sig>

Hi Alex,

Thank you very much indeed for that link. I was actually considering 
using find -0 together with --files-from, and making find to list only 
files that were changed some time ago -- but first, this would probably 
break hardlinks, and second, despite most of the files in question are 
temporary files, a user can still delete something old enough right in 
the middle of an rsync.

However, looking at rsync problems that we have more closely, I now 
realize that most of them are related to time setting issues -- part of 
these similar to bug https://bugzilla.samba.org/show_bug.cgi?id=4977 and 
the like. But I think these problems can and shall be addressed )

I will write more on this when I will finish testing.

Kind regards, George


On 08/04/17 00:01, Axel Kittenberger wrote:
> With this two options on a very live system you may need to take into 
> account this bug as well I reported a while ago:
>
> https://bugzilla.samba.org/show_bug.cgi?id=12569 
>
<https://protect-au.mimecast.com/s/5vYEBwtZLRbJcm?domain=bugzilla.samba.org>
>
> Due to this I'm currently not using  --ignore-missing-args / 
> --delete-missing but rather -exclude=* -include=- and a autocreated 
> hierachical filter rule list on stdin.
>
> PS: Yes to simply ignore code 24 would be correct. However, it creates 
> exit code 2 in these cases.
>
> On Fri, Apr 7, 2017 at 2:58 PM, Georgy Fedorov via rsync 
> <rsync at lists.samba.org <mailto:rsync at lists.samba.org>>
wrote:
>
>     Dear All,
>
>     We sometimes have to replicate large "live" filesystems with
many
>     ( sometimes millions, up to few hundred millions ) files on them.
>     ( Copying actively used files is of course a bad idea, but it
>     really helps to keep the delta small, so one final transfer can
>     later save the day. )
>
>     The problem, as one may guess, is that some files may disappear
>     during the process, so rsync finishes with an error code 23.
>     The good news is that since version 3.1.0 rsync has two options to
>     address this problem: --ignore-missing-args and
>     --delete-missing-args .
>
>     The bad news, however, is that even with any of these, when the
>     file disappears, rsync handles the file transfer properly, but
>     then tries to set times or attributes on these files nevertheless.
>     That fails with errno 22 ( EINVAL ) and still leads to exit code
>     23, which is a bit of an annoyance.
>
>     I am currently trying to come up with a patch to address this
>     issue, since I'd like to have exit code 0 for either of
>     --ignore-missing-args or --delete-missing-args, when the files
>     are, well, missing.
>
>     Probably it is not the right way to address this problem, but in
>     the same vein as --ignore-missing-args are implemented, the patch
>     can go as follows:
>     https://gist.github.com/anonymous/96ba8bf10f864a93fd9203f75c43ffe9
>    
<https://gist.github.com/anonymous/96ba8bf10f864a93fd9203f75c43ffe9> .
>
>     Basically we need to do two changes:
>
>     (1) in options.c, make sure that the "missing_args" value is
>     transferred when set ; and
>     (2) in rsync.c, in the "missing_args" case, replace
FERROR_XFER
>     with something else, since apparently any log message with logcode
>     FERROR_XFER sets the flag 'got_xfer_error' in log.c, and that
>     finally leads to exit code RERR_PARTIAL (23), which is what we are
>     trying to avoid.
>
>     I am currently testing this on fairly big datasets to see if
>     there's something missing, and will write more when I see how it
>     goes ( as I said, the data sets are fairly large ).
>
>     There shall be a better way to address that, but this is all I can
>     do with a very shallow acquaintance with rsync source code.
>     Nevertheless, it would be great if that could be fixed in the
>     trunk one way or another )
>
>     Kind regards, George
>
>     -- 
>     George Fedorov
>     Senior Systems Specialist
>     Melbourne School of Engineering
>     The University of Melbourne, Victoria 3010, Australia
>     phone: *****
>     email: *****
>     http://www.eng.unimelb.edu.au/
>
>
>     --
>     Please use reply-all for most replies to avoid omitting the
>     mailing list.
>     To unsubscribe or change options:
>     https://lists.samba.org/mailman/listinfo/rsync
>    
<https://protect-au.mimecast.com/s/gn98BxSeXmW9fJ?domain=lists.samba.org>
>     Before posting, read:
>     http://www.catb.org/~esr/faqs/smart-questions.html
>    
<https://protect-au.mimecast.com/s/oDLrBgcVgWZDt8?domain=catb.org>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.samba.org/pipermail/rsync/attachments/20170409/a2948a6a/attachment.html>

It is not code 24. RERR_PARTIAL is code 23, which is a bit dangerous to 
ignore.
I may have to fall back to snapshots, but for hierarchical filesystems 
(ZFS in our case) one would have to traverse all .zfs subfolders instead 
of a simple rsync without an -x . This is a unnecessary code 
complication that I was originally trying to avoid.

On 07/04/17 23:12, Kevin Korb via rsync wrote:
> Those options are for handling files/dirs that are specified on the
> actual command line they have nothing to do with files vanishing while
> rsync is running.  The correct solution is to simply ignore exit code 24.
>
> BTW, this is what filesystem snapshots are for.
>
> On 04/07/2017 08:58 AM, Georgy Fedorov via rsync wrote:
>> Dear All,
>>
>> We sometimes have to replicate large "live" filesystems with
many (
>> sometimes millions, up to few hundred millions ) files on them. (
>> Copying actively used files is of course a bad idea, but it really
helps
>> to keep the delta small, so one final transfer can later save the day.
)
>>
>> The problem, as one may guess, is that some files may disappear during
>> the process, so rsync finishes with an error code 23.
>> The good news is that since version 3.1.0 rsync has two options to
>> address this problem: --ignore-missing-args and --delete-missing-args .
>>
>> The bad news, however, is that even with any of these, when the file
>> disappears, rsync handles the file transfer properly, but then tries to
>> set times or attributes on these files nevertheless. That fails with
>> errno 22 ( EINVAL ) and still leads to exit code 23, which is a bit of
>> an annoyance.
>>
>> I am currently trying to come up with a patch to address this issue,
>> since I'd like to have exit code 0 for either of
--ignore-missing-args
>> or --delete-missing-args, when the files are, well, missing.
>>
>> Probably it is not the right way to address this problem, but in the
>> same vein as --ignore-missing-args are implemented, the patch can go as
>> follows:
>> https://gist.github.com/anonymous/96ba8bf10f864a93fd9203f75c43ffe9 .
>>
>> Basically we need to do two changes:
>>
>> (1) in options.c, make sure that the "missing_args" value is
transferred
>> when set ; and
>> (2) in rsync.c, in the "missing_args" case, replace
FERROR_XFER with
>> something else, since apparently any log message with logcode
>> FERROR_XFER sets the flag 'got_xfer_error' in log.c, and that
finally
>> leads to exit code RERR_PARTIAL (23), which is what we are trying to
avoid.
>>
>> I am currently testing this on fairly big datasets to see if
there's
>> something missing, and will write more when I see how it goes ( as I
>> said, the data sets are fairly large ).
>>
>> There shall be a better way to address that, but this is all I can do
>> with a very shallow acquaintance with rsync source code. Nevertheless,
>> it would be great if that could be fixed in the trunk one way or
another )
>>
>> Kind regards, George
>>
>> -- 
>> George Fedorov
>> Senior Systems Specialist
>> Melbourne School of Engineering
>> The University of Melbourne, Victoria 3010, Australia
>> phone: *****
>> email: *****
>> http://www.eng.unimelb.edu.au/
>>
>>
>>
>
>
-- 
George Fedorov
Senior Systems Specialist
Melbourne School of Engineering
The University of Melbourne, Victoria 3010, Australia
phone: +61-(0)3-903-56016
email: gfedorov at unimelb.edu.au
http://www.eng.unimelb.edu.au/

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.samba.org/pipermail/rsync/attachments/20170410/805753e7/attachment.html>