rsync - Sep 2017 - Should I worry about the "vanished files" warning?

Hi

I would like to ask, when the "vanished files" warning is a sign that
something bad is happening somewhere. I know that the `rsync-no-vanished` script
can silence the warning, but I am wondering whether this is a sane thing to do.

My guess based on https://bugzilla.samba.org/show_bug.cgi?id=3653#c26 and
https://bugzilla.samba.org/show_bug.cgi?id=10356 is that it's fine, but
I'd like to verify :)

In my case, I have a batch process that creates ~500 files, some take 30 min,
some take 8h to be created. The files are created in a tmp dir and are moved to
a final dir when they are finished. Every 5 mins, I rsync the files from the
final directory to a remote machine. I guess that during the sync process some
files get re-written, when the batch process moves the fresh ones in the final
dir.

My cron entry:
flock -xn /lock_file -c                                       \
"/usr/bin/rsync -e \"ssh -C\" --bwlimit=512k                  \
--recursive --delay-updates --quiet --update                  \
--include=\"*/\" --include=\"*.xml\"
--include=\"*.csv\"      \
--exclude=\"*\"                                               \
/LOCAL_SOURCE_DIR/ REMOTE_IP:/DEST_DIR/"

Both source and destination are on 
$ rsync --version
rsync  version 3.1.0  protocol version 31

Regards
Vangelis

All it means is that rsync saw a file that needed transferring but the
file was gone when rsync actually tried to open it.  So look at the
filenames and decide if it is a problem or not.

On 09/07/2017 10:27 AM, Vangelis Katsikaros via rsync
wrote:
> Hi
> 
> I would like to ask, when the "vanished files" warning is a sign
that something bad is happening somewhere. I know that the `rsync-no-vanished`
script can silence the warning, but I am wondering whether this is a sane thing
to do.
> 
> My guess based on https://bugzilla.samba.org/show_bug.cgi?id=3653#c26 and
https://bugzilla.samba.org/show_bug.cgi?id=10356 is that it's fine, but
I'd like to verify :)
> 
> In my case, I have a batch process that creates ~500 files, some take 30
min, some take 8h to be created. The files are created in a tmp dir and are
moved to a final dir when they are finished. Every 5 mins, I rsync the files
from the final directory to a remote machine. I guess that during the sync
process some files get re-written, when the batch process moves the fresh ones
in the final dir.
> 
> My cron entry:
> flock -xn /lock_file -c                                       \
> "/usr/bin/rsync -e \"ssh -C\" --bwlimit=512k                
\
> --recursive --delay-updates --quiet --update                  \
> --include=\"*/\" --include=\"*.xml\"
--include=\"*.csv\"      \
> --exclude=\"*\"                                               \
> /LOCAL_SOURCE_DIR/ REMOTE_IP:/DEST_DIR/"
> 
> Both source and destination are on 
> $ rsync --version
> rsync  version 3.1.0  protocol version 31
> 
> Regards
> Vangelis
> 
-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
	Kevin Korb			Phone:    (407) 252-6853
	Systems Administrator		Internet:
	FutureQuest, Inc.		Kevin at FutureQuest.net  (work)
	Orlando, Florida		kmk at sanitarium.net (personal)
	Web page:			http://www.sanitarium.net/
	PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 224 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.samba.org/pipermail/rsync/attachments/20170907/97e5da99/signature.sig>

Hi Kevin

Thanks for the tip. In my case it seems it's fine!

Regards
Vangelis

On 09/07/2017 08:51 PM, Kevin Korb via rsync wrote:
> All it means is that rsync saw a file that needed transferring but the
> file was gone when rsync actually tried to open it.  So look at the
> filenames and decide if it is a problem or not.
> 
> On 09/07/2017 10:27 AM, Vangelis Katsikaros via rsync wrote:
>> Hi
>>
>> I would like to ask, when the "vanished files" warning is a
sign that something bad is happening somewhere. I know that the
`rsync-no-vanished` script can silence the warning, but I am wondering whether
this is a sane thing to do.
>>
>> My guess based on https://bugzilla.samba.org/show_bug.cgi?id=3653#c26
and https://bugzilla.samba.org/show_bug.cgi?id=10356 is that it's fine, but
I'd like to verify :)
>>
>> In my case, I have a batch process that creates ~500 files, some take
30 min, some take 8h to be created. The files are created in a tmp dir and are
moved to a final dir when they are finished. Every 5 mins, I rsync the files
from the final directory to a remote machine. I guess that during the sync
process some files get re-written, when the batch process moves the fresh ones
in the final dir.
>>
>> My cron entry:
>> flock -xn /lock_file -c                                       \
>> "/usr/bin/rsync -e \"ssh -C\" --bwlimit=512k            
\
>> --recursive --delay-updates --quiet --update                  \
>> --include=\"*/\" --include=\"*.xml\"
--include=\"*.csv\"      \
>> --exclude=\"*\"                                              
\
>> /LOCAL_SOURCE_DIR/ REMOTE_IP:/DEST_DIR/"
>>
>> Both source and destination are on 
>> $ rsync --version
>> rsync  version 3.1.0  protocol version 31
>>
>> Regards
>> Vangelis
>>
> 
> 
>