Vangelis Katsikaros
2017-Sep-07 14:27 UTC
Should I worry about the "vanished files" warning?
Hi I would like to ask, when the "vanished files" warning is a sign that something bad is happening somewhere. I know that the `rsync-no-vanished` script can silence the warning, but I am wondering whether this is a sane thing to do. My guess based on https://bugzilla.samba.org/show_bug.cgi?id=3653#c26 and https://bugzilla.samba.org/show_bug.cgi?id=10356 is that it's fine, but I'd like to verify :) In my case, I have a batch process that creates ~500 files, some take 30 min, some take 8h to be created. The files are created in a tmp dir and are moved to a final dir when they are finished. Every 5 mins, I rsync the files from the final directory to a remote machine. I guess that during the sync process some files get re-written, when the batch process moves the fresh ones in the final dir. My cron entry: flock -xn /lock_file -c \ "/usr/bin/rsync -e \"ssh -C\" --bwlimit=512k \ --recursive --delay-updates --quiet --update \ --include=\"*/\" --include=\"*.xml\" --include=\"*.csv\" \ --exclude=\"*\" \ /LOCAL_SOURCE_DIR/ REMOTE_IP:/DEST_DIR/" Both source and destination are on $ rsync --version rsync version 3.1.0 protocol version 31 Regards Vangelis
All it means is that rsync saw a file that needed transferring but the file was gone when rsync actually tried to open it. So look at the filenames and decide if it is a problem or not. On 09/07/2017 10:27 AM, Vangelis Katsikaros via rsync wrote:> Hi > > I would like to ask, when the "vanished files" warning is a sign that something bad is happening somewhere. I know that the `rsync-no-vanished` script can silence the warning, but I am wondering whether this is a sane thing to do. > > My guess based on https://bugzilla.samba.org/show_bug.cgi?id=3653#c26 and https://bugzilla.samba.org/show_bug.cgi?id=10356 is that it's fine, but I'd like to verify :) > > In my case, I have a batch process that creates ~500 files, some take 30 min, some take 8h to be created. The files are created in a tmp dir and are moved to a final dir when they are finished. Every 5 mins, I rsync the files from the final directory to a remote machine. I guess that during the sync process some files get re-written, when the batch process moves the fresh ones in the final dir. > > My cron entry: > flock -xn /lock_file -c \ > "/usr/bin/rsync -e \"ssh -C\" --bwlimit=512k \ > --recursive --delay-updates --quiet --update \ > --include=\"*/\" --include=\"*.xml\" --include=\"*.csv\" \ > --exclude=\"*\" \ > /LOCAL_SOURCE_DIR/ REMOTE_IP:/DEST_DIR/" > > Both source and destination are on > $ rsync --version > rsync version 3.1.0 protocol version 31 > > Regards > Vangelis >-- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone: (407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. Kevin at FutureQuest.net (work) Orlando, Florida kmk at sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 224 bytes Desc: OpenPGP digital signature URL: <http://lists.samba.org/pipermail/rsync/attachments/20170907/97e5da99/signature.sig>
Vangelis Katsikaros
2017-Sep-13 08:57 UTC
Should I worry about the "vanished files" warning?
Hi Kevin Thanks for the tip. In my case it seems it's fine! Regards Vangelis On 09/07/2017 08:51 PM, Kevin Korb via rsync wrote:> All it means is that rsync saw a file that needed transferring but the > file was gone when rsync actually tried to open it. So look at the > filenames and decide if it is a problem or not. > > On 09/07/2017 10:27 AM, Vangelis Katsikaros via rsync wrote: >> Hi >> >> I would like to ask, when the "vanished files" warning is a sign that something bad is happening somewhere. I know that the `rsync-no-vanished` script can silence the warning, but I am wondering whether this is a sane thing to do. >> >> My guess based on https://bugzilla.samba.org/show_bug.cgi?id=3653#c26 and https://bugzilla.samba.org/show_bug.cgi?id=10356 is that it's fine, but I'd like to verify :) >> >> In my case, I have a batch process that creates ~500 files, some take 30 min, some take 8h to be created. The files are created in a tmp dir and are moved to a final dir when they are finished. Every 5 mins, I rsync the files from the final directory to a remote machine. I guess that during the sync process some files get re-written, when the batch process moves the fresh ones in the final dir. >> >> My cron entry: >> flock -xn /lock_file -c \ >> "/usr/bin/rsync -e \"ssh -C\" --bwlimit=512k \ >> --recursive --delay-updates --quiet --update \ >> --include=\"*/\" --include=\"*.xml\" --include=\"*.csv\" \ >> --exclude=\"*\" \ >> /LOCAL_SOURCE_DIR/ REMOTE_IP:/DEST_DIR/" >> >> Both source and destination are on >> $ rsync --version >> rsync version 3.1.0 protocol version 31 >> >> Regards >> Vangelis >> > > >