rsync - Oct 2019 - Seemingly impossible bug: -v not always listing every copied file

Hi,

debian-9, rsync-3.1.2 (both ends)

I have a task that rsyncs files from a list of
candidate files (--files-from=). It's verbose (-v) and
its stdout is captured to a file which is then sent to
the receiving host. The captured verbose output is
examined on the receiving host to know which files were
actually copied so that notification emails can be sent
to various people.

The problem is that, sometimes, not all copied files
are listed in the verbose output, and so some
notification emails don't get sent out. At first, I
thought there was something wrong with the notification
emails not arriving, but the files in question, that
had definitely been copied, did not appear in any of
the captured verbose output files.

This seems like an impossible bug but it really seems
to be happening. Has anyone else encountered behaviour
like this? I didn't have much luck searching the
internet for it. It would probably be hard to notice
if the verbose output wasn't being used for something
like triggering notifications whose absence might be
noticed.

cheers,
raf

It does seem impossible.  I would suggest adding --itemize-changes (-v
isn't really all that useful without it anyway).  If entries are still
missing then I would suspect that either log files are missing (maybe
duplicate file names replacing the occasional log file?) or something
other than rsync is doing things in the same dir.

On 10/29/19 9:00 PM, raf via rsync wrote:
> Hi,
> 
> debian-9, rsync-3.1.2 (both ends)
> 
> I have a task that rsyncs files from a list of
> candidate files (--files-from=). It's verbose (-v) and
> its stdout is captured to a file which is then sent to
> the receiving host. The captured verbose output is
> examined on the receiving host to know which files were
> actually copied so that notification emails can be sent
> to various people.
> 
> The problem is that, sometimes, not all copied files
> are listed in the verbose output, and so some
> notification emails don't get sent out. At first, I
> thought there was something wrong with the notification
> emails not arriving, but the files in question, that
> had definitely been copied, did not appear in any of
> the captured verbose output files.
> 
> This seems like an impossible bug but it really seems
> to be happening. Has anyone else encountered behaviour
> like this? I didn't have much luck searching the
> internet for it. It would probably be hard to notice
> if the verbose output wasn't being used for something
> like triggering notifications whose absence might be
> noticed.
> 
> cheers,
> raf
> 
> 
-- 
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
	Kevin Korb			Phone:    (407) 252-6853
	Systems Administrator		Internet:
	FutureQuest, Inc.		Kevin at FutureQuest.net  (work)
	Orlando, Florida		kmk at sanitarium.net (personal)
	Web page:			https://sanitarium.net/
	PGP public key available on web site.
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.samba.org/pipermail/rsync/attachments/20191029/7ab4d987/signature.sig>

Thanks. I'll try that. But I agree that it'll be
something else. It's unlikely that whole trace files
are being overwritten, because there's locking in place
to prevent that sort of thing, but it's more likely
than anything else. I'll check that the locking is
working properly.

Although, when I last investigated it, it really did
look as though, in a single run of this task, three
files were copied for one directory but only two were
listed. I need to verify that more thoroughly, perhaps
with directory listings on both sides before and after
each run.

cheers,
raf

Kevin Korb via rsync wrote:
> It does seem impossible.  I would suggest adding --itemize-changes (-v
> isn't really all that useful without it anyway).  If entries are still
> missing then I would suspect that either log files are missing (maybe
> duplicate file names replacing the occasional log file?) or something
> other than rsync is doing things in the same dir.
> 
> On 10/29/19 9:00 PM, raf via rsync wrote:
> > Hi,
> > 
> > debian-9, rsync-3.1.2 (both ends)
> > 
> > I have a task that rsyncs files from a list of
> > candidate files (--files-from=). It's verbose (-v) and
> > its stdout is captured to a file which is then sent to
> > the receiving host. The captured verbose output is
> > examined on the receiving host to know which files were
> > actually copied so that notification emails can be sent
> > to various people.
> > 
> > The problem is that, sometimes, not all copied files
> > are listed in the verbose output, and so some
> > notification emails don't get sent out. At first, I
> > thought there was something wrong with the notification
> > emails not arriving, but the files in question, that
> > had definitely been copied, did not appear in any of
> > the captured verbose output files.
> > 
> > This seems like an impossible bug but it really seems
> > to be happening. Has anyone else encountered behaviour
> > like this? I didn't have much luck searching the
> > internet for it. It would probably be hard to notice
> > if the verbose output wasn't being used for something
> > like triggering notifications whose absence might be
> > noticed.
> > 
> > cheers,
> > raf
> 
> -- 
>
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
> 	Kevin Korb			Phone:    (407) 252-6853
> 	Systems Administrator		Internet:
> 	FutureQuest, Inc.		Kevin at FutureQuest.net  (work)
> 	Orlando, Florida		kmk at sanitarium.net (personal)
> 	Web page:			https://sanitarium.net/
> 	PGP public key available on web site.
>
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
> 


> -- 
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

On Wed 30 Oct 2019, raf via rsync wrote:
> 
> I have a task that rsyncs files from a list of
> candidate files (--files-from=). It's verbose (-v) and
It would be helpful to show the complete rsync command line.


Paul

The complete rsync command at the client end is approximately:

  rsync --files-from=/tmp/rptaudit.list.4958 -zltv --compress-level=9 \
  -e 'ssh' --rsync-path 'umask 002; rsync' --chmod=D2770,F640
--exclude=archive \
  ./ user at host:/var/thing/report >> /var/log/rpt2web/rpt2web.trace

cheers,
raf

Paul Slootman via rsync wrote:
> On Wed 30 Oct 2019, raf via rsync wrote:
> > 
> > I have a task that rsyncs files from a list of
> > candidate files (--files-from=). It's verbose (-v) and
> 
> It would be helpful to show the complete rsync command line.
> 
> Paul