rsync - Oct 2019 - Seemingly impossible bug: -v not always listing every copied file

Thanks. I'll try that. But I agree that it'll be
something else. It's unlikely that whole trace files
are being overwritten, because there's locking in place
to prevent that sort of thing, but it's more likely
than anything else. I'll check that the locking is
working properly.

Although, when I last investigated it, it really did
look as though, in a single run of this task, three
files were copied for one directory but only two were
listed. I need to verify that more thoroughly, perhaps
with directory listings on both sides before and after
each run.

cheers,
raf

Kevin Korb via rsync wrote:
> It does seem impossible.  I would suggest adding --itemize-changes (-v
> isn't really all that useful without it anyway).  If entries are still
> missing then I would suspect that either log files are missing (maybe
> duplicate file names replacing the occasional log file?) or something
> other than rsync is doing things in the same dir.
> 
> On 10/29/19 9:00 PM, raf via rsync wrote:
> > Hi,
> > 
> > debian-9, rsync-3.1.2 (both ends)
> > 
> > I have a task that rsyncs files from a list of
> > candidate files (--files-from=). It's verbose (-v) and
> > its stdout is captured to a file which is then sent to
> > the receiving host. The captured verbose output is
> > examined on the receiving host to know which files were
> > actually copied so that notification emails can be sent
> > to various people.
> > 
> > The problem is that, sometimes, not all copied files
> > are listed in the verbose output, and so some
> > notification emails don't get sent out. At first, I
> > thought there was something wrong with the notification
> > emails not arriving, but the files in question, that
> > had definitely been copied, did not appear in any of
> > the captured verbose output files.
> > 
> > This seems like an impossible bug but it really seems
> > to be happening. Has anyone else encountered behaviour
> > like this? I didn't have much luck searching the
> > internet for it. It would probably be hard to notice
> > if the verbose output wasn't being used for something
> > like triggering notifications whose absence might be
> > noticed.
> > 
> > cheers,
> > raf
> 
> -- 
>
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
> 	Kevin Korb			Phone:    (407) 252-6853
> 	Systems Administrator		Internet:
> 	FutureQuest, Inc.		Kevin at FutureQuest.net  (work)
> 	Orlando, Florida		kmk at sanitarium.net (personal)
> 	Web page:			https://sanitarium.net/
> 	PGP public key available on web site.
>
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
> 


> -- 
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Hi raf,

Curious issue you have.  A few things:

What distro(s) are you using?

Same rsync version on both ends?

Hash of files look correct before and after the rsync?

Have you tried using inotify to monitor for changes at the fs level?  You
should see a "read" on the sender and a "read" +
"write" on the receiver.

On Tue, Oct 29, 2019, 11:25 PM raf via rsync <rsync at lists.samba.org>
wrote:
> Thanks. I'll try that. But I agree that it'll be
> something else. It's unlikely that whole trace files
> are being overwritten, because there's locking in place
> to prevent that sort of thing, but it's more likely
> than anything else. I'll check that the locking is
> working properly.
>
> Although, when I last investigated it, it really did
> look as though, in a single run of this task, three
> files were copied for one directory but only two were
> listed. I need to verify that more thoroughly, perhaps
> with directory listings on both sides before and after
> each run.
>
> cheers,
> raf
>
> Kevin Korb via rsync wrote:
>
> > It does seem impossible.  I would suggest adding --itemize-changes (-v
> > isn't really all that useful without it anyway).  If entries are
still
> > missing then I would suspect that either log files are missing (maybe
> > duplicate file names replacing the occasional log file?) or something
> > other than rsync is doing things in the same dir.
> >
> > On 10/29/19 9:00 PM, raf via rsync wrote:
> > > Hi,
> > >
> > > debian-9, rsync-3.1.2 (both ends)
> > >
> > > I have a task that rsyncs files from a list of
> > > candidate files (--files-from=). It's verbose (-v) and
> > > its stdout is captured to a file which is then sent to
> > > the receiving host. The captured verbose output is
> > > examined on the receiving host to know which files were
> > > actually copied so that notification emails can be sent
> > > to various people.
> > >
> > > The problem is that, sometimes, not all copied files
> > > are listed in the verbose output, and so some
> > > notification emails don't get sent out. At first, I
> > > thought there was something wrong with the notification
> > > emails not arriving, but the files in question, that
> > > had definitely been copied, did not appear in any of
> > > the captured verbose output files.
> > >
> > > This seems like an impossible bug but it really seems
> > > to be happening. Has anyone else encountered behaviour
> > > like this? I didn't have much luck searching the
> > > internet for it. It would probably be hard to notice
> > > if the verbose output wasn't being used for something
> > > like triggering notifications whose absence might be
> > > noticed.
> > >
> > > cheers,
> > > raf
> >
> > --
> >
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
> >       Kevin Korb                      Phone:    (407) 252-6853
> >       Systems Administrator           Internet:
> >       FutureQuest, Inc.               Kevin at FutureQuest.net  (work)
> >       Orlando, Florida                kmk at sanitarium.net (personal)
> >       Web page:                       https://sanitarium.net/
> >       PGP public key available on web site.
> >
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
> >
>
>
>
>
> > --
> > Please use reply-all for most replies to avoid omitting the mailing
list.
> > To unsubscribe or change options:
> https://lists.samba.org/mailman/listinfo/rsync
> > Before posting, read:
http://www.catb.org/~esr/faqs/smart-questions.html
>
>
> --
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options:
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.samba.org/pipermail/rsync/attachments/20191030/a8b4e701/attachment.htm>

If you are doing a small test run that duplicates the problem, you could
use inotifywait or tail -f to watch the log in real time on another
terminal. Maybe you'd see something like a line being overwritten, etc.

I don't know what rsync would do with a file name that ends in a
carriage return (not a newline). It's probably smart enough to handle
it, but if it isn't, the current log line might be overwritten by the
next one.

Joe

On 10/30/19 1:24 AM, raf via rsync wrote:
> Thanks. I'll try that. But I agree that it'll be
> something else. It's unlikely that whole trace files
> are being overwritten, because there's locking in place
> to prevent that sort of thing, but it's more likely
> than anything else. I'll check that the locking is
> working properly.
>
> Although, when I last investigated it, it really did
> look as though, in a single run of this task, three
> files were copied for one directory but only two were
> listed. I need to verify that more thoroughly, perhaps
> with directory listings on both sides before and after
> each run.
>
> cheers,
> raf
>
> Kevin Korb via rsync wrote:
>
>> It does seem impossible.  I would suggest adding --itemize-changes (-v
>> isn't really all that useful without it anyway).  If entries are
still
>> missing then I would suspect that either log files are missing (maybe
>> duplicate file names replacing the occasional log file?) or something
>> other than rsync is doing things in the same dir.
>>
>> On 10/29/19 9:00 PM, raf via rsync wrote:
>>> Hi,
>>>
>>> debian-9, rsync-3.1.2 (both ends)
>>>
>>> I have a task that rsyncs files from a list of
>>> candidate files (--files-from=). It's verbose (-v) and
>>> its stdout is captured to a file which is then sent to
>>> the receiving host. The captured verbose output is
>>> examined on the receiving host to know which files were
>>> actually copied so that notification emails can be sent
>>> to various people.
>>>
>>> The problem is that, sometimes, not all copied files
>>> are listed in the verbose output, and so some
>>> notification emails don't get sent out. At first, I
>>> thought there was something wrong with the notification
>>> emails not arriving, but the files in question, that
>>> had definitely been copied, did not appear in any of
>>> the captured verbose output files.
>>>
>>> This seems like an impossible bug but it really seems
>>> to be happening. Has anyone else encountered behaviour
>>> like this? I didn't have much luck searching the
>>> internet for it. It would probably be hard to notice
>>> if the verbose output wasn't being used for something
>>> like triggering notifications whose absence might be
>>> noticed.
>>>
>>> cheers,
>>> raf
>> -- 
>>
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>> 	Kevin Korb			Phone:    (407) 252-6853
>> 	Systems Administrator		Internet:
>> 	FutureQuest, Inc.		Kevin at FutureQuest.net  (work)
>> 	Orlando, Florida		kmk at sanitarium.net (personal)
>> 	Web page:			https://sanitarium.net/
>> 	PGP public key available on web site.
>>
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
>>
>
>
>
>> -- 
>> Please use reply-all for most replies to avoid omitting the mailing
list.
>> To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/rsync
>> Before posting, read:
http://www.catb.org/~esr/faqs/smart-questions.html
>

Kyle Bassett via rsync wrote:
> Hi raf,
> 
> Curious issue you have.  A few things:
> 
> What distro(s) are you using?
debian-9
> Same rsync version on both ends?
Yes, rsync-3.1.2
> Hash of files look correct before and after the rsync?
I didn't check but I expect so. I don't expect rsync to
ever do the wrong thing. :-)
> Have you tried using inotify to monitor for changes at the fs level?  You
> should see a "read" on the sender and a "read" +
"write" on the receiver.
No, but that's a good idea. I'll investigate that.
> On Tue, Oct 29, 2019, 11:25 PM raf via rsync <rsync at
lists.samba.org> wrote:
> 
> > Thanks. I'll try that. But I agree that it'll be
> > something else. It's unlikely that whole trace files
> > are being overwritten, because there's locking in place
> > to prevent that sort of thing, but it's more likely
> > than anything else. I'll check that the locking is
> > working properly.
> >
> > Although, when I last investigated it, it really did
> > look as though, in a single run of this task, three
> > files were copied for one directory but only two were
> > listed. I need to verify that more thoroughly, perhaps
> > with directory listings on both sides before and after
> > each run.
> >
> > cheers,
> > raf
> >
> > Kevin Korb via rsync wrote:
> >
> > > It does seem impossible.  I would suggest adding
--itemize-changes (-v
> > > isn't really all that useful without it anyway).  If entries
are still
> > > missing then I would suspect that either log files are missing
(maybe
> > > duplicate file names replacing the occasional log file?) or
something
> > > other than rsync is doing things in the same dir.
> > >
> > > On 10/29/19 9:00 PM, raf via rsync wrote:
> > > > Hi,
> > > >
> > > > debian-9, rsync-3.1.2 (both ends)
> > > >
> > > > I have a task that rsyncs files from a list of
> > > > candidate files (--files-from=). It's verbose (-v) and
> > > > its stdout is captured to a file which is then sent to
> > > > the receiving host. The captured verbose output is
> > > > examined on the receiving host to know which files were
> > > > actually copied so that notification emails can be sent
> > > > to various people.
> > > >
> > > > The problem is that, sometimes, not all copied files
> > > > are listed in the verbose output, and so some
> > > > notification emails don't get sent out. At first, I
> > > > thought there was something wrong with the notification
> > > > emails not arriving, but the files in question, that
> > > > had definitely been copied, did not appear in any of
> > > > the captured verbose output files.
> > > >
> > > > This seems like an impossible bug but it really seems
> > > > to be happening. Has anyone else encountered behaviour
> > > > like this? I didn't have much luck searching the
> > > > internet for it. It would probably be hard to notice
> > > > if the verbose output wasn't being used for something
> > > > like triggering notifications whose absence might be
> > > > noticed.
> > > >
> > > > cheers,
> > > > raf
> > >
> > > --
> > >
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
> > >       Kevin Korb                      Phone:    (407) 252-6853
> > >       Systems Administrator           Internet:
> > >       FutureQuest, Inc.               Kevin at FutureQuest.net 
(work)
> > >       Orlando, Florida                kmk at sanitarium.net
(personal)
> > >       Web page:                       https://sanitarium.net/
> > >       PGP public key available on web site.
> > >
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,

Unfortunately, I can't replicate the problem at will.
But I'll investigate using iwatch/inotify to log
everything from the file system's point of view.
That should shed some light on what's happening.

The file names don't contain any control characters
(but good thinking!).

thanks,
raf

Joe via rsync wrote:
> If you are doing a small test run that duplicates the problem, you could
> use inotifywait or tail -f to watch the log in real time on another
> terminal. Maybe you'd see something like a line being overwritten, etc.
> I don't know what rsync would do with a file name that ends in a
> carriage return (not a newline). It's probably smart enough to handle
> it, but if it isn't, the current log line might be overwritten by the
> next one.
> 
> Joe
> 
> On 10/30/19 1:24 AM, raf via rsync wrote:
> > Thanks. I'll try that. But I agree that it'll be
> > something else. It's unlikely that whole trace files
> > are being overwritten, because there's locking in place
> > to prevent that sort of thing, but it's more likely
> > than anything else. I'll check that the locking is
> > working properly.
> >
> > Although, when I last investigated it, it really did
> > look as though, in a single run of this task, three
> > files were copied for one directory but only two were
> > listed. I need to verify that more thoroughly, perhaps
> > with directory listings on both sides before and after
> > each run.
> >
> > cheers,
> > raf
> >
> > Kevin Korb via rsync wrote:
> >
> >> It does seem impossible.  I would suggest adding --itemize-changes
(-v
> >> isn't really all that useful without it anyway).  If entries
are still
> >> missing then I would suspect that either log files are missing
(maybe
> >> duplicate file names replacing the occasional log file?) or
something
> >> other than rsync is doing things in the same dir.
> >>
> >> On 10/29/19 9:00 PM, raf via rsync wrote:
> >>> Hi,
> >>>
> >>> debian-9, rsync-3.1.2 (both ends)
> >>>
> >>> I have a task that rsyncs files from a list of
> >>> candidate files (--files-from=). It's verbose (-v) and
> >>> its stdout is captured to a file which is then sent to
> >>> the receiving host. The captured verbose output is
> >>> examined on the receiving host to know which files were
> >>> actually copied so that notification emails can be sent
> >>> to various people.
> >>>
> >>> The problem is that, sometimes, not all copied files
> >>> are listed in the verbose output, and so some
> >>> notification emails don't get sent out. At first, I
> >>> thought there was something wrong with the notification
> >>> emails not arriving, but the files in question, that
> >>> had definitely been copied, did not appear in any of
> >>> the captured verbose output files.
> >>>
> >>> This seems like an impossible bug but it really seems
> >>> to be happening. Has anyone else encountered behaviour
> >>> like this? I didn't have much luck searching the
> >>> internet for it. It would probably be hard to notice
> >>> if the verbose output wasn't being used for something
> >>> like triggering notifications whose absence might be
> >>> noticed.
> >>>
> >>> cheers,
> >>> raf
> >> -- 
> >>
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
> >> 	Kevin Korb			Phone:    (407) 252-6853
> >> 	Systems Administrator		Internet:
> >> 	FutureQuest, Inc.		Kevin at FutureQuest.net  (work)
> >> 	Orlando, Florida		kmk at sanitarium.net (personal)
> >> 	Web page:			https://sanitarium.net/
> >> 	PGP public key available on web site.
> >>
~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,
> >>
> >
> >
> >
> >> -- 
> >> Please use reply-all for most replies to avoid omitting the
mailing list.
> >> To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/rsync
> >> Before posting, read:
http://www.catb.org/~esr/faqs/smart-questions.html
> >
> 
> -- 
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html