rsync - May 2019 - [Bug 13953] New: error message instead of --stats informations (in case of vanished files), using rsync 3.1.1 and rsync 3.0.9

https://bugzilla.samba.org/show_bug.cgi?id=13953

            Bug ID: 13953
           Summary: error message instead of --stats informations (in case
                    of vanished files), using rsync 3.1.1 and rsync 3.0.9
           Product: rsync
           Version: 3.1.1
          Hardware: x64
                OS: Linux
            Status: NEW
          Severity: normal
          Priority: P5
         Component: core
          Assignee: wayne at opencoder.net
          Reporter: gac at 4js.com
        QA Contact: rsync-qa at samba.org

Hello,

The issue I'm reporting is about the stats informations usually shown (by
--stats).

This stats information is missing in case of vanished files. I got an error
about protocol data stream instead.
Probably an issue between the both version/protocol.

It's appened since I have updated the rsync version on FOO (from 3.0.9 to
3.1.1).
Now the rsync version between "client"/"server" are not the
same.
In my case, the issue only appears in case of vanished files.

1) before (no issue)

on FOO rsync 3.0.9 (debian 7)
on BAR rsync 3.0.9 (RHEL 7)

2) after (got the issue)

on FOO rsync 3.1.1 (debian 8)
on BAR rsync 3.0.9 (RHEL 7)

I got this error instead of the stats informations :
> rsync: writefd_unbuffered failed to write 162 bytes to socket [sender]:
Broken pipe (32)
> rsync error: error in rsync protocol data stream (code 12) at io.c(1532)
[sender=3.0.9]
> rsync: [generator] write error: Broken pipe (32)
> rsync error: error in rsync protocol data stream (code 12) at io.c(820)
[generator=3.1.1]

On my backup serveur named FOO trying to get data from a source target named
BAR

The minimal command seems like :
    rsync [...] --stats [...] BAR:/path/to/dir/ ./

if you are curious the real used command is :
    rsync \
        --hard-links --numeric-ids --human-readable --prune-empty-dirs
--archive --one-file-system --force \
        --stats \
        --compare-dest=/path/to/dir1 --delete
--exclude-from=/path/to/file1.exclude --rsh=ssh \
        BAR:/path/to/dir/ ./




Additionnal informations
=======================
On 5/9/19 3:10 AM, root wrote:> Thu May  9 01:48:03 CEST
2019
> 
> Number of files: 1565540
> Number of files transferred: 591395
> Total file size: 1181.13G bytes
> Total transferred file size: 136.68G bytes
> Literal data: 129.42G bytes
> Matched data: 7.33G bytes
> File list size: 48.55M
> File list generation time: 9.667 seconds
> File list transfer time: 0.000 seconds
> Total bytes sent: 37.16M
> Total bytes received: 129.52G
> 
> sent 37.16M bytes  received 129.52G bytes  26.40M bytes/sec
> total size is 1181.13G  speedup is 9.12
> rsync warning: some files vanished before they could be transferred (code
24) at main.c(1536) [generator=3.0.9]
> 
> # there are some file vanished :
> file has vanished:
"/path/to/files/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
> file has vanished:
"/path/to/files/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
> file has vanished:
"/path/to/files/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
> file has vanished:
"/path/to/files/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"[...]

On 5/15/19 3:20 AM, root wrote:
> Wed May 15 02:31:00 CEST 2019
> 
> rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]:
Broken pipe (32)
> rsync error: error in rsync protocol data stream (code 12) at io.c(1532)
[sender=3.0.9]
> rsync: [generator] write error: Broken pipe (32)
> rsync error: error in rsync protocol data stream (code 12) at io.c(820)
[generator=3.1.1]
> 
> # there are some file vanished :
> file has vanished:
"/path/to/files/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
> file has vanished:
"/path/to/files/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
> file has vanished:
"/path/to/files/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
> file has vanished:
"/path/to/files/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
...


On 5/16/19 3:14 AM, root wrote:
> Thu May 16 02:03:05 CEST 2019
>
> rsync: writefd_unbuffered failed to write 162 bytes to socket [sender]:
Broken pipe (32)
> rsync error: error in rsync protocol data stream (code 12) at io.c(1532)
[sender=3.0.9]
> rsync: [generator] write error: Broken pipe (32)
> rsync error: error in rsync protocol data stream (code 12) at io.c(820)
[generator=3.1.1]
>
> # there are some file vanished :
> file has vanished:
"/path/to/files/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
> file has vanished:
"/path/to/files/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
> file has vanished:
"/path/to/files/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
> file has vanished:
"/path/to/files/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"


On FOO
# rsync --version
rsync  version 3.1.1  protocol version 31
Copyright (C) 1996-2014 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
    append, ACLs, xattrs, iconv, symtimes, prealloc

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.



On BAR
# rsync --version
rsync  version 3.0.9  protocol version 30
Copyright (C) 1996-2011 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
    append, ACLs, xattrs, iconv, symtimes

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.