rsync - Nov 2017 - [Bug 13147] New: inconsistent behaviour regaring vanished files information

https://bugzilla.samba.org/show_bug.cgi?id=13147

            Bug ID: 13147
           Summary: inconsistent behaviour regaring vanished files
                    information
           Product: rsync
           Version: 3.0.9
          Hardware: x64
                OS: Mac OS X
            Status: NEW
          Severity: minor
          Priority: P5
         Component: core
          Assignee: wayned at samba.org
          Reporter: devzero at web.de
        QA Contact: rsync-qa at samba.org

i found that for our osx host "gonzo" , vanished files (not the
warning message
itself) appear in stdout - for linux hosts they _both_ appear in stderr. no
information about vanished files appears in stdout (in my backup script, i´m
redirecting stdout to rsync.log and stderr to rsync.err)

what explains this difference in behaviour? 

is this a bug ? or is it related to rsynv version ?

backupvm1->spacewalk is 3.0.9->3.0.9 and backupvm1->gonzo is
3.0.9->3.1.2
(installed from brew repo) 

[root at backupvm1 backup]# grep -i vanished spacewalk/rsync.*
spacewalk/rsync.err:file has vanished:
"/var/lib/rhn/search/indexes/server/_80a2.cfs"
spacewalk/rsync.err:file has vanished:
"/var/lib/rhn/search/indexes/server/_80a2_6.del"
spacewalk/rsync.err:file has vanished:
"/var/lib/rhn/search/indexes/server/_80a3.cfs"
spacewalk/rsync.err:file has vanished:
"/var/lib/rhn/search/indexes/server/_80a4.cfs"
spacewalk/rsync.err:file has vanished:
"/var/lib/rhn/search/indexes/server/_80a5.cfs"
spacewalk/rsync.err:file has vanished:
"/var/lib/rhn/search/indexes/server/segments_n801"
spacewalk/rsync.err:rsync warning: some files vanished before they could be
transferred (code 24) at main.c(1518) [generator=3.0.9]

[root at backupvm1 backup]# grep -i vanished gonzo/rsync.*
gonzo/rsync.err:rsync warning: some files vanished before they could be
transferred (code 24) at main.c(1518) [generator=3.0.9]
gonzo/rsync.log:file has vanished:
"/Library/Server/Mail/Data/spool/maildrop/AFD811E2D07C"
gonzo/rsync.log:file has vanished:
"/Library/Server/Wiki/Database.xpg/Cluster.pg/pg_xlog/000000010000004F00000025"
gonzo/rsync.log:file has vanished:
"/Library/Server/Wiki/Database.xpg/Cluster.pg/pg_xlog/archive_status/000000010000004F00000025.done"
gonzo/rsync.log:file has vanished:
"/Library/Server/Wiki/Database.xpg/backup/000000010000004F00000026.partial"

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=13147

--- Comment #1 from roland <devzero at web.de> ---
i searched into this a little bit and it`s not a problem with differing rsync
versions, it also happens with rsync 3.1.2 (linux) -> rsync 3.1.2 (osx)

what i found is, that rsync 3.1.2 behaves correctly when run on osx locally,
i.e. if i rsync -av /tmp/localdir/rapidly-appearing+disappearing-files-inside 
/tmp/localdir2  on osx locally, i`m getting all vanished files messages
including which files are missing on stdout. 

if i do the same from linux/remote, then the information which files are
vanished goes to stdout instead of sterr.
the message "rsync warning: some files vanished before they could be
transferred (code 24) at main.c(1650) [generator=3.1.2]"  still goes to
sterr
(as expected)

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=13147

--- Comment #2 from roland <devzero at web.de> ---
it`s also no linux->osx interacting issue, the problem also appears when
doing
rsync on osx via ssh/localhost.


but why ?

ssh & osx does handle stdout and stdout correctly, as shown here:


gonzo:tmp root# ssh root at localhost "echo stdout; echo stderr
>&2" >/dev/null
stderr
gonzo:tmp root# ssh root at localhost "echo stdout; echo stderr
>&2" 2>/dev/null
stdout

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=13147

--- Comment #3 from roland <devzero at web.de> ---
apparently, ssh as a transport is the culprit, as it doesn`t happen with rsync
in daemon mode.

i found some osx related ticket on "stderr/stdout merge" (
https://github.com/ansible/ansible/issues/14960 ) 

after further searching, i found some infos on pty/tty allocation:
https://stackoverflow.com/questions/7114990/pseudo-terminal-will-not-be-allocated-because-stdin-is-not-a-terminal
 (see comment "Optional background information:")

 and...


# ssh root at localhost "echo stdout; echo stderr >&2"
>/dev/null
stderr

# ssh root at localhost "echo stdout; echo stderr >&2"
2>/dev/null
stdout

# ssh -t root at localhost "echo stdout; echo stderr >&2"
2>/dev/null
stdout
stderr

gotcha !

now the question is how to work around the problem...

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=13147

--- Comment #4 from roland <devzero at web.de> ---
mhh, if the error is not a warning but an error, the offending file is shown
(for the warning of vanished files, the warning itself goes to stderr but the
vanished filename goes to stdout). but only on OSX

why that difference ?

rsync: readlink
"/private/var/folders/7p/mcsl60wn0djct6d9_kpzy1nc0000gp/0/com.apple.LaunchServices.dv"
failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code
23) at main.c(1518) [generator=3.0.9]

-- 
You are receiving this mail because:
You are the QA Contact for the bug.