rsync - Oct 2014 - [Bug 10857] New: weirdly named files fail remotely

https://bugzilla.samba.org/show_bug.cgi?id=10857

           Summary: weirdly named files fail remotely
           Product: rsync
           Version: 3.1.1
          Platform: x86
        OS/Version: Linux
            Status: NEW
          Severity: normal
          Priority: P5
         Component: core
        AssignedTo: wayned at samba.org
        ReportedBy: samba.org at tange.dk
         QAContact: rsync-qa at samba.org


Created attachment 10329
  --> https://bugzilla.samba.org/attachment.cgi?id=10329
Example of failing filename

The attached example creates a file that rsyncs locally, but fails to rsync
remotely.

-- 
Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the QA contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=10857

Wayne Davison <wayned at samba.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |WORKSFORME
             Status|NEW                         |RESOLVED

--- Comment #1 from Wayne Davison <wayned at samba.org> ---
Your problem is that you tried to specify a weird filename as a part of a
remote filename without escaping the characters from the remote shell
translating them.  You can use --protect-args to avoid the issue (which I have
set as my default due to my using "./configure --with-protected-args"
whenever
I compile rsync).  You would also not have run into an issue if you had just
specified a destination path and let rsync send the filename as a part of the
file list.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=10857

--- Comment #2 from samba.org at tange.dk <samba.org at tange.dk> ---
I understand what is happening and I know the workarounds for the bug - that is
not the issue.

Why is it that rsync prefers having a syntax, where if I transfer the file
locally I need to write something, but if I transfer the file remotely I need
to write something completely different?

Can you elaborate what situations this behaviour is beneficial and what would
break if this was changed to being able to write the same name whether the
transfer was local or remote?

I discovered the problem when writing a script, and I find it surprising that
you prefer, that I deal with the quoting in my script if and only if the
transfer is remote; instead of having rsync Do The Right Thing(TM) by default.

There may be situations where the current behaviour is preferable that I am
unaware of, and it would be great if you could describe those.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=10857

--- Comment #3 from Kevin Korb <rsync at sanitarium.net> ---
This isn't an rsync problem this is the way the shell works.  When you run
rsync over ssh as you are doing there rsync is running 'ssh remotehost rsync
[options] path'.  There is a shell between the sshd process and the remote
rsync process.  Therefore you have 2 layers of shell involved and have to deal
with both of them rather than just 1.

You would run into the same problem any time you double stack a shell:
echo "doesn't work"
vs
sh -c 'echo "doesn't work"'

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=10857

--- Comment #4 from samba.org at tange.dk <samba.org at tange.dk> ---
Can we start by agreeing that rsync _could_ be aware that it is starting a
remote shell and thus _could_ quote anything that needed quoting?

Currently it clearly does not quote and puts that responsibility on the user.

My question is: Why put that responsibility on the user? Why not help the user?
Rsync knows in advance that files with weird characters will cause problems, so
why not help the user by quoting them correctly?

In other words: What situations will break if --protect-args becomes the
default?

Is rsync following the principle of least surprise by not having --protect-args
be default, and thus requiring users to write the remote file differently than
the local file?

I would say rsync is breaking that principle: I as a user assume that I can
express the remote file name the same way as the local. There might be good
reasons for breaking the POLS, but so far I have not heard any. So let me ask
again: Are there any?

I can see how the current behaviour can be abused by a malicious user if the
user knows that 'root' copies files like this:

    find . -type f -print0 | parallel -0 rsync {} remote:backup/{}

Just:

    touch 'foo`perl -e '"'"'print
map{chr}114,109,32,47,101,116,99,47,112,97,115,115,119,100'"'"'|bash`bar'

and next time root copies, /etc/passwd will vanish with NO WARNING AT ALL.

Before I discovered this behaviour, I always assumed that:

    find . -type f -print0 | parallel -0 rsync {} remote:dir/{}

would be just as safe as:

    find . -type f -print0 | parallel -0 rsync {} dir/{}

So if you still want to keep the current behaviour, I believe the man page
should be clear that giving the remote file (without --protect-args) may result
in mayhem.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=10857

--- Comment #5 from Wayne Davison <wayned at samba.org> ---
If rsync begins quoting remote args, it would make assumptions about what needs
to be quoted and its rules for quoting things.  There is also a historical use
of arg-splitting that was primarily used on the source side, e.g. "rsync -a
host:"file1 file2" /dest/" that has been replaced by "rsync
-a
host:{file1,file2} /dest/" (which results in 2 host:file* args).

If we assume that enough time has passed to discard the backward compatibility
issue, the more proper solution is to avoid sending such args to the remote
shell in the first place, and that is what --protect-args does.  So, the really
we should just make --protect-args the default (forcing users interacting with
older rsync versions to specify --no-s when they get an option error).  This is
certainly what I have done for years now, and is probably what I should go
ahead and make the default in an upcoming configure.

Folks can also affect their own rsync use by putting "export
RSYNC_PROTECT_ARGS=1" in their environment, perhaps even in an
/etc/profile.d/rsync.sh file.  We can at the very least make this clearer in
the manpage.

As for this:

    find . -type f -print0 | parallel -0 rsync {} remote:backup/{}

The better way to do that is to specify -R (and -a) and omit the second {}
(especially since the original way can fail if the destination dir doesn't
exist).  FYI.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=10857

--- Comment #6 from samba.org at tange.dk <samba.org at tange.dk>
---
> As for this:
>
>    find . -type f -print0 | parallel -0 rsync {} remote:backup/{}
Is there also a better way for:

    find . -type f -size +1000 -print0 | parallel -0 rsync {} remote:backup/{}

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=10857

--- Comment #7 from Wayne Davison <wayned at samba.org> ---
Those are both the same on the rsync side of the pipe.  Use: rsync -aR {} dest/
(where dest is a directory path or a host:directory combo for the root dir of
the destination hierarchy).

-- 
You are receiving this mail because:
You are the QA Contact for the bug.