Hello,
I'm using Ruby's Shellwords module, which generates a string from an
array, suitable for shell evaluation.
Ruby's implementation prefers escaping whitespace with a backslash
rather than quotes. However, this appears to cause some kind of issue
in Rsync when it computes argv from -e option.
Here is an example command generated by some Ruby code:
rsync --archive --stats -e ssh\ -l\ backup\ -i\ /etc/synco/id_rsa\ -o\
ConnectTimeout\\\=60\ -o\ BatchMode\\\=yes --link-dest
../../latest/etc/ /etc/
example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/
We can check that something like this is valid:
files% echo foo\ bar\\\=baz
foo bar\=baz -- what Rsync should be receiving
files% echo foo bar\=baz
foo bar=baz -- What Rsync should be executing
However this gives me an error
command-line: line 0: Bad configuration option: connecttimeout\\
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.2]
I think the problem here is the "ConnectTimeout\\\=60", in particular
how the equals symbol is escaped.
I'm looking in the function:
static pid_t do_cmd(char *cmd, char *machine, char *user, char
**remote_argv, int remote_argc,
int *f_in_p, int *f_out_p)
This function splits based purely on whitespace:
args[argc++] = t;
while (*f != ' ' || in_quote) {
// consume token...
I feel that this function should also handle backslash escapes.
I also checked using strace and it appears that this is the issue, but
I'm open to suggestions/ideas.
Kind regards,
Samuel
On Thu 20 Oct 2016, Samuel Williams wrote:> > I'm using Ruby's Shellwords module, which generates a string from an > array, suitable for shell evaluation. > > Ruby's implementation prefers escaping whitespace with a backslash > rather than quotes. However, this appears to cause some kind of issue > in Rsync when it computes argv from -e option. > > Here is an example command generated by some Ruby code: > > rsync --archive --stats -e ssh\ -l\ backup\ -i\ /etc/synco/id_rsa\ -o\ > ConnectTimeout\\\=60\ -o\ BatchMode\\\=yes --link-dest > ../../latest/etc/ /etc/ > example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/There's no reason to escape an "=" sign in the above command. Try: rsync --archive --stats -e ssh\ -l\ backup\ -i\ /etc/synco/id_rsa\ -o\ ConnectTimeout=60\ -o\ BatchMode=yes --link-dest ../../latest/etc/ /etc/ example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/ Or even: rsync --archive --stats -e ssh\ -i\ /etc/synco/id_rsa\ -o\ ConnectTimeout=60\ -o\ BatchMode=yes --link-dest ../../latest/etc/ /etc/ backup at example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/ I'd probably create an entry for this host in ~/.ssh/config : Host example.backup.server.com User backup IdentityFile /etc/synco/id_rsa ConnectTimeout 60 BatchMode=yes and then just use: rsync --archive --stats --link-dest ../../latest/etc/ /etc/ example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/ If you're dynamically doing this, you can pass a config file with -F. Paul
> There's no reason to escape an "=" sign in the above command.Okay, so at a certain level I agree with you and that's how I've fixed the issue. All the stuff about config files is not feasible for a variety of reasons - I'm aware of those options. I appreciate that you took the time to explain them. I think the problem is the assumptions in do_cmd about the contents of the -e argument. In my opinion, the way this is split needs to be deferred back to the user's shell to work correctly. I'm not sure if there is an easy way to do this that works for any kind of shell. Assuming that -e is stored into rsh_cmd, and is user supplied for executing on the shell, the only valid thing you can really do with it is: execl("/bin/sh", "sh", "-c", rsh_cmd, (char *) NULL); (assuming that sh is the user's current shell, might want to use $SHELL or something else). It might make more sense if rsh_cmd used string substitution rather than argument splitting, e.g. -e 'ssh $HOST' this way, no splitting would be required. In lots of ways this is more flexible. For example, appending the host might not be useful in some contexts. Anyway, rsync is pretty awesome. the -e option is only a minor annoyance, but it would be nice if this discussion could result in a tangible improvement. On 21 October 2016 at 00:03, Paul Slootman <paul+rsync at wurtel.net> wrote:> On Thu 20 Oct 2016, Samuel Williams wrote: >> >> I'm using Ruby's Shellwords module, which generates a string from an >> array, suitable for shell evaluation. >> >> Ruby's implementation prefers escaping whitespace with a backslash >> rather than quotes. However, this appears to cause some kind of issue >> in Rsync when it computes argv from -e option. >> >> Here is an example command generated by some Ruby code: >> >> rsync --archive --stats -e ssh\ -l\ backup\ -i\ /etc/synco/id_rsa\ -o\ >> ConnectTimeout\\\=60\ -o\ BatchMode\\\=yes --link-dest >> ../../latest/etc/ /etc/ >> example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/ > > There's no reason to escape an "=" sign in the above command. Try: > > rsync --archive --stats -e ssh\ -l\ backup\ -i\ /etc/synco/id_rsa\ -o\ ConnectTimeout=60\ -o\ BatchMode=yes --link-dest ../../latest/etc/ /etc/ example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/ > > Or even: > > rsync --archive --stats -e ssh\ -i\ /etc/synco/id_rsa\ -o\ ConnectTimeout=60\ -o\ BatchMode=yes --link-dest ../../latest/etc/ /etc/ backup at example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/ > > I'd probably create an entry for this host in ~/.ssh/config : > > Host example.backup.server.com > User backup > IdentityFile /etc/synco/id_rsa > ConnectTimeout 60 > BatchMode=yes > > and then just use: > > rsync --archive --stats --link-dest ../../latest/etc/ /etc/ example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/ > > > If you're dynamically doing this, you can pass a config file with -F. > > > Paul > > -- > Please use reply-all for most replies to avoid omitting the mailing list. > To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
On 2016-10-20 10:24, Samuel Williams wrote:> Hello, > > I'm using Ruby's Shellwords module, which generates a string from an > array, suitable for shell evaluation. > > Ruby's implementation prefers escaping whitespace with a backslash > rather than quotes. However, this appears to cause some kind of issue > in Rsync when it computes argv from -e option.The man page for rsync <https://download.samba.org/pub/rsync/rsync.html> explicitly forbids the use of backslashes: "Command-line arguments are permitted in COMMAND provided that COMMAND is presented to rsync as a single argument. You must use spaces (not tabs or other whitespace) to separate the command and args from each other, and you can use single- and/or double-quotes to preserve spaces in an argument (but not backslashes)." So I think the rest of the question is moot ... Cheers, Dave> Here is an example command generated by some Ruby code: > > rsync --archive --stats -e ssh\ -l\ backup\ -i\ /etc/synco/id_rsa\ -o\ > ConnectTimeout\\\=60\ -o\ BatchMode\\\=yes --link-dest > ../../latest/etc/ /etc/ > example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/ > > We can check that something like this is valid: > > files% echo foo\ bar\\\=baz > foo bar\=baz -- what Rsync should be receiving > files% echo foo bar\=baz > foo bar=baz -- What Rsync should be executing > > However this gives me an error > > command-line: line 0: Bad configuration option: connecttimeout\\ > rsync: connection unexpectedly closed (0 bytes received so far) [sender] > rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.2] > > I think the problem here is the "ConnectTimeout\\\=60", in particular > how the equals symbol is escaped. > > I'm looking in the function: > > static pid_t do_cmd(char *cmd, char *machine, char *user, char > **remote_argv, int remote_argc, > int *f_in_p, int *f_out_p) > > This function splits based purely on whitespace: > > args[argc++] = t; > while (*f != ' ' || in_quote) { > // consume token... > > I feel that this function should also handle backslash escapes. > > I also checked using strace and it appears that this is the issue, but > I'm open to suggestions/ideas. > > Kind regards, > Samuel >
The \ escapes are for the shell. Rsync never sees them and therefore can't honor them. On 10/20/2016 05:24 AM, Samuel Williams wrote:> Hello, > > I'm using Ruby's Shellwords module, which generates a string from an > array, suitable for shell evaluation. > > Ruby's implementation prefers escaping whitespace with a backslash > rather than quotes. However, this appears to cause some kind of issue > in Rsync when it computes argv from -e option. > > Here is an example command generated by some Ruby code: > > rsync --archive --stats -e ssh\ -l\ backup\ -i\ /etc/synco/id_rsa\ -o\ > ConnectTimeout\\\=60\ -o\ BatchMode\\\=yes --link-dest > ../../latest/etc/ /etc/ > example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/ > > We can check that something like this is valid: > > files% echo foo\ bar\\\=baz > foo bar\=baz -- what Rsync should be receiving > files% echo foo bar\=baz > foo bar=baz -- What Rsync should be executing > > However this gives me an error > > command-line: line 0: Bad configuration option: connecttimeout\\ > rsync: connection unexpectedly closed (0 bytes received so far) [sender] > rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.2] > > I think the problem here is the "ConnectTimeout\\\=60", in particular > how the equals symbol is escaped. > > I'm looking in the function: > > static pid_t do_cmd(char *cmd, char *machine, char *user, char > **remote_argv, int remote_argc, > int *f_in_p, int *f_out_p) > > This function splits based purely on whitespace: > > args[argc++] = t; > while (*f != ' ' || in_quote) { > // consume token... > > I feel that this function should also handle backslash escapes. > > I also checked using strace and it appears that this is the issue, but > I'm open to suggestions/ideas. > > Kind regards, > Samuel >-- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., Kevin Korb Phone: (407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. Kevin at FutureQuest.net (work) Orlando, Florida kmk at sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._., -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 181 bytes Desc: OpenPGP digital signature URL: <http://lists.samba.org/pipermail/rsync/attachments/20161020/6d915c0d/signature.sig>
Hi Dave, thanks for point that out. I didn't realise there was a detailed explanation of that field in the man page, I only saw the summary. Yes, that clearly explains how it's supposed to work. On 21 October 2016 at 01:46, Dave Howorth <dhoworth at mrc-lmb.cam.ac.uk> wrote:> On 2016-10-20 10:24, Samuel Williams wrote: >> >> Hello, >> >> I'm using Ruby's Shellwords module, which generates a string from an >> array, suitable for shell evaluation. >> >> Ruby's implementation prefers escaping whitespace with a backslash >> rather than quotes. However, this appears to cause some kind of issue >> in Rsync when it computes argv from -e option. > > > The man page for rsync <https://download.samba.org/pub/rsync/rsync.html> > explicitly forbids the use of backslashes: > > "Command-line arguments are permitted in COMMAND provided that COMMAND is > presented to rsync as a single argument. You must use spaces (not tabs or > other whitespace) to separate the command and args from each other, and you > can use single- and/or double-quotes to preserve spaces in an argument (but > not backslashes)." > > So I think the rest of the question is moot ... > > Cheers, Dave > > >> Here is an example command generated by some Ruby code: >> >> rsync --archive --stats -e ssh\ -l\ backup\ -i\ /etc/synco/id_rsa\ -o\ >> ConnectTimeout\\\=60\ -o\ BatchMode\\\=yes --link-dest >> ../../latest/etc/ /etc/ >> example.backup.server.com:/tank/backup/servers/blah/latest.snapshot/etc/ >> >> We can check that something like this is valid: >> >> files% echo foo\ bar\\\=baz >> foo bar\=baz -- what Rsync should be receiving >> files% echo foo bar\=baz >> foo bar=baz -- What Rsync should be executing >> >> However this gives me an error >> >> command-line: line 0: Bad configuration option: connecttimeout\\ >> rsync: connection unexpectedly closed (0 bytes received so far) [sender] >> rsync error: unexplained error (code 255) at io.c(226) [sender=3.1.2] >> >> I think the problem here is the "ConnectTimeout\\\=60", in particular >> how the equals symbol is escaped. >> >> I'm looking in the function: >> >> static pid_t do_cmd(char *cmd, char *machine, char *user, char >> **remote_argv, int remote_argc, >> int *f_in_p, int *f_out_p) >> >> This function splits based purely on whitespace: >> >> args[argc++] = t; >> while (*f != ' ' || in_quote) { >> // consume token... >> >> I feel that this function should also handle backslash escapes. >> >> I also checked using strace and it appears that this is the issue, but >> I'm open to suggestions/ideas. >> >> Kind regards, >> Samuel >> > > -- > Please use reply-all for most replies to avoid omitting the mailing list. > To unsubscribe or change options: > https://lists.samba.org/mailman/listinfo/rsync > Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html