I develop GNU Parallel. GNU Parallel uses rsync to transfer files. GNU Parallel has a design goal of not requiring the users to change their setup to be able to use GNU Parallel. In other words: If you are called as a consultant to work on a Centos3 server untouched since 2007, then you can expect even the newest version of GNU Parallel will work flawlessly. Centos3 has rsync 2.5.7 installed. On my development machine I have so far been using rsync prior to 3.1.0 and there has been no hiccups: Rsync has worked flawlessly against Centos3. Yesterday I upgraded my development machine, which also upgraded rsync to 3.1.0. This breaks rsync against Centos3. I get the following error: rsync-3.1.0 protocol version mismatch - is your shell clean? rsync-3.1.0 (see the rsync man page for an explanation) rsync-3.1.0 rsync error: protocol incompatibility (code 2) at compat.c(62) rsync-3.1.0 rsync: connection unexpectedly closed (0 bytes received so far) [sender] rsync-3.1.0 rsync error: error in rsync protocol data stream (code 12) at io.c(226) [sender=3.1.0] This is also the case for 3.1.1: rsync-3.1.1 protocol version mismatch - is your shell clean? rsync-3.1.1 (see the rsync man page for an explanation) rsync-3.1.1 rsync error: protocol incompatibility (code 2) at compat.c(62) rsync-3.1.1 rsync: connection unexpectedly closed (0 bytes received so far) [sender] rsync-3.1.1 rsync error: error in rsync protocol data stream (code 12) at io.c(226) [sender=3.1.1] But this is not the case for 3.0.[0-9]: They all work fine against rsync 2.5.7. The workaround seems to be adding --protocol 30 if you are running rsync 3.1.x against 2.5.7, but that workaround fails when running 2.5.7 against 2.5.7, so it cannot always be used: [centos3]$ rsync --protocol 30 /tmp/foo localhost: rsync: --protocol: unknown option rsync error: syntax or usage error (code 1) at main.c(994) Obviously as a consultant you cannot require the customer to neither upgrade nor downgrade their rsync installation, just so that you can use GNU Parallel. I am quite surprised that rsync-3.1.x does not simply negotiate the protocol and downgrade to protocol 30 automatically when 31 fails. And I find the error message could be more helpful in finding the workaround. My question is now: What should I do as a developer? How do I make sure that GNU Parallel's use of rsync never fails because of protocol issues when I cannot predict which version of rsync is used by the remote machine? If this is a bug that will be fixed in 3.1.2, then that solves it for future versions. But there will still be machines with 3.1.[0-1] out there for years to come. How do your recommend I support those in GNU Parallel? Do you have a compatibility matrix between all version of rsync? Are other incompatibilities (that will be turned on by default and which does not get negotiated with the remote) planned? /Ole
On Fri, Jul 25, 2014 at 11:50 AM, Ole Tange <tange at gnu.org> wrote:> I am quite surprised that rsync-3.1.x does not simply negotiate > the protocol and downgrade to protocol 30 automatically when 31 fails.Sadly, it can't do any of that because it is the 2.5.7 version that is killing the connection due to the advent of protocol 31 -- 2.5.7 has some code in it to die if what it thinks is a protocol number isn't in a particular range of values that maxes out at protocol 30. And I find the error message could be more helpful in finding> the workaround. >Again, it's the old 2.5.7 assuming that the bytes it is encountering are crap instead of valid data, which usually means that some ascii text got into the transmission stream. A sad effect of having no decent starting byte sequence to the protocol, and no way to change it without becoming incompatible with all prior versions of the software. My question is now: What should I do as a developer?>There aren't any particularly easy answers for dealing with an old bit of software that is refusing to play nice with newer software. One possibility is for your wrapping code to do a configure pass that checks the rsync version on both of the hosts and figures out if one is too old (2.6.0 and newer are fine, so only 2.5.7 and older have this issue) and one is too new (3.1.0 is the first to start using protocol 31). If that occurs, you can specify the --protocol=30 option on the newer rsync to avoid the issue. Depending on the client's willingness to tweak their rsync, the fix for such an old 2.5.7 would really be as simple as changing one number in the executable from comparing against 30 to comparing against 40. If someone were to take the code for 2.5.7 and tweak that one use of the MAX_PROTOCOL_VERSION (it only gets used in compat.c) then the 2.5.7 version would work fine. That would essentially be a 1-byte patch to the executable, but obviously you can't expect everyone to be willing to apply such a change, even if you were able to figure out exactly what byte to twiddle. ..wayne.. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.samba.org/pipermail/rsync/attachments/20140725/f755837b/attachment.html>