rsync - Feb 2006 - RSYNC via pipe/socket ?

Hello !

I`m trying to find a way to use lzo compression for the data being transferred
by rsync.

rsync only supports gzip (or maybe ssh -c, which is also zlib), which gives cpu
intensive workload - and
this cannot be exchanged by a commandline param like we have with tar
(--use-compress-program=....)

i like to use lzo because it does much faster compression than gzip.


I read the following in the mailinglist archive:

----------
> Is it possible to use rsync without ssh for those machines that don't
> have ssh?
If they don't have ssh, what *do* they have?  It is certainly possible
to use rsync over insecure-berkeley-rsh (sometimes called remsh) and
over kerberized rsh (v4 or v5), or basically anything with an rsh-like
syntax...  if you were really clever you could write a shell-script
wrapper that called netcat, if you had a prearranged netcat invocation
on the other end :-)

This ignores the "rsync server" mode, where rsync talks to rsyncd
directly and doesn't use an existing remote-shell channel...
----------


so, this gives me some hope that this _could_ possible somehow.

unfortunately i don`t have a clue, how to write the mentioned wrapper script.

i think of something like this:

rsync -a  -dontConnect2RemoteHostButPipeToSTDOUT ./localdata2sync | lzop -c |
netcat remothost 12345

where on remotehosts netcat listens to port 12345 and passes the data to lzop
and then to rsync.....


otoh, thinking about this, i`m not sure if a pipe could work, because rsync does
bidirectional communication with the "remote" end.


is something like this possible, or yet existing - or maybe someone can give
some tips, how to do this ?

regards
roland
-------------- next part --------------
HTML attachment scrubbed and removed

On Fri, 2005-12-09 at 00:46 +0100, roland wrote:
> I`m trying to find a way to use lzo compression for the data being
> transferred by rsync.
It's easy set this up.  Consider this (for gzip but it's easy to do the
same for any compression program):

/usr/local/bin/gzip-wrap:
	#!/bin/bash
	gzip | "$@" | gunzip

/usr/local/bin/gunzip-wrap:
	#!/bin/bash
	gunzip | "$@" | gzip

Then run:
	rsync --rsh='gzip-wrap ssh' --rsync-path='gunzip-wrap rsync'
		<options source dest>

As elegant as this technique is, it fails because compression programs
perform internal buffering.  One rsync will send a block of data and
wait for an acknowledgement that the other side has received it, but
since the end of the data is buffered in the compression program, the
other side never responds and deadlock results.  There might be a way
around this, but I can't think of one.

The next best thing is to use rsync to generate a batch file on the
sender, bring the batch file to the receiver by hand (compressing and
decompressing), and apply the batch file on the receiver using rsync.
This effectively lets you compress the mainstream file-data part of the
transmission, but much of the metadata must still go over the wire
during the creation of the batch file.  See the man page for details.
-- 
Matt McCutchen
hashproduct@verizon.net
http://mysite.verizon.net/hashproduct/

hello matt, 

thank you for your reply.

as is see, the method you describe is just "theoretical", because it
won`t work due to buffering issue.
furthermore, it still needs ssh or maybe another remote shell.

i'd like to leave out ssh or probably any remote shell entirely because
encryption is slow and needs cpu.
i read that there is a patch to openssh that you can use -c none, i.e. Cipher
none for using ssh without
encryption, but this is no option because i cannot install patched ssh on all
machines.
> The next best thing is to use rsync to generate a batch file on the
> sender, bring the batch file to the receiver by hand (compressing and
> decompressing), and apply the batch file on the receiver using rsync.
> This effectively lets you compress the mainstream file-data part of the
> transmission, but much of the metadata must still go over the wire
> during the creation of the batch file.  See the man page for details.

       $ rsync --write-batch=pfx -a /source/dir/ /adest/dir/
       $ rcp pfx.rsync_* remote:
       $ ssh remote rsync --read-batch=pfx -a /bdest/dir/
       # or alternatively
       $ ssh remote ./pfx.rsync_argvs /bdest/dir/

ah - i understand. quite interesting and cool stuff, but unfortunately it should
need tons
of temporary space if you do the first rsync or if there is significant diff
between the two
hosts.

conclusion:
both methods seem to be no option for me at the moment.

any chance to see "pluggable compression" in rsync one day ? maybe
it`s a planned feature ?

regards
roland



Matt McCutchen <hashproduct@verizon.net> schrieb am 09.02.06
02:02:47:
> 
> On Fri, 2005-12-09 at 00:46 +0100, roland wrote:
> > I`m trying to find a way to use lzo compression for the data being
> > transferred by rsync.
> 
> It's easy set this up.  Consider this (for gzip but it's easy to do
the
> same for any compression program):
> 
> /usr/local/bin/gzip-wrap:
> 	#!/bin/bash
> 	gzip | "$@" | gunzip
> 
> /usr/local/bin/gunzip-wrap:
> 	#!/bin/bash
> 	gunzip | "$@" | gzip
> 
> Then run:
> 	rsync --rsh='gzip-wrap ssh' --rsync-path='gunzip-wrap
rsync'
> 		<options source dest>
> 
> As elegant as this technique is, it fails because compression programs
> perform internal buffering.  One rsync will send a block of data and
> wait for an acknowledgement that the other side has received it, but
> since the end of the data is buffered in the compression program, the
> other side never responds and deadlock results.  There might be a way
> around this, but I can't think of one.
> 
> The next best thing is to use rsync to generate a batch file on the
> sender, bring the batch file to the receiver by hand (compressing and
> decompressing), and apply the batch file on the receiver using rsync.
> This effectively lets you compress the mainstream file-data part of the
> transmission, but much of the metadata must still go over the wire
> during the creation of the batch file.  See the man page for details.
> -- 
> Matt McCutchen
> hashproduct@verizon.net
> http://mysite.verizon.net/hashproduct/
> 

______________________________________________________________
Verschicken Sie romantische, coole und witzige Bilder per SMS!
Jetzt bei WEB.DE FreeMail: http://f.web.de/?mc=021193