rsync - Apr 2011 - rsync in theory.

I am looking at add some code to the rsync tool but want to
know if I am "totally out to lunch".

I realize that my example is so trivial that I am sure I will
get replies of "don't do it that way", but bear in mind
that it is just an example, and there are real world cases
where I think this functionality would be useful.

I am trying to figure out if rsync can do something like

cat myfile.dat | rsync - remoteHost:/some/path/myfile.dat

which would take a stream of data and send/store it onto the
remote host.

I don't think that this is possible with the current command line
rsync utility. My questions is more about "can the rsync algorithm
do this?".

Technically, the question boils down to "Is rsync a single pass
algorithm, or is it a multi-pass algorithm?"

If it is a single pass algorithm then all is good.

If it is a multi-pass algorithm then how big of a buffer does it
need to perform the passes? Namely, is it a block by block multi-pass,
or is it a complete file/object multi-pass algorithm.

Any help or pointers to this detail are appreciated.

-- 
Evan

On 12/04/11 01:58, Evan Rempel wrote:
>
> I am looking at add some code to the rsync tool but want to
> know if I am "totally out to lunch".
>
> I realize that my example is so trivial that I am sure I will
> get replies of "don't do it that way", but bear in mind
> that it is just an example, and there are real world cases
> where I think this functionality would be useful.
>
> I am trying to figure out if rsync can do something like
>
> cat myfile.dat | rsync - remoteHost:/some/path/myfile.dat
>
> which would take a stream of data and send/store it onto the
> remote host.
Not possible with rsync (as far as I know), but is possible with 
librsync (which is a completely different code base than rsync
itself).
>
> My questions is more about "can the rsync algorithm
> do this?".
>
As far as I can tell, you need two passes on one end (either the 
receiving or the sending). There is no reason for the other end to be 
completely one pass.
> Technically, the question boils down to "Is rsync a single pass
> algorithm, or is it a multi-pass algorithm?"
>
> If it is a single pass algorithm then all is good.
>
> If it is a multi-pass algorithm then how big of a buffer does it
> need to perform the passes?
The definition of "one pass" is "can be performed with one
reading of
the file and a O(1) buffer". If you can answer the question, then it is, 
by definition, one pass.
> Namely, is it a block by block multi-pass,
> or is it a complete file/object multi-pass algorithm.
Again, the question is meaningless. If you can apply an algorithm one 
block at a time, then it's one pass by definition.

Shachar

-- 
Shachar Shemesh
Lingnu Open Source Consulting Ltd.
http://www.lingnu.com