Adding support for versioned files in rsync

Andy Henson
2003-10-14T02:18:00Z

I agree version support should be added to rsync, indeed I was about to
propose an extension, but I start from a very different point of view.

We already use versioned files.  For example, the rsync download is called
rsync-2.5.6.tar.gz and the previous one is rsync-2.5.5.tar.gz.

I agree some systems may prefer a ;1 syntax for versioning, but rsync can
(should?) treat this as just part of the file name, an underlying
versioning file system can do the rest.

What we do need to do is provide a version of rsync which permits the
following efficiently using the rsync protocol:

$ ls rsync*
rsync-2.5.5.tar.gz
$ rsync remote.host:rsync-2.5.6.tar.gz .

Now at present this will transfer the entire file, not just the
differences.  Now I can write clever code to overcome this, indeed I do
use a script with the following when transferring CD ISOs:

BASE=rsync-2.5.5.tar.gz
NEW=rsync-2.5.6.tar.gz
if [ ! -f $NEW ]; then
        echo copying...
        cp -l $BASE $NEW
elif [ `stat -t $NEW | cut -d' ' -f2` -lt 600334000 ]; then
        echo appending...
        cat $BASE >>$NEW
fi
rsync -vvtPB remote.host:$NEW .

But that's not very good or friendly - I'm creating a rogue copy of the
old version under the new version's name which I hope will soon go.
If it partially copies and is restarted, say 300MB into the copy, it
appends the old to the partial new file the second time around (if >600MB
it might be all transferred ok so it doesn't alter it).

But this doesn't work if uploading to a machine which I have no shell
access to.  It really should be in rsync itself.

I propose a new option:
--comp-file=<file>

allowing:
rsync -vvtPB --comp-file=$BASE remote.host:$NEW .

Then the receiving rsync will generate checksums on *both* the old BASE
file *and* the any partially transferred NEW file to match against the NEW
file on the sending end.  Thus a 600MB ISO image which is 95% unchanged,
and a 300MB partial transfer can be both used as match material for the
retransfer.

I'm highly tempted to permit the default if not -comp-file is specified to
be "look for a similarly named file with just a number changed", but
perhaps we need an option for that too :)

I've had a read through the relevant source code and I am confident I
could add this functionality.  I was waiting to see what others are doing
before starting anything...

Andy Henson