rsync - Feb 2005 - Rsync signatures and incremental tape backup

Hello,

Andrew Trigdell's original rsync paper (actually
thesis) discussed the idea of using rsync to make
incremental tape backups based not on whole files but
rather parts of files. Sadly, this functionality is
not actually present in the rsync program. I'd like to
explore adding this ability.

There are basically three things that need to be done
to enable efficient (partial-file) incremental tape
backups, namely:
1) Generation of rsync signatures from existing files.
2) Generation of a binary patch from existing files +
an rsync signature.
3) Application of a binary patch to existing files.

Rsync already has the facility to generate and apply
binary patches -- namely, batch-mode operation. Thus
all that remains to be added is read and write support
for signatures.

It seems to me the best way is through two new
options: --read-signature and --write-signature.
--read-signature would operate similarly to
--compare-dest. --write-signature would generate
signature files from the destination.

Finally, one would need to be able to run rsync
without an actual destination parameter when used with
--write-batch. Thus we would have the following
combinations:
For a full backup:
--write-batch --write-signature
For a differential backup:
--write-batch --read-signature --write-sigature
For a leaf incremental backup:
--write-batch --read-signature
For a restore, one would specify --read-batch with no
source.

I'm willing to implement this, but wanted to get some
feedback first. Comments?

--Ian Turner


		
__________________________________ 
Do you Yahoo!? 
Yahoo! Sports - Sign up for Fantasy Baseball. 
http://baseball.fantasysports.yahoo.com/

On Thu, Feb 24, 2005 at 04:01:52PM -0800, Richard Patterson
wrote:
> Hello,
> 
> Andrew Trigdell's original rsync paper (actually
> thesis) discussed the idea of using rsync to make
> incremental tape backups based not on whole files but
> rather parts of files. Sadly, this functionality is
> not actually present in the rsync program. I'd like to
> explore adding this ability.
> 
> There are basically three things that need to be done
> to enable efficient (partial-file) incremental tape
> backups, namely:
> 1) Generation of rsync signatures from existing files.
> 2) Generation of a binary patch from existing files +
> an rsync signature.
> 3) Application of a binary patch to existing files.
> 
> Rsync already has the facility to generate and apply
> binary patches -- namely, batch-mode operation. Thus
> all that remains to be added is read and write support
> for signatures.
> 
> It seems to me the best way is through two new
> options: --read-signature and --write-signature.
> --read-signature would operate similarly to
> --compare-dest. --write-signature would generate
> signature files from the destination.
> 
> Finally, one would need to be able to run rsync
> without an actual destination parameter when used with
> --write-batch. Thus we would have the following
> combinations:
> For a full backup:
> --write-batch --write-signature
> For a differential backup:
> --write-batch --read-signature --write-sigature
> For a leaf incremental backup:
> --write-batch --read-signature
> For a restore, one would specify --read-batch with no
> source.
> 
> I'm willing to implement this, but wanted to get some
> feedback first. Comments?
Perhaps this would also be a useful optimization for server mode.
During a period that the servers repository is not changing it could
run from the signature file, which would effectively be a cache of the
checksums. (as long as client didn't request a different checksum seed)


Also, I think this might not be too hard to implement.  Presumably,
the signature file contains exactly what a "write-batch file" to an
empty destination directory would contain, minus that actual file
blocks.  Is that what you're thinking?

-chris