rsync - Apr 2008 - Feature request: Store diffed files separately.

Hello all,

I'm going to feel dumb if rsync can already do this, but I honestly
haven't found anything to indicate that.  I would like to be able to
store diffed files in a separate directory on the receiver so it makes
doing an incremental backup easy and efficient.  Let me explain:

Day 1: take a full backup of a directory: rsync -r ./source/ ./dest/
Day 2: Diff the contents of ./source/ and ./dest/ and store only the
new/modified files in ./incremental_day1
Day 3: Diff the contents of ./source/ and ./dest/ and store only the
new/modified files in ./incremental_day2
etc.

This way to restore a backup of Day 2 just do: rsync -Pvr
./incremental_day2 ./dest/ # But make the updates to ./source/ instead
so you can preserve existing backups.  Does this make sense?  I
thought something like this might be useful since your on-going
incremental backups aren't storing unchanged files again which will
save space.

Ben Wilber

> -----Original Message-----
> From: rsync-bounces+brock=kuhsefamily.org@lists.samba.org
> [mailto:rsync-bounces+brock=kuhsefamily.org@lists.samba.org] On Behalf
> Of Ben Wilber
> Sent: Tuesday, April 01, 2008 4:12 PM
> To: rsync@lists.samba.org
> Subject: Feature request: Store diffed files separately.
> 
> Hello all,
> 
> I'm going to feel dumb if rsync can already do this, but I honestly
> haven't found anything to indicate that.  I would like to be able to
> store diffed files in a separate directory on the receiver so it makes
> doing an incremental backup easy and efficient.
Use hard links (--link-dest argument, IIRC) to link your full backup to the
incremental backup directory, if your file system supports hard links.  Then
every backup will look like a full backup, but take up the disk space of an
incremental (more or less).

- Brock

On Tue, 2008-04-01 at 16:12 -0700, Ben Wilber wrote:
> I would like to be able to
> store diffed files in a separate directory on the receiver so it makes
> doing an incremental backup easy and efficient.  Let me explain:
For completeness, I'll point out that you can accomplish what you
originally wanted with --compare-dest, although I think the --link-dest
approach Brock mentioned is better.
> Day 1: take a full backup of a directory: rsync -r ./source/ ./dest/
> Day 2: Diff the contents of ./source/ and ./dest/ and store only the
> new/modified files in ./incremental_day1
rsync -r --compare-dest=../dest/ ./source/ ?./incremental_day1/
> Day 3: Diff the contents of ./source/ and ./dest/ and store only the
> new/modified files in ./incremental_day2
I take you to mean that incremental_day2 contains files created/modified
since the original dest, not since incremental_day1.  The command is
similar:

?rsync -r --compare-dest=../dest/ ./source/ ?./incremental_day2/

If you wanted to create "chained" incremental backups, the command is:

rsync -r --compare-dest=../incremental_day1/ \
    --compare-dest=../dest/ ./source/ ./incremental_day2/

Unfortunately, the semantics of --compare-dest are not quite right: this
command will omit files that changed back to their original state on day
2, but those files are needed to restore day 2 correctly.
> This way to restore a backup of Day 2 just do: rsync -Pvr
> ./incremental_day2 ./dest/ # But make the updates to ./source/ instead
> so you can preserve existing backups.
The restore command would be:

rsync -Pvr ./incremental_day2/ ./dest/ ./source/

When you give multiple source arguments, files from earlier source
arguments take priority over corresponding files from later source
arguments.  Thus, the command will restore the incremental_day2 version
of a modified file, not the dest version.

Files that are present in dest but *deleted* in incremental_day2 will be
restored, which is wrong.  To avoid this, you could save a "find ."
listing of the source in each incremental backup and restore only the
files in this listing using --files-from.

Matt