rsync - Sep 2009 - LVM snapshots vs. --link-dest

I currently do incremental backups using --link-dest.  Unchanged files 
are hard links to the previous snapshot; changed files are new copies.

Where this "fails" is for large files that have received small
changes.
The directory containing my main IMAP account, for example, typically 
generates between 1 and 2 G of daily backup data as I file messages in my 
inbox.  Yesterday, though, I filed some old messages in my inbox.  This 
touched files I don't typically touch.  They were minor changes in the 
scheme of things, but the result was an 11G backup of that volume last 
night.

I was thinking that an alternative to links, which do nothing to preserve 
space when small file changes have been made, would be using LVM 
snapshots.  Instead of creating a new directory for a new backup, and 
specifying --link-dest to the previous directory, I'd create a snapshot 
of the current backup volume to preserve it, and then do a normal (ie. w/
o --link-dest) rsync to the "live" volume for the new backup.

Has anyone done anything like this?  I'm curious about the experiences of 
others, pro and con.

I can see one immediate problem: this really only works with a local file 
system.  On one of our backup servers, for example, the volumes to which 
backups are written are mounted via NFS from a file server.  So the 
backup server has no ability to take snapshots as far as I can see.

Am I missing something?

But I can think of at least one work-around: Switching to iSCSI would 
move the file system to the backup server, even if the storage itself 
were still remote.

Anything better than this?

For the backup servers that use local storage, though, using LVM 
snapshots seems like a decent solution.  But I'm worried about managing 
"little details", like assuring that there's enough space in the
snapshot
to handle the changes to the live volume.

That's why, before I start experimenting, I thought I'd ask what others 
have done/tried/abandoned.

Thanks...
	
	Andrew

On Sun, 27 Sep 2009 17:14:42 +0000, Andrew Gideon wrote:

> Where this "fails" is for large files that have received small
changes.
> The directory containing my main IMAP account, for example, typically
> generates between 1 and 2 G of daily backup data as I file messages in
> my inbox.  Yesterday, though, I filed some old messages in my inbox. 
> This touched files I don't typically touch.  They were minor changes in
> the scheme of things, but the result was an 11G backup of that volume
> last night.
Several people have emailed me suggesting that I change mailbox formats.  
Rather than replying to each of them individually...

You're all quite right.  I should (and eventually will) make that 
change.  I do appreciate the suggestion.

But my mailbox case was just one example of a general issue: --link-dest 
doing nothing to save backup space in the case where a large file is 
subjected to a small change.  LVM snapshots may offer a solution to this 
general problem, and it is that idea that I'm hoping to explore here.

Thanks again...

	Andrew

On Sun, 27 Sep 2009 17:14:42 +0000, Andrew Gideon wrote:
> I was thinking that an alternative to links, which do nothing to
> preserve space when small file changes have been made, would be using
> LVM snapshots.  Instead of creating a new directory for a new backup,
> and specifying --link-dest to the previous directory, I'd create a
> snapshot of the current backup volume to preserve it, and then do a
> normal (ie. w/ o --link-dest) rsync to the "live" volume for the
new
> backup.
I've been thinking a little more about this.  It assumes that --inplace 
actually does what it sounds like it does: it modifies only the disk 
pages of a file that have changed, as opposed to changing the entire file 
or creating a new file or some other thing which causes the file to 
"move" on disk.

Does --inplace do what it sounds like?  I was reading a dialog from 2004 
which claimed that adding this to rsync would be tough.

The --update option is also useful in this regard?  What would happen if 
--inplace were used but --update were not?

Thanks...

	Andrew

On Mon 28 Sep 2009, Andrew Gideon wrote:
> 
> The --update option is also useful in this regard?  What would happen if 
> --inplace were used but --update were not?
--update only influences what files are considered for transfer, not how
that transfer is performed; hence it is completely irrelevant to how
--inplace works.


Paul

Andrew Gideon wrote:
> I currently do incremental backups using --link-dest.  Unchanged files 
> are hard links to the previous snapshot; changed files are new copies.
>
> Where this "fails" is for large files that have received small
changes.
> The directory containing my main IMAP account, for example, typically 
> generates between 1 and 2 G of daily backup data as I file messages in my 
> inbox.  Yesterday, though, I filed some old messages in my inbox.  This 
> touched files I don't typically touch.  They were minor changes in the 
> scheme of things, but the result was an 11G backup of that volume last 
> night.
>   
If you resist using a fileformat which has some kind of implicit 
"chunking" capability (eg maildir isn't perfect, but kind of goes
in the
opposite direction and chunks your mail into lots of smaller files - 
perhaps too many in the opinion of some...), then you really need a 
backup system which can "chunk" your backup files and only store the 
differences

There are a few like rsnapshot which might suit your needs.  Or if you 
want something different then perhaps try brackup, which is kind of 
rsync like, but has a "chunking" phase built into it which tries to be
rather smart about dividing files up into the hopefully static bit and 
the variable bit.  eg in the case of MP3s it would create one chunk for 
the data and one chunk for the tagging, that way you can retag your mp3 
collection and it won't re-upload the whole lot

Ideally I would like to see a kind of half and half maildir/mbox format 
emerge for email (perhaps dbox from dovecot will get there?).  The idea 
is that it would use mbox for storage, but break the files up into 
chunks of about a couple of MB for each mbox file.  This way you would 
have much smaller files to defrag in the case of deletes, but you would 
gain the packing efficiency of mbox (especially if you use compressed 
mailboxes, eg dovecot)

Good luck

Ed W

On Mon, 2009-09-28 at 14:18 +0000, Andrew Gideon wrote:
> It assumes that --inplace 
> actually does what it sounds like it does: it modifies only the disk 
> pages of a file that have changed, as opposed to changing the entire file 
> or creating a new file or some other thing which causes the file to 
> "move" on disk.
Yes, --inplace works that way:

http://gitweb.samba.org/?p=rsync.git;a=blob;f=receiver.c;h=965bb3909dc75d6b4ba675e201a64549ab8ac8dc;hb=HEAD#l295

-- 
Matt