rsync - Jun 2011 - Using rsync as an incremental backup

I'm using rsync to do an incremental backup of my desktop here, to a
remote server as follows:

#/usr/bin/bash

old=$(date -d 'now - 1 week' +%Y-%m-%d)
new=$(date +%Y-%m-%d)

rsync -avP --delete --link-dest=../$dir /home/bakers
bakers at perturb.org:/home/bakers/backup/$new/

This is actually working GREAT! The only problem is that sometimes the
cronjob won't complete (internet is down, something like that). When it
tries to run the next week it does --link-dest against a dir that
doesn't exist. It happily complies and transfers EVERY file because
there is no source to hardlink from.

I'd really like rsync to exit and throw an error if the --link-dest
isn't present. I can't find anything in the man page about any fancy
--list-dest options. Am I missing it?

- Scott

On 06/23/2011 03:40 PM, Lancashire, Pete wrote:
> pretty easy to put a wrapper around the script or add it ..
If it were local that would be easy. This is remote (via SSH), so I'd
have to login before I run rsync to verify that directory is that. I was
hoping I could save that step since rsync is already doing all that already.

A --link-dest-required flag would be great. That way rsync doesn't copy
a bunch of files since there is no hard link source.

- Scott

Scott Baker (scott at perturb.org) wrote on 23 June 2011 15:30:
 >I'm using rsync to do an incremental backup of my desktop here, to a
 >remote server as follows:
 >
 >#/usr/bin/bash
 >
 >old=$(date -d 'now - 1 week' +%Y-%m-%d)
 >new=$(date +%Y-%m-%d)
 >
 >rsync -avP --delete --link-dest=../$dir /home/bakers
 >bakers at perturb.org:/home/bakers/backup/$new/
 >
 >This is actually working GREAT! The only problem is that sometimes the
 >cronjob won't complete (internet is down, something like that). When it
 >tries to run the next week it does --link-dest against a dir that
 >doesn't exist. It happily complies and transfers EVERY file because
 >there is no source to hardlink from.
 >
 >I'd really like rsync to exit and throw an error if the --link-dest
 >isn't present. I can't find anything in the man page about any
fancy
 >--list-dest options. Am I missing it?

You should handle it in your script. You can pass several directories
with link-dest and rsync will search for the file in all of them, in
the order you gave. If it doesn't find one for hardlinking it pulls
the file through the net.

We do it here in our backup script to avoid useless copying, both when
the backup machine starts the process and when the source of the files
starts it.

> I'm using rsync to do an incremental backup of my desktop here, to a
> remote server as follows:
> 
> #/usr/bin/bash
> 
> old=$(date -d 'now - 1 week' +%Y-%m-%d)
> new=$(date +%Y-%m-%d)
> 
> rsync -avP --delete --link-dest=../$dir /home/bakers
> bakers at perturb.org:/home/bakers/backup/$new/
> 
> This is actually working GREAT! The only problem is that sometimes the
> cronjob won't complete (internet is down, something like that). When it
> tries to run the next week it does --link-dest against a dir that
> doesn't exist. It happily complies and transfers EVERY file because
> there is no source to hardlink from.
> 
> I'd really like rsync to exit and throw an error if the --link-dest
> isn't present. I can't find anything in the man page about any
fancy
> --list-dest options. Am I missing it?
Hopefully, Scott's suggestion will solve this issue for you.

You may also be interested in the way LBackup works for hard-linked incremental
backups. It uses a numbering system rather than a date system.

If you still want to have the date named entries within the file system, it is
just a matter of enabling a post-action which keeps a date-named symbolic links
directory to these backup snapshots up to date.

Details on the LBackup post and pre actions are available from the following
URLs :

  - http://www.lbackup.org/pre_and_post_actions (basic intro on LBackup pre/post
actions)
  - http://bit.ly/lIMfAo (source code for the example post action script
mentioned)

This is all just another way of approaching this issue which you may find of
interest. The one issue is that the script (source code link above) will need to
be run on a system which has access to the file system which stores the
snapshots. It would require modification if you wanted to run it on a remote
system.

Hope this helps :^)


---------------------------------------------------------------------
This email is protected by LBackup, an open source backup solution.
http://www.lbackup.org