rsync - Jul 2010 - rsync mirror solution: how to prevent accidental mirror deletion

I had a recent disaster scenario with rsync. I was wondering if there
were any suggestions to guard against in the future:

I used to maintain "mirror" backups of the /home dir on our
production_server using rsync to a backup_server.

The primary server had a rsyncd daemon running and the backup_server
had this line in the crontab:

10 01 * * * rsync -av  --delete root at production_server::home
/production_server_home_bkup

Things worked fine and whenever we tested backup_server it always had
a faithful image of the production_server. Unfortunately, one day the
RAID array on primary_server had a failure. As a result /home mount
was lost temporarily. This happened over the weekend and by the time
we got around to checking, the backup_server had a chance to run its
cron job. This job merrily erased every file on the backup since it
thought /home was now supposed to be empty when it synced the two.
Luckily we could recover the RAID so all is well now.

But how should I change our procedures to guard against this (or
similar) outcomes? Is there a better way out for a mirror via rsync?

-- 
Rahul

Hi Rahul,

I have two suggestions:
1. Use --max-delete option to limit the amount of damage that can be
done when this happens.
2. Add some redundancy to your backup by keeping multiple past backups
and rotating between them.  You can use --link-dest to allow sharing
of unchanged files between backups.  A set of perl scripts called
rsnapshot (http://www.rsnapshot.org/) makes this easy to set up.

David

On 30 July 2010 10:41, Rahul Nabar <rpnabar at gmail.com>
wrote:
> I had a recent disaster scenario with rsync. I was wondering if there
> were any suggestions to guard against in the future:
>
> I used to maintain "mirror" backups of the /home dir on our
> production_server using rsync to a backup_server.
>
> The primary server had a rsyncd daemon running and the backup_server
> had this line in the crontab:
>
> 10 01 * * * rsync -av ?--delete root at production_server::home
> /production_server_home_bkup
>
> Things worked fine and whenever we tested backup_server it always had
> a faithful image of the production_server. Unfortunately, one day the
> RAID array on primary_server had a failure. As a result /home mount
> was lost temporarily. This happened over the weekend and by the time
> we got around to checking, the backup_server had a chance to run its
> cron job. This job merrily erased every file on the backup since it
> thought /home was now supposed to be empty when it synced the two.
> Luckily we could recover the RAID so all is well now.
>
> But how should I change our procedures to guard against this (or
> similar) outcomes? Is there a better way out for a mirror via rsync?
>
> --
> Rahul
> --
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>

On Thu, 2010-07-29 at 19:41 -0500, Rahul Nabar wrote:
> I used to maintain "mirror" backups of the /home dir on our
> production_server using rsync to a backup_server.
> 
> The primary server had a rsyncd daemon running and the backup_server
> had this line in the crontab:
> 
> 10 01 * * * rsync -av  --delete root at production_server::home
> /production_server_home_bkup
> 
> Things worked fine and whenever we tested backup_server it always had
> a faithful image of the production_server. Unfortunately, one day the
> RAID array on primary_server had a failure. As a result /home mount
> was lost temporarily. This happened over the weekend and by the time
> we got around to checking, the backup_server had a chance to run its
> cron job. This job merrily erased every file on the backup since it
> thought /home was now supposed to be empty when it synced the two.
If you can arrange for the source directory to appear nonexistent rather
than empty in the event of a disk failure, rsync would exit with code 23
instead of cleaning out the destination.  One hacky way to do that would
be to put a symlink "me -> ." in the source directory and specify
the
source as root at production_server::home/me/ .

-- 
Matt