rsync - Jul 2010 - Rsync backup issues using relative paths and LVM snapshots

Hi everyone,

I have been using rsync for years to perform off-site backups, and have recently
come up with a problem that is starting to hurt my head. I am at the point where
I am starting to think that there is a method to achieve my desired outcome that
I cannot see.

First, a bit of history. I usually just set up a script that runs many rsync
commands in sequence to perform the desired actions. As it is usually an
off-site sync, I pick and choose what directories should be sent over the wire.
I have one site that has about 6 servers, all with custom scripts, and I now
need to perform the off-site sync to more than one location. I could have just
put more into these scripts, but as they were getting quite long, I decided to
streamline the scripts a little.

The new scripts take options from a configuration file, and the sync script will
be the same on all servers, which will make on-going management easier. These
configuration options include a list of directories to be rsync'd. As some
of these directories are a few levels deep, I have been using the -R (relative)
option to automatically generate the full paths on the destination system. Up
until this point, everything is working as expected.

The problem arises when I have a directory that will more than likely change
between each sync. One example of this is my mail spool directory. Since emails
come in all the time, the sync will be marked as failed most of the time. The
way I was dealing with this in my sequential scripts was to take a LVM snapshot,
and then rsync the snapshot to the destination server.

My problem is that if I mark a directory to have a snapshot created before rsync
and use the -R (relative) option, the directory structure on the destination
system will be the relative path of where I mounted the snapshot
(/mnt/sync-snapshot in my case). If I don't use the -R option, and I am
trying to back up a snapshot of the directory /var/spool/cyrus, how can I get
rsync to create the /var/spool directories for me before the sync?

The following works, but creates /mnt/sync-snapshot on the destination server
root at percy:/# rsync -auRv /mnt/sync-snapshot ds9::kingston

The following doesn't work because /var/spool does not exist on the
destination server, and therefore the mkdir fails.
root at percy:/# rsync -auv /mnt/sync-snapshot ds9::kingston/var/spool/cyrus/
sending incremental file list
rsync: mkdir "/var/spool/cyrus" (in kingston) failed: No such file or
directory (2)
rsync: read error: Connection reset by peer (104)
rsync error: error in rsync protocol data stream (code 12) at io.c(760)
[sender=3.0.7]

Creating the directories manually on the destination server is not desirable. I
really want to put the logic in the script so it is easy to bring another backup
location online easily.

Thanks for your time.

Alex Ferrara
Director
Receptive IT Soutions

> I really want to put the logic in the script so it is easy to bring another
backup location online easily.
If you have shell access to the destination system from your backup script then
one option may be to issue 'mkdir -p' via ssh.
> Creating the directories manually on the destination server is not
desirable
If you using mkdir command via ssh is what you are referring to as a manual
process then okay another approach will be required.

If your OS has support for sparse bundle images then one possible approach could
be staging the backup to local media and then once finished unmount the sparse
bundle image and then sync this image your remote locations. Below is a link to
a script (which currently only supporting Mac OS X) which will synchronize a
sparse bundle image to a remote server.

I would also suggest that a pull backup strategy may also be worth considering. 

  - http://www.lbackup.org/network_backup_strategies

If you are distributing your backups to multiple systems one advantage of a pull
backup strategy is the ability to move the job of distribution to the backup
destination system.



----------------------------------
This email is protected by LBackup
http://www.lbackup.org

Alex Ferrara wrote:
> My problem is that if I mark a directory to have a snapshot created
> before rsync and use the -R (relative) option, the directory
> structure on the destination system will be the relative path of
> where I mounted the snapshot (/mnt/sync-snapshot in my case). If I
> don't use the -R option, and I am trying to back up a snapshot of
> the directory /var/spool/cyrus, how can I get rsync to create the
> /var/spool directories for me before the sync?
> 
> The following works, but creates /mnt/sync-snapshot on the destination
server
> root at percy:/# rsync -auRv /mnt/sync-snapshot ds9::kingston
mkdir -p /mnt/alt-snapshot/var/spool/cyrus
mount /mnt/sync-snapshot /mnt/alt-snapshot/var/spool/cyrus -t bind
rsync -auRv /mnt/alt-snapshot/./var/spool/cyrus ds9::kingston
umount /mnt/alt-snapshot/var/spool/cyrus

-- Jamie