rsync - Nov 2005 - Rsync, subversion and directories

Hi all,
        I'm attempting to build a backup system using rsync and subversion.

I have a samba share which I would like to replicate via rsync.  The
replicated copy is under subversion control to store a 'history'. 
This appears to work well until a directory is removed on the primary
(non-backup) file system.

The rsync command which I assumed would do the job:
$ rsyc -zrv --delete --force --exclude=".svn" SRC DEST

Since the destination folders are under subversion control, each
folder has a ".svn" folder holding the versioning information.  I
would prefer these not be deleted each time rsync ran, hence the
--exclude=".svn" flag.  However, when a folder is removed on the
source machine I would like that folder to disappear on the
destination machine, ".svn" subdirectory and all.  Reading the short
description of "--force", it appears this flag should do the trick -
further investigation reveals this is not quite so.

Furthermore, the output of the "-v" flag doesn't imply that the
directory can be deleted, except when doing a dry run:

$ rsync -rvz --delete --exclude=".svn" --dry-run  ~/TestA
~/testme/TestB
building file list ... done
deleting TestA/TEST/     <-------
TestA/blerg.txt
TestA/file with space.txt
TestA/file2.txt
TestA/sync.sh

$ rsync -rvz --delete --exclude=".svn" ~/TestA ~/testme/TestB
building file list ... done
TestA/blerg.txt
TestA/file with space.txt
TestA/file2.txt
TestA/sync.sh

I was wondering, short of moving the subversion server to the source
machine, or running rsync twice and doing some output matching, how
this might be accomplished?

Cheers,
Andrew

On Fri, 2005-11-11 at 16:20 +1100, Andrew Crouch wrote:
> I'm attempting to build a backup system using rsync and subversion.
> [...]
Are you sure you want to be using Subversion and not some other
mechanism?

If I understand correctly, during each backup, you rsync an ordinary
file tree into a Subversion working copy and then commit the working
copy to the repository.  The trouble with this arrangement is that
Subversion almost always needs to be notified when files in a working
copy are created, moved, or deleted (with `svn add', `svn move', and
similar) in order for committing to work correctly.  I imagine you have
an automated script that runs some `svn ___' commands to add and remove
files from version control, but deletion of directories is probably one
of several cases that it cannot handle.  (Not to mention, what happens
if a Subversion working copy is among the things you are backing up?)

A common incremental backup package such as rsnapshot or rdiff-backup
would probably do a better job of keeping a "history" than Subversion.
Either of these tools will let you send the data from the source machine
to a repository of incremental backups in one step instead of having to
move it to the backup machine and then "check it in" somehow.
-- 
Matt McCutchen, ``hashproduct''
hashproduct@verizon.net -- http://mysite.verizon.net/hashproduct/

On Fri, Nov 11, 2005 at 04:20:09PM +1100, Andrew Crouch
wrote:
> Reading the short description of "--force", it appears this flag
> should do the trick - further investigation reveals this is not quite
> so.
The --force option doesn't do anything extra if --delete was already
specified (--force would allow a file to replace a directory without
the --delete option).  When you tell rsync to protect some files,
rsync will not remove the containing directory that holds the protected
files.
> Furthermore, the output of the "-v" flag doesn't imply that
the
> directory can be deleted, except when doing a dry run:
Yes, because --dry-run is not smart enough to know that the directory's
deletion is being prevented by the preservation of some excluded
content.
> I was wondering, short of moving the subversion server to the source
> machine, or running rsync twice and doing some output matching, how
> this might be accomplished?
You could always run a simple perl script on the resulting dirs and let
it remove any directory that only contains a .svn dir (which would allow
you to run an svn delete command at that time).  For instance, this
script finds these empty dirs and runs "rm -rf" on them:

#!/usr/bin/perl
use strict;

my %dirs;

open(IN, 'find /dest/path/foo |') or die $!;
while (<IN>) {
    next if m#/.svn/#;
    chomp;
    my($path, $name) = m#^(.*)/([^/]+)$#;
    if ($name eq '.svn') {
	$dirs{$path} += 0;
    } else {
	$dirs{$path}++;
    }
    print "$path = $dirs{$path}\n";
}

foreach (reverse sort keys %dirs) {
    if ($dirs{$_} == 0) {
	system "rm -rf '$_'";
	# Handle a nested sequence of dirs going empty.
	my($path) = m#^(.*)/#;
	$dirs{$path}--;
    }
}

..wayne..