rsync - Jun 2006 - rsync --delete problem

Hello,

 

I need to create a local backup every night of a filesystem that has
about 3.5 million files.  I've been using rsync and everything has been
fine until I started to use the -delete option to delete files on the
backup that don't exist in the production data.  When I added the
-delete option, my backup started taking about two days to finish.  So I
changed my backup script to rsync individual top-level directories
instead of the whole filesystem at once.  This finished must faster,
except now rsync seems to be only copying files that have changed on the
production data filesystem, and deleting files on my backup that haven't
changed.

 

Here are the three different scripts that I've tried and they all do the
same thing:

 

1:

cd /backup/directory

rsync -au --delete /path/to/files/a* .

rsync -au --delete /path/to/files/b* .

rsync -au --delete /path/to/files/c* .

...

 

2:

cd /backup/directory 

rsync -au --delete --include /path/to/files/a* --exclude /path/to/files/
/path/to/files/a* .

rsync -au --delete --include /path/to/files/b* --exclude /path/to/files/
/path/to/files/b* .

rsync -au --delete --include /path/to/files/c* --exclude /path/to/files/
/path/to/files/c* .

...

 

3:

cd /backup/directory

for file in /path/to/files/*; do rsync -au --delete $file .; done

 

 

Does anyone know what I'm doing wrong here?

 

Thanks,

 

Richard

-------------- next part --------------
HTML attachment scrubbed and removed

On Thu, Jun 01, 2006 at 09:31:49AM -0500, Smemoe, Richard L.
wrote:
> When I added the -delete option, my backup started taking about two
> days to finish.  
You'd be better off using --del (--delete-during) than --delete, as
it doesn't require a separate scan through the destination hierarchy.
Also, older versions of rsync created an additional in-memory
representation of the entire destination file tree to do the delete,
so it is possible that such an older version caused rsync to start
swapping, which could easily slow it down by an order of magnitude.
So, make sure you're using a modern version (at least 2.6.4).

As for any incorrect behavior of --delete, you'll need to be more
explicit in what you think it is doing wrong.

..wayne..

Wayne,

Thanks for your reply.  I upgraded to 2.6.8 and switched to
--delete-during.  But I still have the same problem as I did with
--delete.  I'll try to explain the behavior I'm seeing a little better.
The directory I'm trying to rsync from has ~3.5 million files with the
upper level directory having about 2000 directories.  I'm running 'cd
/backup/directory', then 'rsync -a --delete-during
/backup/directory/directorytocopy .' on every directory in this top
level directory.  If I leave off the --delete-during (or --delete),
everything works fine (except files removed from the original filesystem
aren't deleted), but if I use the --delete-during option, I only end up
with a small fraction of the original filesystem in my backup.  It looks
like it's only copying the changes to the original filesystem since the
last rsync.  I'm not 100% sure that this is what's happening though.
The original filesystem has directories starting with a* to z*, but the
resultant backup filesystem only has directories from m* to z*, so maybe
it is deleting directories from earlier in the filesystem...I'm not
sure.  I have tried changing options in my backup script to get
different behavior, but roughly the same thing happens every night when
I run the script.  How would you recommend running the rsync copy?

Also, after upgrading to 2.6.8, I tried copying the whole filesystem at
once with 'cd /backup/directory; rsync -a --delete-during
/production/filesystem/ .', but this gave me nothing in my backup
filesystem.

Thanks,

Richard

-----Original Message-----
From: Wayne Davison [mailto:wayned@samba.org] 
Sent: Thursday, June 01, 2006 11:37 AM
To: Smemoe, Richard L.
Cc: rsync@lists.samba.org
Subject: Re: rsync --delete problem

On Thu, Jun 01, 2006 at 09:31:49AM -0500, Smemoe, Richard L.
wrote:
> When I added the -delete option, my backup started taking about two
> days to finish.  
You'd be better off using --del (--delete-during) than --delete, as
it doesn't require a separate scan through the destination hierarchy.
Also, older versions of rsync created an additional in-memory
representation of the entire destination file tree to do the delete,
so it is possible that such an older version caused rsync to start
swapping, which could easily slow it down by an order of magnitude.
So, make sure you're using a modern version (at least 2.6.4).

As for any incorrect behavior of --delete, you'll need to be more
explicit in what you think it is doing wrong.

..wayne..