Hello, I need to create a local backup every night of a filesystem that has about 3.5 million files. I've been using rsync and everything has been fine until I started to use the -delete option to delete files on the backup that don't exist in the production data. When I added the -delete option, my backup started taking about two days to finish. So I changed my backup script to rsync individual top-level directories instead of the whole filesystem at once. This finished must faster, except now rsync seems to be only copying files that have changed on the production data filesystem, and deleting files on my backup that haven't changed. Here are the three different scripts that I've tried and they all do the same thing: 1: cd /backup/directory rsync -au --delete /path/to/files/a* . rsync -au --delete /path/to/files/b* . rsync -au --delete /path/to/files/c* . ... 2: cd /backup/directory rsync -au --delete --include /path/to/files/a* --exclude /path/to/files/ /path/to/files/a* . rsync -au --delete --include /path/to/files/b* --exclude /path/to/files/ /path/to/files/b* . rsync -au --delete --include /path/to/files/c* --exclude /path/to/files/ /path/to/files/c* . ... 3: cd /backup/directory for file in /path/to/files/*; do rsync -au --delete $file .; done Does anyone know what I'm doing wrong here? Thanks, Richard -------------- next part -------------- HTML attachment scrubbed and removed
On Thu, Jun 01, 2006 at 09:31:49AM -0500, Smemoe, Richard L. wrote:> When I added the -delete option, my backup started taking about two > days to finish.You'd be better off using --del (--delete-during) than --delete, as it doesn't require a separate scan through the destination hierarchy. Also, older versions of rsync created an additional in-memory representation of the entire destination file tree to do the delete, so it is possible that such an older version caused rsync to start swapping, which could easily slow it down by an order of magnitude. So, make sure you're using a modern version (at least 2.6.4). As for any incorrect behavior of --delete, you'll need to be more explicit in what you think it is doing wrong. ..wayne..
Wayne, Thanks for your reply. I upgraded to 2.6.8 and switched to --delete-during. But I still have the same problem as I did with --delete. I'll try to explain the behavior I'm seeing a little better. The directory I'm trying to rsync from has ~3.5 million files with the upper level directory having about 2000 directories. I'm running 'cd /backup/directory', then 'rsync -a --delete-during /backup/directory/directorytocopy .' on every directory in this top level directory. If I leave off the --delete-during (or --delete), everything works fine (except files removed from the original filesystem aren't deleted), but if I use the --delete-during option, I only end up with a small fraction of the original filesystem in my backup. It looks like it's only copying the changes to the original filesystem since the last rsync. I'm not 100% sure that this is what's happening though. The original filesystem has directories starting with a* to z*, but the resultant backup filesystem only has directories from m* to z*, so maybe it is deleting directories from earlier in the filesystem...I'm not sure. I have tried changing options in my backup script to get different behavior, but roughly the same thing happens every night when I run the script. How would you recommend running the rsync copy? Also, after upgrading to 2.6.8, I tried copying the whole filesystem at once with 'cd /backup/directory; rsync -a --delete-during /production/filesystem/ .', but this gave me nothing in my backup filesystem. Thanks, Richard -----Original Message----- From: Wayne Davison [mailto:wayned@samba.org] Sent: Thursday, June 01, 2006 11:37 AM To: Smemoe, Richard L. Cc: rsync@lists.samba.org Subject: Re: rsync --delete problem On Thu, Jun 01, 2006 at 09:31:49AM -0500, Smemoe, Richard L. wrote:> When I added the -delete option, my backup started taking about two > days to finish.You'd be better off using --del (--delete-during) than --delete, as it doesn't require a separate scan through the destination hierarchy. Also, older versions of rsync created an additional in-memory representation of the entire destination file tree to do the delete, so it is possible that such an older version caused rsync to start swapping, which could easily slow it down by an order of magnitude. So, make sure you're using a modern version (at least 2.6.4). As for any incorrect behavior of --delete, you'll need to be more explicit in what you think it is doing wrong. ..wayne..
Reasonably Related Threads
- [Bug 2583] New: rsync fails with "authentication failure"
- DO NOT REPLY [Bug 3825] New: rsync won't delete directory with excluded files
- How to know the remote version of rsync daemon?
- Request to include delete-dir-and-suffix patch in rsync-2.6.4
- DO NOT REPLY [Bug 3415] New: -R and --delete cause --delete to be ignored