rsync - Jan 2005 - BIG delete list makes for BIGGER RAM requirements

So I'm doing daily backups with rsync, and weekly, I run it with
--delete after archiving the whole thing (this way I don't lose any
deleted files). All week long this runs fine, but when I add --delete,
rsync runs for a few hours then aborts because the box runs out of
memory.

Jan 30 06:31:09 backup kernel: Out of Memory: Killed process 3490
(rsync).

Source box:
RH8+Custom Patches
Rsync v2.5.7
Linux v2.4.27
RAM: 2GB

Destination box (backup, from above message):
Fedora Core 1
Rsync v2.6.3
Linux 2.6.8
RAM: 512MB
Swap: 256MB
XFS on LVM2 on Software RAID5

Cmdline:

rsync --delete --force --safe-links -a --exclude=log --exclude=cricket
--exclude=backup --exclude=matt --exclude=ftp --exclude=*_tmp*
--exclude=mysql --exclude=lost+found --exclude=db/tempfiles/3hours
--exclude=db/tempfiles/1day --exclude=httpd/backhand* --exclude=jim
--exclude=clint --exclude=*.file --exclude=*tempfile*
--exclude=db/files -e ssh --whole-file
remotebox:/home/ /back/daily/remotebox/home

The box has nothing using up a ton of RAM, so there is always at least
500MB of virtual memory available. I've never been able to capture
exactly how much RAM rsync was using when this happens, but thats
probably about it.

During the week, it transfers about 85,000-100,000 new small files each
run. For --delete, I'm guessing it needs to delete maybe 20 times that
many files sometimes.

Is there a calculation I can do to find out how much RAM I'll need to do
this delete? The files are spread out pretty sporadically, so it would
be a real pain to try and do it in segments. Am I better off just
torching the whole tree on the backup server and retransferring all
60GB? Would it help to upgrade the sending machine to v2.6.3?

Thanks.
-- 
Clint Byrum <cbyrum@spamaps.org>

On Mon, 31 Jan 2005, Clint Byrum <cbyrum@spamaps.org>
wrote:
>
> So I'm doing daily backups with rsync, and weekly, I run it with
> --delete after archiving the whole thing (this way I don't lose any
> deleted files). All week long this runs fine, but when I add --delete,
> rsync runs for a few hours then aborts because the box runs out of
> memory.
When a --delete option is given, the receiver not only has in memory
the list of files from the sender, but it also builds its own list of
local files so it can determine what to delete.  This means that the
memory requirements are roughly doubled when you do deletions.

However - good news!

Wayne has recently implemented a --delete-during option that deletes
files in directories as they are processed.  This means there's no more
monolithic filelist for the local files on the target.

Wayne - it appears that the monolithic target file list is no longer
built for --delete or --delete-after as well.  Is that true?  If so,
that should be mentioned in the NEWS, as it will be an improvement
over the current procedure.

Clint - try the current version in CVS (or use a nightly snapshot).
It might solve your out-of-memory problem.  Use with --dry-run, first,
to see how things go.
-- 
        John Van Essen  Univ of MN Alumnus  <vanes002@umn.edu>

On Mon, Jan 31, 2005 at 08:35:44PM -0600, John Van Essen
wrote:
> Wayne - it appears that the monolithic target file list is no longer
> built for --delete or --delete-after as well.  Is that true?
No, at least not yet.  Both the before and after file-deletion options
use their old algorithms.  However, they could be optimized to be less
memory hungry in the future.

..wayne..