rsync - Aug 2008 - memory usage in rsync 3.0.3 -- how much RAM should I have to transfer 13 million files?

Hi.  I am trying to recursively rsync a directory containing  13 million files.

Right now this is killing my server, in terms of memory usage.

I've upgraded from rsync 2.6.9 to 3.0.3 on both ends, but memory usage
is still too high.   I killed the rsync process when it reached 256 MB
in size.

I only have 1 GB of RAM in this server.

We've planned an outage to upgrade it to 3 GB, but will that be enough?

How much memory will rsync use?   I didn't specificy any of the
switches that disable incremental recursion.

Thanks,
Aleksey

Aleksey Tsalolikhin wrote:
> I've upgraded from rsync 2.6.9 to 3.0.3 on both ends, but memory usage
> is still too high.
Why should rsync 3's memory usage depend on the number of files? Does it 
keep files it already knows should not be transferred in memory?

If not, then maybe we should hold back rsync's very useful, very speed 
productive, read ahead of the file list. If we see that the "todo
list"
piles up, maybe we should hold of the continued scan until the back log 
gets smaller.

Yes, I know, it's the typical someone sitting on the fence, hardly ever 
doing anything useful for the project, and dispensing "invaluable" 
advice. Fact is, I need this. If Wayne doesn't do it, I will get around 
to it eventually. The problem is that the key word here is
"eventually".

Shachar

On Tue, Aug 12, 2008 at 11:46:11AM -0700, Aleksey Tsalolikhin
wrote:
> How much memory will rsync use?   I didn't specificy any of the
> switches that disable incremental recursion.
It depends on your options, and possibly on the maximum number of files
in a directory.  I've seen a recursive scan use about 20MB for a huge
set of files, which is not much more memory than a typical bash or zsh
process uses.

You should make extra sure that you're not disabling incremental
recursion by specifying at least one -v option and checking that rsync
says that it is sending/receiving an incremental file list at the start
(not the older "building file list" or "receiving file list"
messages).

If it is doing an incremental scan, it still processes the immediate
contents of every individual directory as a unit (and has at least one
directory of read-ahead in memory), so if you have really huge numbers
of files in your directories, that will increase the maximum memory
used.

Other options will also affect memory use, such as -H which makes rsync
search for hard-link matches over the whole of the hierarchy, and that
can really bloat things, particularly if a large percentage of the files
in the transfer have more than one link.  (Since you didn't cite your
options, I can't be more specific.)

..wayne..