rsync - Dec 2004 - very big rsync only worked partially what are size limitations?

Hi, I am very grateful for rsync!!!

Two days ago i started a backup of a 
630 GB directory on a 700GB raid which has 13,945 subdirectories, to another 
server. It copied over 525GB worth of data, and I see that there are 12,627 
directories on the destination. The command I used was 
rsync -a -e ssh /big/dir/   192.168.1.2:/big/dir.  

I thought maybe it ran out of memory or something - yes there are thousands of 
files in the subdirectories - ie some of the directories may have as many as 
as a couple of hundred or so files in sub sub directories.

I then tried again and started the rsync script on the directory again, and it 
ended without copying over any more stuff, as far as I can see  - size wise 
and number of subdirectories.  :(.

Then I just wanted to get it all over with: so I  just wrote a perl script to 
rsync the remainder of the directories one at a time, and that is running now 
and I will see what happens.

So my question, what are the size limitations on this sort of stupid original 
way of doing it?    My machine has 1GB of RAM and I have 2GB of swap space. 
Of course I am also running Postgresql too which is a bit of a memory hog.

Thanks! and I am very happy to have met rsync!
Thank you all!

On Sat, Dec 18, 2004 at 09:50:53PM -0500, mlaks@verizon.net
wrote:
> I then tried again and started the rsync script on the directory
> again, and it ended without copying over any more stuff, as far as I
> can see  - size wise and number of subdirectories.  :(.
Did it output an error? Or end normally? If it did not output an error,
it believes that the files you asked it to transfer are up-to-date.
> So my question, what are the size limitations on this sort of stupid
> original way of doing it?
The maximum file count is mainly limited by memory.  Figure that it
probably takes 100 bytes or so per file/directory in the transfer list.
If you're not using a modern rsync (2.6.3 is the latest) this memory
requirement will be much higher, particularly on the receiving system:
the receiving side forks off an extra rsync process, which should share
a lot of the same memory as the parent process (if your fork uses copy-
on-write memory-sharing).  Depending on your OS and the age of your
rsync (modern ones do a better job of keeping the memory shared between
the processes), the receiving side's memory use could bloat to be double
what the sending side needs.  Your best bet is to just run the command
and look at memory use, keeping an eye on the total proces size and (on
the receiving side) how much memory is shared.

..wayne..