rsync - Mar 2006 - AIX 5.1 rsync large file

Hello,

I have inherited a setup where there are 2 AIX 5.1 systems in 2 separate
sites.  There are large database files that are backed from each site to
the other via rsync.  Currently, it is using rsync version 2.5.4.  It does
it via ssh with the options -avz.  This has all been merrily plodding along
for some time.  There is one file that is over 45 GB, and it started having
trouble with that file.  I don't know what the size of the file was before
it started having trouble or if they made a bunch of changes in this .dbf
fille before it started having trouble.  There are other large files that
are smaller than this (i.e., 30 GB, etc.).   The error message was:
ERROR: out of memory in build_hash_table
rsync error: error allocating core memory buffers (code 22) at util.c(115)
rsync: connection unexpectedly closed (585911608 bytes read so far)
rsync error: error in rsync protocol data stream (code 12) at io.c(165)
rsync: connection unexpectedly closed (1224 bytes read so far)
rsync error: error in rsync protocol data stream (code 12) at io.c(165)
Rsyncget completed file transfer at Sat Mar 11 02:49:52 EST 2006

We tried pushing and pulling both, and it gets that error both ways.  I
tried downloading an already compiled version of rsync 2.6.4 (and a version
I compiled as well that includes support for AIX large files), and using
this on both ends, it seems to hang, and the hidden .file.xyz that it
creates stalls at size 2097152.  ulimit -a shows:
time(seconds)        unlimited
file(blocks)         unlimited
data(kbytes)         131072
stack(kbytes)        32768
memory(kbytes)       32768
coredump(blocks)     2097151
nofiles(descriptors) 2000

Any help and suggestions would be appreciated, I'm at my wit's end with
this stuff, since I've never used it previous to this.

Thanks a lot!

Jackie Andrason
****************IMPORTANT--PLEASE READ*******************
This electronic message, including its attachments, is COMPANY CONFIDENTIAL
and may contain PROPRIETARY or LEGALLY PRIVILEGED information.  If you are 
not the intended recipient, you are hereby notified that any use, disclosure,
copying, or distribution of this message or any of the information included
in it is unauthorized and strictly prohibited.  If you have received this
message in error, please immediately notify the sender by reply e-mail and
permanently delete this message and its attachments, along with any copies
thereof. If this electronic message contains a zipped attachment and you do
not have a decompression tool, you may download unZIP (free of cost) from:
http://www.mk-net-work.com/us/uz/unzip.htm. Alternatively, you may request
that the attachment be resent in an uncompressed format.        Thank you. 
************************************************************************

Let's try.


Jackie.E.Andrason@Quintiles.com wrote:
>There is one file that is over 45 GB, and it started having
>trouble with that file.
>
>  I don't know what the size of the file was before
>it started having trouble
>
>We tried pushing and pulling both, and it gets that error both ways. 
>
Rsync has two, unrelated, definitions for connection endpoints. The side
initiating the connection is called the "client", while the other side
is called the "server". Then again, the side that has the original
files
to be synched is called the "sender", while the other side is called
the
"receiver".

Switching from pushing to pulling switches the client and server sides,
but does nothing to change the sender and receiver tags. The code that
freaks out on you is in the sender, and so the switch has no effect.
>The error message was:
>ERROR: out of memory in build_hash_table
>
>ulimit -a shows:
>memory(kbytes)       32768
>  
>
So you are telling rsync to synchronize a 45GB file using no more than
32MB of memory? Try setting it much higher, if "unlimited" is not an
option. Under "bash" and Linux, this is done with "ulimit -v
unlimited".
I'm not sure that the corresponding AIX command will be the same.
>Any help and suggestions would be appreciated, I'm at my wit's end
with
>this stuff, since I've never used it previous to this.
>  
>
Yes.

rsync 2.6.7 has a change that is meant to reduce the memory consumption
of the hash table during transfer of large files. While on the subject,
the "patches" directory has a patch that is meant to speed things up
when using files larger than 2.5GB. Switching to 2.6.7 is, therefor,
adviseable (though it may well be that 32MB of memory will still not be
enough).

While on the matter, any feedback regarding comparing 2.6.7 with and
without the "dynamic_hash" patch would be greatly appreciated. I'm
trying to push Wayne into putting it into the actual rsync code. A
benchmark saying "it took so and so time with vanilla 2.6.7 and so and
so with 2.6.7 + dynamic_hash" would be much appreciated.

          Shachar