Hello,
I did some searching through the rsync archives at mail-archive.com
and didn't find anything less than 2 years old. I am hoping there have
been some developments since then, or that I can look towards debian for
clues to this problem.
I am trying to rsync some pretty large directories. These
directories have about 100-200 users in them. Sometimes we get this on a
single users home directory. Normally this works pretty well, but on
occasion we get repeatable errors:
# rsync -av --numeric-ids
...snip...
ERROR: out of memory in make_file [sender]
rsync: writefd_unbuffered failed to write 4 bytes [sender]: Broken pipe (32)
rsync: connection unexpectedly closed (1960976 bytes received so far)
[sender]
rsync error: error allocating core memory buffers (code 22) at io.c(548)
[sender=3.0.0cvs]
We're running this on a system which is in live use a file server. It is
copying from an NFS mount to an AOE block device. The current output is
it iterating through a Maildir. There is plenty of free memory available
on the server as far as I can tell:
bottom:~# free
total used free shared buffers cached
Mem: 3090232 3068536 21696 0 164 2289396
-/+ buffers/cache: 778976 2311256
Swap: 0 0 0
No limits:
bottom:~# ulimit
unlimited
Linux bottom 2.6.17.14-c1 #1 SMP Mon Nov 6 12:35:31 EST 2006 x86_64
GNU/Linux
It is Debian testing/unstable. We can reproduce this problem on the
packaged debian version (2.6.7) and the current CVS checkout
(2007-04-02, around 11am Pacific time.) There is another current rsync
process running on the server using 171m VIRT, 164m RES, and 728 SHR. It
is the sender and receiver.
I was wondering if anyone could shed some light on this? Is it still an
issue of "break it up into smaller hunks?" Any insight is appreciated.
I
can provide any debugging information you need on the system, tuning,
kernel build information, etc, that you may need.
Thank you in advance,
Kelly
Carson Gaspar wrote:> Kelly Kane wrote: > >> bottom:~# free >> total used free shared buffers cached >> Mem: 3090232 3068536 21696 0 164 >> 2289396 >> -/+ buffers/cache: 778976 2311256 >> Swap: 0 0 0 > > Is that output while rsync is running? rsync can eat a _lot_ of RAM, > although newer versions use less. > > That's mostly useless. Try ulimit -a. >I have done further testing on the machine, and yes, the `free` output reported above is pretty consistent regardless of when rsync crashes. It is difficult to get an exact memory report just before it crashes, sometimes it takes 30+ minutes to die. Do you have any suggestions for snagging that debugging output? # ssh bottom ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited max nice (-e) 0 file size (blocks, -f) unlimited pending signals (-i) unlimited max locked memory (kbytes, -l) unlimited max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) unlimited max rt priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) unlimited virtual memory (kbytes, -v) unlimited file locks (-x) unlimited Cheers, Kelly