I recently installed rsync 2.5.5 on both my rsync server and client being used. I installed the latest version because I was having problems with rsync stalling with version 2.4.6 (I read that 2.5.5 was supposed to clear this up or at least give more appropriate errors). I am still having problems with rsync stalling even after upgrading to 2.5.5. It only stalls in the "/home" tree and there are approximately 385,000 (38.5 MB of memory give or take if the 100 bytes/file still pertains) files in that tree. It doesn't always stall on the same file, but that probably doesn't really matter since this seems like it may be a kernel problem or some other system specific problem. I am curious as to whether or not anyone has seen this problem or knows of a fix for it or if it's something I'm doing wrong. I would be grateful for any information. I'm receiving a couple of error messages (both refer to the same problem as far as I know): ERROR: out of memory in build_hash_table rsync error: error allocating core memory buffers (code 22) at util.c(232) The following are the commands being used from a bash script (not the whole script of course): rsync -zaHlv --progress -e /usr/bin/ssh --exclude log --exclude exim /var/* Server_IP:/home/host/var 2>>$error_log rsync -zaHlv --progress -e /usr/bin/ssh /etc/* Server_IP:/home/host/etc 2>>$error_log rsync -zaHlv --progress -e /usr/bin/ssh /usr/local/apache/conf/* Server_IP:/home/host/usr/local/apache/conf 2>>$error_log rsync -zaHlv --progress -e /usr/bin/ssh --exclude backup --exclude backups --exclude usr /home/* Server_IP:/home/host/home The following is some information about the machines: server running rsync --daemon: Linux RH: Kernel 2.4.9-31 512 MB RAM rsync 2.5.5 OpenSSH_3.1p1, SSH protocols 1.5/2.0 rsync client Linux RH: 2.4.9-31enterprise 2 GB RAM rsync 2.5.5 OpenSSH_3.1p1, SSH protocols 1.5/2.0 Thank you for any help and I would be happy to include any further information if needed.
Eric Echter [eric.echter@mchsi.com] writes:> I recently installed rsync 2.5.5 on both my rsync server and client being > used. I installed the latest version because I was having problems with > rsync stalling with version 2.4.6 (I read that 2.5.5 was supposed to clear > this up or at least give more appropriate errors). I am still having > problems with rsync stalling even after upgrading to 2.5.5. It onlystalls> in the "/home" tree and there are approximately 385,000 (38.5 MB of memory > give or take if the 100 bytes/file still pertains) files in that tree.The key growth factor for the file construction is going to be per-file information that's about 56 bytes (last I looked) per file. However, the kicker is that the file storage block gets resized exponentially as you have more files. So for 385,000 files it'll actually be a 512,000 file block of about 30MB. (So yeah, I suppose an ~50 byte file chunk in memory growing as a power of 2 might average out close to 100 bytes/file as an estimate :-))> ERROR: out of memory in build_hash_table > rsync error: error allocating core memory buffers (code 22) at util.c(232)Seems like that's just a real out of memory error. You'll only get that error if a malloc() call returned NULL. I presume there's still enough virtual memory available on the server at the point when this fails? Could you be running into a process limit on virtual memory? What's a "ulimit -a" show for a server process? I think under Linux the default settings are in /etc/security/limits.conf, maybe by default processes on the server are limited to 32MB of memory or something? -- David /-----------------------------------------------------------------------\ \ David Bolen \ E-mail: db3l@fitlinxx.com / | FitLinxx, Inc. \ Phone: (203) 708-5192 | / 860 Canal Street, Stamford, CT 06902 \ Fax: (203) 316-5150 \ \-----------------------------------------------------------------------/
Eric Echter [eric.echter@mchsi.com] writes:> I also checked the /etc/security/limits.conf file and everything in > the file is commented out. Are there default limits if there are no > actual settings in this file that may be causing problems? Your > assumption about the memory limit on processes sounds correct, but I > can't find any reasoning for this from the system settings. Thanks > a bunch for the response.I'm not that familiar with Linux defaults, but your ulimit -a should reflect what the process actually has and it certainly looks good. That assumes that you are running rsync on the server under root (either via the rsh/ssh path, or as a daemon). If you're running it as some other user, you should ensure you check the ulimit -a under a process running as that user. Perhaps /etc/profile makes adjustments? If not, you might watch virtual memory stats while running the failing operation (e.g., have a window using vmstat or top running on the server when you try the copy) to see if there's anything amiss looking at the overall server level - or if perhaps rsync is somehow burning up much more memory than we're estimating. Beyond that though, I suppose perhaps a linux-oriented group would offer further suggestions, under the assumption that something must be leading to the malloc() failing. -- David /-----------------------------------------------------------------------\ \ David Bolen \ E-mail: db3l@fitlinxx.com / | FitLinxx, Inc. \ Phone: (203) 708-5192 | / 860 Canal Street, Stamford, CT 06902 \ Fax: (203) 316-5150 \ \-----------------------------------------------------------------------/