I have downloaded and built rsync 2.6.0 on our HP ProLiant DL380-G3 servers (2 Hyperthread processors, 4 GB RAM, 840 GB Hard Disk, etc. etc.) running Red Hat 8 with the latest errata kernel (2.4.20-28smp) and updates from HP for RH8. rsync is run in daemon mode from xinetd. When doing transfers the machine requesting the transfer starts to run out of memory. After every rsync transfer there is a large amount of memory that is not freed up. The amount seems to be related to the number of files being transfered, not the number of bytes transfered. It is interesting to note that the transfer is started from the receiving machine, and the memory leak is on the receiving side. The sending side does not experience the leak. I have read other lists where there were memory leak problems with older versions that were related to the files lists and tokens, and I am wondering if that has crept back into the 2.6.0 code. Or, (perhaps more likely =8^) there is something wrong with my configuration or there were some configure options I did not set. Any help would be appreciated. Thanks J. Kelly Garrett, CISSP Security Engineer Level (3) Communications 180 Peachtree Street, Suite 300 Atlanta, GA 30303 (404) 526-4518
On Tue, Jan 20, 2004 at 02:45:47PM -0700, Garrett, Kelly wrote:> After every rsync transfer there is a large amount of memory that is > not freed up.Your report sounds like you're not talking about process size, but a free-memory report from something like "top". If so, Linux uses unused memory as disk cache, so the more disk I/O that happens, the less free memory you'll see on your system. This isn't a bad thing, though, as this disk-cache memory will get used for process memory as needed. Also keep in mind that once a process terminates, there's no way it can continue to hold memory (unless there's a bug in the kernel). If you meant something else, please explain what you're measuring. In my tests rsync's memory size stays steady throughout the transfer (once the file list has been built). Shared memory between the forked processes on the receiving side does slowly become unshared, but that happened in prior rsync versions too (and we've got an internal change in CVS that should make this better for the next release). ..wayne..
Thanks Wayne, This just may explain the behavior. I noticed that when I deleted the files that were transferred the memory was freed back up. Thanks for your help, I really appreciate it - I am not a Linux kernel guru so some of the low-level subtleties of the OS are lost on me =8^). I am starting to think there may be a kernel bug (or, maybe more likely, a misconfiguration), though. If it is the disk cache that is eating up this memory it does not seem to be freeing up for other processes when needed. I will eventually get a kernel panic on the machine when RAM and swap have been totally consumed during the rsync transfer. If this is the case, I may be out of luck, as Red Hat has retired RH8 so there will be no bug fixes on it - and I can't get 800GB+ of RAM into the machine to keep it running past the transfers. Rebooting a busy production machine every hour or so (before the panic can happen) is just not practical (geesh, sounds like I am running Windows). The application we are running will not run with the newer versions of RH yet, so I may have to go to a Sun solution. Cheers Kelly -----Original Message----- From: Wayne Davison [mailto:wayned@samba.org] Sent: Tuesday, January 20, 2004 9:09 PM To: Garrett, Kelly Cc: rsync@lists.samba.org Subject: Re: rsync 2.6.0 - suspected memory leak bug On Tue, Jan 20, 2004 at 02:45:47PM -0700, Garrett, Kelly wrote:> After every rsync transfer there is a large amount of memory that is > not freed up.Your report sounds like you're not talking about process size, but a free-memory report from something like "top". If so, Linux uses unused memory as disk cache, so the more disk I/O that happens, the less free memory you'll see on your system. This isn't a bad thing, though, as this disk-cache memory will get used for process memory as needed. Also keep in mind that once a process terminates, there's no way it can continue to hold memory (unless there's a bug in the kernel). If you meant something else, please explain what you're measuring. In my tests rsync's memory size stays steady throughout the transfer (once the file list has been built). Shared memory between the forked processes on the receiving side does slowly become unshared, but that happened in prior rsync versions too (and we've got an internal change in CVS that should make this better for the next release). ..wayne..
> On Wed, Jan 21, 2004 at 03:35:37PM +0000, Kelly Garrett wrote:>>> >>> Does anyone know how to build a version of the kernel that either does no disk >>> cacheing (we have very fast RAID processors and SCSI disks on the machine) or >>> limit the amount of cache that the system will allocate for disk? Kelly, we have a similar setup here (RH8 with latest bigmem kernel on a machine with 4GB ram and 1.4TB fs) and see a similar behaviour. Howevever, long ago I have accepted this as a "feature" of the linux kernel and I've yet to find that this causes any performance issues. It's true that if your file access is truly random caching the filesystem in RAM doesn't help, but I can't imagine that this is a significant performance hit. The system simply releases the cache as needed so that even if at any time you're seeing 100% memory usage, when a new process needs memory the RAM cache will give way. I'm sure that there are ways to override this behaviour (see for instance the linux kernel hacking howto for hints) but I doubt that this is worth the effort unless you need to squeeze every last bit of performance out of your box. And if you do, I would suggest looking at installing a 2.6.x kernel instead. -- Alberto P.S. I found that in our case one thing that actually helped quite a bit with overall performance was tuning some kernel parameters for the particular raid controller we have (3ware IDE raid). I mention this because if you start going down the performance tuning path there are a number of things that you should look at. ************************************************************************** Alberto Accomazzi, NASA Astrophysics Data System http://ads.harvard.edu Harvard-Smithsonian Center for Astrophysics http://cfa-www.harvard.edu 60 Garden St, MS 31, Cambridge, MA 02138, USA aaccomazzi@cfa.harvard.edu **************************************************************************