Hi, folks. I've started getting these errors from rsync, and any help would be appreciated:>ERROR: out of memory in string_area_new buffer >rsync error: error allocating core memory buffers (code 22) at util.c(115) >ERROR: out of memory in string_area_new buffer >rsync error: error allocating core memory buffers (code 22) at util.c(115) >ERROR: out of memory in string_area_new buffer >rsync error: error allocating core memory buffers (code 22) at util.c(115)The file system transfer in question then fails. After Googling and looking at the man pages, I have surmised that this is likely a malloc/out of memory error. From the man page:> RERR_MALLOC 22 > Error allocating core memory buffersI'm using version ...>$ rsync --version >rsync version 2.5.6 protocol version 26 >Copyright (C) 1996-2002 by Andrew Tridgell and others ><http://rsync.samba.org/> >Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles, > no IPv6, 32-bit system inums, 64-bit internal inums > >rsync comes with ABSOLUTELY NO WARRANTY. This is free software, and you >are welcome to redistribute it under certain conditions. See the GNU >General Public Licence for details.HP-UX 11i is the OS flavor, and the file [sub]systems to be transferred have many files. I'll be testing further with more verbosity, but I know that the files to be transferred would be in the 300,000 count range. The actual quantity of data is fairly small (<20GB). I also see that ... "As a rule of thumb you should expect rsync to consume about 100 bytes per file in the file list." That would make some 30,000,000 bytes, or ~30MB, which is not much on our system. Could there be other issues? I also see that there is an HP-UX compile time option (-Ae from http://samba.org/cgi-bin/cvsweb/rsyncweb/faqbody.html?rev=1.4). I suspect that this applies to the HP ANSI compiler, not gcc as I am using, but I do not know if this is the cause. I see some memory issues in the TODO list with my distribution, but I'm not familiar with the fix(es). It perplexes me that the site above references 80MB transfers - over twice the size of my current issue ... oh wait, my test is back (with double-v) ... receiving file list ... expand file_list to 4000 bytes, did move expand file_list to 8000 bytes, did move expand file_list to 8000 bytes, did move expand file_list to 16000 bytes, did move expand file_list to 16000 bytes, did move expand file_list to 32000 bytes, did move expand file_list to 32000 bytes, did move expand file_list to 64000 bytes, did move expand file_list to 64000 bytes, did move expand file_list to 128000 bytes, did move expand file_list to 128000 bytes, did move expand file_list to 256000 bytes, did move expand file_list to 256000 bytes, did move expand file_list to 512000 bytes, did move expand file_list to 512000 bytes, did move expand file_list to 1024000 bytes, did move expand file_list to 1024000 bytes, did move expand file_list to 2048000 bytes, did move expand file_list to 2048000 bytes, did move expand file_list to 4096000 bytes, did move expand file_list to 4096000 bytes, did move expand file_list to 8192000 bytes, did move expand file_list to 8192000 bytes, did move expand file_list to 16384000 bytes, did move expand file_list to 16384000 bytes, did move expand file_list to 32768000 bytes, did move expand file_list to 32768000 bytes, did move ERROR: out of memory in receive_file_entry rsync error: error allocating core memory buffers (code 22) at util.c(115) This looks like one of those pesky sizeof-type issues ... Would a 64-bit compile help? Are there other configure/compile options which should help? Google and the list archives do not return anything on this issue specifically. Any direction would be appreciated, even where to dig or specific cvs information ... thanks in advance! A. Daniel King, System Analyst HP-UX, Linux, Solaris :wq
On Fri, Nov 07, 2003 at 11:33:07AM -0600, King, Daniel wrote:> Hi, folks. > > I've started getting these errors from rsync, and any help would be > appreciated: > > >ERROR: out of memory in string_area_new buffer > >rsync error: error allocating core memory buffers (code 22) at util.c(115) > >ERROR: out of memory in string_area_new buffer > >rsync error: error allocating core memory buffers (code 22) at util.c(115) > >ERROR: out of memory in string_area_new buffer > >rsync error: error allocating core memory buffers (code 22) at util.c(115) > > The file system transfer in question then fails. After Googling and looking > at the man pages, I have surmised that this is likely a malloc/out of memory > error. From the man page: > > > RERR_MALLOC 22 > > Error allocating core memory buffers > > I'm using version ... > > >$ rsync --version > >rsync version 2.5.6 protocol version 26 > >Copyright (C) 1996-2002 by Andrew Tridgell and others > ><http://rsync.samba.org/> > >Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles, > > no IPv6, 32-bit system inums, 64-bit internal inums > > > >rsync comes with ABSOLUTELY NO WARRANTY. This is free software, and you > >are welcome to redistribute it under certain conditions. See the GNU > >General Public Licence for details. > > HP-UX 11i is the OS flavor, and the file [sub]systems to be transferred have > many files. I'll be testing further with more verbosity, but I know that > the files to be transferred would be in the 300,000 count range. The actual > quantity of data is fairly small (<20GB). > > I also see that ... > > "As a rule of thumb you should expect rsync to consume about 100 bytes per > file in the file list."Thumbs come in all sizes. The file list itself consumes approximately 72 bytes (depending on word size and the size of some of the datatypes used in stat(2)) per file including directories. In addition there is the space for the names. Running "find . -print|wc" is a halfway decent way to estimate the space requirements. Line count * 72 + character count should give you an overestimate.> That would make some 30,000,000 bytes, or ~30MB, which is not much on our > system. Could there be other issues? I also see that there is an HP-UX > compile time option (-Ae from > http://samba.org/cgi-bin/cvsweb/rsyncweb/faqbody.html?rev=1.4). I suspect > that this applies to the HP ANSI compiler, not gcc as I am using, but I do > not know if this is the cause. > > I see some memory issues in the TODO list with my distribution, but I'm not > familiar with the fix(es). It perplexes me that the site above references > 80MB transfers - over twice the size of my current issue ... oh wait, my > test is back (with double-v) ... > > receiving file list ... > expand file_list to 4000 bytes, did move[snip]> expand file_list to 16384000 bytes, did move > expand file_list to 16384000 bytes, did move > expand file_list to 32768000 bytes, did move > expand file_list to 32768000 bytes, did move > ERROR: out of memory in receive_file_entry > rsync error: error allocating core memory buffers (code 22) at util.c(115)It uses realloc to expand the file_list. If "did move" appears that means that the realloc was not able to simply extend the list but had to allocate new memory to hold it and then copy from the old to the new locations. So when it died on you it was trying to allocate a new 64MB worth of memory in addition tot he 32MB it already had for the list (==96MB) plus whatever is needed for holding all the names and other data structures.> This looks like one of those pesky sizeof-type issues ... Would a 64-bit > compile help? Are there other configure/compile options which should help?I doubt it.> Google and the list archives do not return anything on this issue > specifically. > > Any direction would be appreciated, even where to dig or specific cvs > information ... thanks in advance!The only approach is to split the job up. Hardlinks permitting. -- ________________________________________________________________ J.W. Schultz Pegasystems Technologies email address: jw@pegasys.ws Remember Cernan and Schmitt
Regis_Pei@capgroup.com
2003-Nov-11 07:00 UTC
malloc errors? out of memory with many files on HP-UX
Have you checked the maxdsiz kernel parameter? The default is 0x4000000 which is only 64MBs. |---------+----------------------------------------------> | | "King, Daniel" | | | <Daniel.King@Fiserv.com> | | | Sent by: | | | rsync-bounces+rep=capgroup.com@list| | | s.samba.org | | | | | | | | | 11/07/2003 09:33 AM | | | | |---------+----------------------------------------------> >------------------------------------------------------------------------------------------------------------------------------| | | | To: rsync@lists.samba.org | | cc: (bcc: Regis Pei/CDS/CG/CAPITAL) | | Subject: malloc errors? out of memory with many files on HP-UX | >------------------------------------------------------------------------------------------------------------------------------| Hi, folks. I've started getting these errors from rsync, and any help would be appreciated:>ERROR: out of memory in string_area_new buffer >rsync error: error allocating core memory buffers (code 22) at util.c(115) >ERROR: out of memory in string_area_new buffer >rsync error: error allocating core memory buffers (code 22) at util.c(115) >ERROR: out of memory in string_area_new buffer >rsync error: error allocating core memory buffers (code 22) at util.c(115)The file system transfer in question then fails. After Googling and looking at the man pages, I have surmised that this is likely a malloc/out of memory error. From the man page:> RERR_MALLOC 22 > Error allocating core memory buffersI'm using version ...>$ rsync --version >rsync version 2.5.6 protocol version 26 >Copyright (C) 1996-2002 by Andrew Tridgell and others ><http://rsync.samba.org/> >Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles,> no IPv6, 32-bit system inums, 64-bit internal inums > >rsync comes with ABSOLUTELY NO WARRANTY. This is free software, and you >are welcome to redistribute it under certain conditions. See the GNU >General Public Licence for details.HP-UX 11i is the OS flavor, and the file [sub]systems to be transferred have many files. I'll be testing further with more verbosity, but I know that the files to be transferred would be in the 300,000 count range. The actual quantity of data is fairly small (<20GB). I also see that ... "As a rule of thumb you should expect rsync to consume about 100 bytes per file in the file list." That would make some 30,000,000 bytes, or ~30MB, which is not much on our system. Could there be other issues? I also see that there is an HP-UX compile time option (-Ae from http://samba.org/cgi-bin/cvsweb/rsyncweb/faqbody.html?rev=1.4). I suspect that this applies to the HP ANSI compiler, not gcc as I am using, but I do not know if this is the cause. I see some memory issues in the TODO list with my distribution, but I'm not familiar with the fix(es). It perplexes me that the site above references 80MB transfers - over twice the size of my current issue ... oh wait, my test is back (with double-v) ... receiving file list ... expand file_list to 4000 bytes, did move expand file_list to 8000 bytes, did move expand file_list to 8000 bytes, did move expand file_list to 16000 bytes, did move expand file_list to 16000 bytes, did move expand file_list to 32000 bytes, did move expand file_list to 32000 bytes, did move expand file_list to 64000 bytes, did move expand file_list to 64000 bytes, did move expand file_list to 128000 bytes, did move expand file_list to 128000 bytes, did move expand file_list to 256000 bytes, did move expand file_list to 256000 bytes, did move expand file_list to 512000 bytes, did move expand file_list to 512000 bytes, did move expand file_list to 1024000 bytes, did move expand file_list to 1024000 bytes, did move expand file_list to 2048000 bytes, did move expand file_list to 2048000 bytes, did move expand file_list to 4096000 bytes, did move expand file_list to 4096000 bytes, did move expand file_list to 8192000 bytes, did move expand file_list to 8192000 bytes, did move expand file_list to 16384000 bytes, did move expand file_list to 16384000 bytes, did move expand file_list to 32768000 bytes, did move expand file_list to 32768000 bytes, did move ERROR: out of memory in receive_file_entry rsync error: error allocating core memory buffers (code 22) at util.c(115) This looks like one of those pesky sizeof-type issues ... Would a 64-bit compile help? Are there other configure/compile options which should help? Google and the list archives do not return anything on this issue specifically. Any direction would be appreciated, even where to dig or specific cvs information ... thanks in advance! A. Daniel King, System Analyst HP-UX, Linux, Solaris :wq -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html