search for: uordblk

...sentially the patch provided by Martin Nowack in the bug report but I have added a test case and an ifdef macro around the modified code in Process::GetMallocUsage() The issue seems to be that the ptmalloc2 allocator used in glibc (in my case 2.19) does not include mmap()'ed memory in mallinfo.uordblks and so mallinfo.hblkhd needs to be added to the value that Process::GetMallocUsage() will return. Another easy way to see this issue (other than running the unit test) is to build and run the attached "malloc-test.c" program on a system that uses glibc. You will see that after malloc...

...1:03:11 rsync[8049] (sender) heap statistics: arena: 147992 (bytes from sbrk) ordblks: 3 (chunks not in use) smblks: 0 hblks: 0 (chunks from mmap) hblkhd: 0 (bytes from mmap) usmblks: 0 fsmblks: 0 uordblks: 42648 (bytes used) fordblks: 105344 (bytes free) keepcost: 11168 (bytes in releasable chunk) Number of files: 1 Number of files transferred: 1 Total file size: 8388608512 bytes Total transferred file size: 8388608512 bytes Literal data: 8388608512 bytes Matched data: 0...

...ena: 204800 (bytes from sbrk) ordblks: 7 (chunks not in use) smblks: 2 hblks: 2 (chunks from mmap) hblkhd: 401408 (bytes from mmap) allmem: 606208 (bytes from sbrk + mmap) usmblks: 0 fsmblks: 192 uordblks: 58736 (bytes used) fordblks: 146064 (bytes free) keepcost: 133904 (bytes in releasable chunk) generate_files finished rsync[32680] (generator) heap statistics: arena: 204800 (bytes from sbrk) ordblks: 10 (chunks not in use) smblks:...

...ena: 135168 (bytes from sbrk) ordblks: 2 (chunks not in use) smblks: 2 hblks: 2 (chunks from mmap) hblkhd: 401408 (bytes from mmap) allmem: 536576 (bytes from sbrk + mmap) usmblks: 0 fsmblks: 192 uordblks: 57200 (bytes used) fordblks: 77968 (bytes free) keepcost: 64288 (bytes in releasable chunk) rsync[2430] (generator) heap statistics: arena: 135168 (bytes from sbrk) ordblks: 2 (chunks not in use) smblks: 4 hblks:...

...sync[6790] (receiver) heap statistics: arena: 118120 (bytes from sbrk) ordblks: 5 (chunks not in use) smblks: 2 hblks: 0 (chunks from mmap) hblkhd: 0 (bytes from mmap) usmblks: 0 fsmblks: 80 uordblks: 101704 (bytes used) fordblks: 16416 (bytes free) keepcost: 3352 (bytes in releasable chunk) Number of files: 2 Number of files transferred: 1 Total file size: 6883446 bytes Total transferred file size: 6883446 bytes Literal data: 18546 bytes Matched data: 6864900 by...

...800 >> (bytes from sbrk) ordblks: 7 (chunks not in use) >> smblks: 2 hblks: 2 (chunks from mmap) >> hblkhd: 401408 (bytes from mmap) allmem: 606208 >> (bytes from sbrk + mmap) usmblks: 0 fsmblks: >> 192 uordblks: 58736 (bytes used) fordblks: 146064 >> (bytes free) keepcost: 133904 (bytes in releasable chunk) >> generate_files finished >> >> rsync[32680] (generator) heap statistics: arena: 204800 >> (bytes from sbrk) ordblks: 10 (chunks...

Thanks for the reply. On 28 April 2014 20:01, Reid Kleckner <rnk at google.com> wrote: > The problem with tcmalloc seems like a real problem. I can't think of any > good workarounds. My best worst idea is to try to figure out if malloc is > coming from libc with dlsym and dlopen, and then use that to decide whether > we add these two numbers together. I'm not sure we

Hello, I've had some problems using rsync to transfer directories with more than 3 million files. Here's the error message from rsync: <snip> ERROR: out of memory in receive_file_entry rsync error: error allocating core memory buffers (code 22) at util.c(116) rsync: connection unexpectedly closed (63453387 bytes read so far) rsync error: error in rsync protocol data stream (code

...denied rsync[17002] (sender) heap statistics: arena: 1659224 (bytes from sbrk) ordblks: 51 (chunks not in use) smblks: 6 hblks: 1 (chunks from mmap) hblkhd: 266240 (bytes from mmap) usmblks: 0 fsmblks: 232 uordblks: 1509880 (bytes used) fordblks: 149344 (bytes free) keepcost: 132928 (bytes in releasable chunk) The script is being run from the root account. My understanding is that running as root will eliminate the permission errors, but apparently that's not so.

...om sbrk) rsync[5964] (sender) heap statistics: arena: 53880 (bytes from sbrk) ordblks: 3 (chunks not in use) smblks: 0 hblks: 0 (chunks from mmap) hblkhd: 0 (bytes from mmap) usmblks: 0 fsmblks: 0 uordblks: 42424 (bytes used) fordblks: 11456 (bytes free) keepcost: 4152 (bytes in releasable chunk) Number of files: 44 Number of files transferred: 0 Total file size: 20229310 bytes Total transferred file size: 0 bytes Literal data: 0 bytes Matched data: 0 bytes File list si...

...0:52:32 rsync[8871] (sender) heap statistics: arena: 70552 (bytes from sbrk) ordblks: 3 (chunks not in use) smblks: 0 hblks: 1 (chunks from mmap) hblkhd: 266240 (bytes from mmap) usmblks: 0 fsmblks: 0 uordblks: 58120 (bytes used) fordblks: 12432 (bytes free) keepcost: 8112 (bytes in releasable chunk) Number of files: 1 Number of files transferred: 1 Total file size: 616431616 bytes Total transferred file size: 616431616 bytes Literal data: 85710848 bytes Matched data: 53072...

...denied rsync[17002] (sender) heap statistics: arena: 1659224 (bytes from sbrk) ordblks: 51 (chunks not in use) smblks: 6 hblks: 1 (chunks from mmap) hblkhd: 266240 (bytes from mmap) usmblks: 0 fsmblks: 232 uordblks: 1509880 (bytes used) fordblks: 149344 (bytes free) keepcost: 132928 (bytes in releasable chunk) The script is being run from the root account. My understanding is that running as root will eliminate the permission errors, but apparently that's not so.

Im running into an issue with exclude options not working correctly. Here are the details, if anybody can see what Im doing incorrectly, please let me know.. Thanks rsync version 2.5.7 protocol version 26 Command: "/usr/bin/rsync""--archive""--compress""--delete""--group""--perms""--st

...todate rsync[2509] (receiver) heap statistics: arena: 135168 (bytes from sbrk) ordblks: 6 (chunks not in use) smblks: 2 hblks: 0 (chunks from mmap) hblkhd: 0 (bytes from mmap) usmblks: 0 fsmblks: 40 uordblks: 6888 (bytes used) fordblks: 128280 (bytes free) keepcost: 90576 (bytes in releasable chunk) Number of files: 14 Number of files transferred: 0 Total file size: 8819 bytes Total transferred file size: 0 bytes Literal data: 0 bytes Matched data: 0 bytes File list size:...

...ena: 270336 (bytes from sbrk) ordblks: 1 (chunks not in use) smblks: 4 hblks: 2 (chunks from mmap) hblkhd: 532480 (bytes from mmap) allmem: 802816 (bytes from sbrk + mmap) usmblks: 0 fsmblks: 208 uordblks: 211632 (bytes used) fordblks: 58704 (bytes free) keepcost: 58496 (bytes in releasable chunk) deleting in . Number of files: 3 Number of files transferred: 2 Total file size: 25 bytes Total transferred file size: 25 bytes Literal data: 25 bytes Matched data: 0 bytes Fil...

...======= rsync[23169] (sender) heap statistics: arena: 17339064 (bytes from sbrk) ordblks: 509 (chunks not in use) smblks: 2 hblks: 2 (chunks from mmap) hblkhd: 1294336 (bytes from mmap) usmblks: 0 fsmblks: 80 uordblks: 17180952 (bytes used) fordblks: 158112 (bytes free) keepcost: 132840 (bytes in releasable chunk) Number of files: 161530 Number of files transferred: 327 Total file size: 97936829135 bytes Total transferred file size: 945709165 bytes Literal data: 741315984 bytes Matched d...

...0 (chunks from mmap) 2008/12/05 13:59:27 [940] hblkhd: 327680 (bytes from mmap) 2008/12/05 13:59:27 [940] allmem: 655360 (bytes from sbrk + mmap) 2008/12/05 13:59:27 [940] usmblks: 2686976 2008/12/05 13:59:27 [940] fsmblks: 0 2008/12/05 13:59:27 [940] uordblks: 479936 (bytes used) 2008/12/05 13:59:27 [940] fordblks: 175424 (bytes free) 2008/12/05 13:59:27 [940] keepcost: 44072 (bytes in releasable chunk) 2008/12/05 13:59:27 [940] Number of files: 1 2008/12/05 13:59:27 [940] Number of files transferred: 1 2008/12/05 13:59:27 [94...

Here is the structure of my servers : Source : /home/test1/link_dir sub_dir1 file1 file2 ... sub_dir2 ... Destination: /home/test2/real_dir sub_dir1 file1 file2 ... subdir2

...denied rsync[19081] (sender) heap statistics: arena: 348792 (bytes from sbrk) ordblks: 3 (chunks not in use) smblks: 0 hblks: 1 (chunks from mmap) hblkhd: 266240 (bytes from mmap) usmblks: 0 fsmblks: 0 uordblks: 341344 (bytes used) fordblks: 7448 (bytes free) keepcost: 3352 (bytes in releasable chunk) Number of files: 22 Number of files transferred: 22 Total file size: 2483555627 bytes Total transferred file size: 2483555627 bytes Literal data: 814454539 bytes Matched data:...

When I use the preallocate patch and create a 77GB file using the function I get a CPU spike on the server-side. The spike lasts about 20 minutes and uses about 20%-25% of the cpu associated with the rsync instance creating the file. The spike is directly linked to the time it takes to create the file. I compiled rsync using cygwin CVS. I initially suspected the implementation of