rsync - Oct 2019 - [Bug 12769] error allocating core memory buffers (code 22) depending on source file system

https://bugzilla.samba.org/show_bug.cgi?id=12769

            Bug ID: 12769
           Summary: error allocating core memory buffers (code 22)
                    depending on source file system
           Product: rsync
           Version: 3.1.0
          Hardware: All
                OS: Linux
            Status: NEW
          Severity: normal
          Priority: P5
         Component: core
          Assignee: wayned at samba.org
          Reporter: mss-it at fau.de
        QA Contact: rsync-qa at samba.org

We run an openSuSE Leap 42.2 and an Ubuntu 14.04.5 on two servers. Copying a
large number of files (in this case about 28 million) leads to different
results depending on the source file system. 
We copy with rsync -rlptgoDAxHnP --info=progress2 --delete
--link-dest=$LINK_DEST root@$SERVER:/$FOLDER /backup/rsynctest/ . Replacing
--delete by --delete-delay doesn't change the behaviour as expected. The
error
occurs with and without the option -n, in this case it is just for testing
reasons included.
In case the source is located on an Ext4 file system we run into the following
error message after about 26 million files copied:
ERROR: out of memory in hashtable_node [sender]
rsync error: error allocating core memory buffers (code 22) at util2.c(102)
[sender=3.1.0]
In case the source is located on an XFS file system the above command copies
all files without error.
Both of the file systems hold the same data as the one is the backup copy of
the other. The behaviour appears as well when we use rsync via an rsync server
and not via SSH and as well when we copy locally on one of the two machines.
And it appears regardless of the operating system (openSuSE 42.2 or Ubuntu
14.04.5). 
I did not try in the last time but replaying the backup from an Ext4 showed
this error at least one year ago as well. With the change to XFS on the source
file system the error suddenly disappeared.
As the error appears even if just doing a --dry-run it seems to be related to
the way rsync handles metadata. The data size seems to be unimportant.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=12769

--- Comment #1 from Roland Haberkorn <mss-it at fau.de> ---
If you want me to run further testings with other file systems I am totally
willing to produce fake data and run tests. I just haven't done yet because
of
my lack of knowledge about the underlying mechanisms and because I am not
totally sure whether this is a rsync's problem or a kernel issue.
To add two more things: We saw this issue also when having mounted the data
with NFSv3 or v4. The target file system does not matter, we've had this
issue
with btrfs, Ext4 and XFS.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=12769

--- Comment #2 from Roland Haberkorn <mss-it at fau.de> ---
I did some further investigation... 
First thing to add: The ext4 file systems are hard-linked differential rsync
backups of the real data on XFS.
I changed the testcase by deleting the --link-dest option.
When rsyncing from an XFS, the rsync process on the client uses about 3% RAM
(of total 8GB). When rsyncing from an ext4, it uses up to about 50% RAM.
This picture totally changes when I delete the option -H. In this case also
copying from an ext4 uses only less than 2% RAM. 
My guess would be that perhaps -H breaks the incremental recursion when copying
from an ext4.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=12769

--- Comment #3 from Roland Haberkorn <mss-it at fau.de> ---
Ok, I digged somewhat deeper. I've found a second difference between my two
sources. The one is the original data, the other one is a diffential rsync
backup with hard links.
I then built a testcase with about 50 million dummy files with something like
this:

#!/bin/bash
for i in {1..50}
do
mkdir $i
#cd $i
for j in {1..1000}
do
mkdir $i/$j
#cd $j
for k in {1..1000}
do
touch $i/$j/$k
done
done
done

Rsyncing this testfolder work fine from and to any of the tested file systems
(ext4 64Bit, XFS, btrfs). This is true for with and without the Option -H and
as long as in the source file system there is no hard linked copy of the source
folder. 
In the moment when there is at least one hard linked copy the option -H breaks
the run:

roland at msspc25:~$ stat /mnt/rsynctestsource/1/1/1
  File: /mnt/rsynctestsource/1/1/1
  Size: 0               Blocks: 0          IO Block: 4096   regular empty file
Device: 811h/2065d      Inode: 153991349   Links: 2
Access: (0644/-rw-r--r--)  Uid: ( 2001/  roland)   Gid: (  100/   users)
Access: 2017-05-09 10:54:10.341300841 +0200
Modify: 2017-05-08 09:33:51.535967423 +0200
Change: 2017-05-26 16:21:57.610628573 +0200
 Birth: -
roland at msspc25:~$ rsync -rlptgoDxAn --info=name,progress2  --delete
--link-dest=/mnt2/link3/ /mnt/rsynctestsource/ /mnt2/link1/.
              0 100%    0.00kB/s    0:00:00 (xfr#0, to-chk=0/49049050)   
roland at msspc25:~$ rsync -rlptgoDxHAn --info=name,progress2  --delete
--link-dest=/mnt2/link3/ /mnt/rsynctestsource/ /mnt2/link1/.
              0 100%    0.00kB/s    0:00:00 (xfr#0, ir-chk=1000/25191050)
ERROR: out of memory in hashtable_node [sender]
rsync error: error allocating core memory buffers (code 22) at util2.c(106)
[sender=3.1.2]

You can see, the first run without -H works, the last one with -H doesn't. 

So I would have to somewhat rename the bug report into "-H breaks the
incremental recursion on hard linked sources". This is as well true for all
the
three file systems tested.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=12769

--- Comment #4 from Ovidiu Stanila <ovidiu.stanila at jentla.com> ---
We hit the same issue on a CentOS 6 server (kernel 2.6.32-754.18.2.el6.x86_64),
the sync would break with the following error:

# /usr/bin/rsync --debug=HASH --stats --no-inc-recursive -aHn --delete /app/
<remote>:/app/
[sender] created hashtable 2013770 (size: 16, keys: 64-bit)
[sender] created hashtable 2015370 (size: 512, keys: 64-bit)
[sender] growing hashtable 2015370 (size: 1024, keys: 64-bit)
[sender] growing hashtable 2015370 (size: 2048, keys: 64-bit)
[sender] growing hashtable 2015370 (size: 4096, keys: 64-bit)
[sender] growing hashtable 2015370 (size: 8192, keys: 64-bit)
[sender] growing hashtable 2015370 (size: 16384, keys: 64-bit)
[sender] growing hashtable 2015370 (size: 32768, keys: 64-bit)
[sender] growing hashtable 2015370 (size: 65536, keys: 64-bit)
[sender] growing hashtable 2015370 (size: 131072, keys: 64-bit)
[sender] growing hashtable 2015370 (size: 262144, keys: 64-bit)
[sender] growing hashtable 2015370 (size: 524288, keys: 64-bit)
[sender] growing hashtable 2015370 (size: 1048576, keys: 64-bit)
[sender] growing hashtable 2015370 (size: 2097152, keys: 64-bit)
[sender] growing hashtable 2015370 (size: 4194304, keys: 64-bit)
[sender] growing hashtable 2015370 (size: 8388608, keys: 64-bit)
[sender] growing hashtable 2015370 (size: 16777216, keys: 64-bit)
[sender] growing hashtable 2015370 (size: 33554432, keys: 64-bit)
ERROR: out of memory in hashtable_node [sender]
rsync error: error allocating core memory buffers (code 22) at util2.c(106)
[sender=3.1.2]

Both the sender and receiver part have the same rsync and OS versions.

The source we try to transfer is over 3.4Tb (65 million files) with a big
number of hard links between various directories(for avoiding duplicate files).

We initially increased the memory from 16Gb to 32Gb but even with that rsync
would die with the same error. On above example, there were over 22Gb free RAM
at the point rsync stopped working.

We also tried "--inplace" instead of "--no-inc-recursive"
with the same result.

Did we hit some kind of limitation inside of rsync ? Is there anything else we
should check ?

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=12769

--- Comment #5 from Simon Matter <simon.matter at invoca.ch> ---
I'm suffering the same problem and was wondering if anyone found a solution
or
work around or other tool to do the job?

First I thought maybe it's a bug which is fixed already and tried with the
latest release 3.1.3. Unfortunately no joy and I still get the same issue.

Any help would be much appreciated!

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

On 11/10/2019 13:53, just subscribed for rsync-qa from bugzilla via
rsync wrote:
> https://bugzilla.samba.org/show_bug.cgi?id=12769
>
> --- Comment #5 from Simon Matter <simon.matter at invoca.ch> ---
> I'm suffering the same problem and was wondering if anyone found a
solution or
> work around or other tool to do the job?
>
> First I thought maybe it's a bug which is fixed already and tried with
the
> latest release 3.1.3. Unfortunately no joy and I still get the same issue.
>
> Any help would be much appreciated!
The hash table doubles each time it reaches 75% full.
A hash table for 32m items @ 16 bytes each (8 byte key, 8 byte void *
data) needs 512MB of memory.
At the next doubling (up to 64m items) it hits the array allocator limit
in utils2.c:

#define MALLOC_MAX 0x40000000

void *_new_array(unsigned long num, unsigned int size, int use_calloc)
{
??????? if (num >= MALLOC_MAX/size)
??????????????? return NULL;
??????? return use_calloc ? calloc(num, size) : malloc(num * size);
}
void *_realloc_array(void *ptr, unsigned int size, size_t num)
{
??????? if (num >= MALLOC_MAX/size)
??????????????? return NULL;
??????? if (!ptr)
??????????????? return malloc(size * num);
??????? return realloc(ptr, size * num);
}

No single array allocation or reallocation is allowed to exceed
MALLOC_MAX (1GB).
Hence rsync can only handle up to 32m items per invocation if a hash
table is required (e.g. for tracking hardlinks when -H is specified).

HTH,
Dave

https://bugzilla.samba.org/show_bug.cgi?id=12769

--- Comment #6 from Dave Gordon <dg32768 at zoho.eu> ---
The hash table doubles each time it reaches 75% full. A hash table for 32m
items @ 16 bytes each (8 byte key, 8 byte void *data) needs 512MB of memory. At
the next doubling (up to 64m items) it hits the array allocator limit in
utils2.c:

#define MALLOC_MAX 0x40000000

void *_new_array(unsigned long num, unsigned int size, int use_calloc)
{
        if (num >= MALLOC_MAX/size)
                return NULL;
        return use_calloc ? calloc(num, size) : malloc(num * size);
}
void *_realloc_array(void *ptr, unsigned int size, size_t num)
{
        if (num >= MALLOC_MAX/size)
                return NULL;
        if (!ptr)
                return malloc(size * num);
        return realloc(ptr, size * num);
}

No single array allocation or reallocation is allowed to exceed MALLOC_MAX
(1GB). Hence rsync can only handle up to 32m items per invocation if a hash
table is required (e.g. for tracking hardlinks when -H is specified).

HTH,
Dave

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=12769

--- Comment #7 from Simon Matter <simon.matter at invoca.ch> ---
I've patches like this to solve our issues:

--- rsync-3.1.3/util2.c.orig    2018-01-15 04:55:07.000000000 +0100
+++ rsync-3.1.3/util2.c    2020-03-11 13:07:07.138141415 +0100
@@ -59,7 +59,7 @@
     return True;
 }

-#define MALLOC_MAX 0x40000000
+#define MALLOC_MAX 0x100000000

 void *_new_array(unsigned long num, unsigned int size, int use_calloc)
 {

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=12769

--- Comment #8 from Roland Haberkorn <mss-it at fau.de> ---
Is it possible to totally get rid of the restriction? I'd prefer running in
out
of memory situations rather than in this restriction. Without I could just
throw some more RAM on the machine, with this restriction I would have to
rebuild rsync whenever there is a newer version.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=12769

--- Comment #14 from Roland Haberkorn <mss-it at fau.de> ---
After the new version made it into my system I can confirm it works like a
charm. Many thanks for the effort.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.