thr3ads.net - rsync - rsync buffer overflow detected [Apr 2017]

If this information is useful, please help other people find it:
Share via:

Boris Savelev

2017-Apr-14 17:22 UTC

rsync buffer overflow detected

Hello!

I use rsync from python on my Debian Jessie amd64 and get this error:
*** buffer overflow detected ***: <snip>/rsync terminated
======= Backtrace:
========/lib/x86_64-linux-gnu/libc.so.6(+0x731af)[0x7ffff78971af]
/lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7ffff791caa7]
/lib/x86_64-linux-gnu/libc.so.6(+0xf6cc0)[0x7ffff791acc0]
/lib/x86_64-linux-gnu/libc.so.6(+0xf8a17)[0x7ffff791ca17]
<snip>/rsync(+0x30c78)[0x555555584c78]
<snip>/rsync(+0x31cfe)[0x555555585cfe]
<snip>/rsync(+0x31ef6)[0x555555585ef6]
<snip>/rsync(+0x336ed)[0x5555555876ed]
<snip>/rsync(+0x22417)[0x555555576417]
<snip>/rsync(+0x2395e)[0x55555557795e]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7ffff7845b45]
<snip>/rsync(+0x7f89)[0x55555555bf89]

I guess that problem is about too many open fds
STR for this is a small script on python:
import os
import subprocess

F = 'test'
OPENS = 1600

cmd = [
    #'gdb', '--args',
    './rsync',
    '-aviH',
    '/etc/passwd',
    '/tmp/passwd'
]

for i in xrange(OPENS):
    fd = os.open(F, os.O_WRONLY | os.O_CREAT)
print(cmd)
subprocess.check_call(cmd)

I rebuild rsync-3.1.1 from Debian source with debug and -O1 and get bt from gdb:
(gdb) bt
#0  0x00007ffff7859067 in __GI_raise (sig=sig at entry=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007ffff785a448 in __GI_abort () at abort.c:89
#2  0x00007ffff78971b4 in __libc_message (do_abort=do_abort at entry=2,
fmt=fmt at entry=0x7ffff7989cb3 "*** %s ***: %s terminated\n")
    at ../sysdeps/posix/libc_fatal.c:175
#3  0x00007ffff791caa7 in __GI___fortify_fail
(msg=msg at entry=0x7ffff7989c4a "buffer overflow detected") at
fortify_fail.c:31
#4  0x00007ffff791acc0 in __GI___chk_fail () at chk_fail.c:28
#5  0x00007ffff791ca17 in __fdelt_chk (d=d at entry=1606) at fdelt_chk.c:25
#6  0x0000555555584c78 in safe_read (fd=fd at entry=1606,
buf=buf at entry=0x7fffffffa810 "\037", len=len at entry=4) at
io.c:245
#7  0x0000555555585cfe in read_buf (f=f at entry=1606,
buf=buf at entry=0x7fffffffa810 "\037", len=len at entry=4) at
io.c:1815
#8  0x0000555555585ef6 in read_int (f=f at entry=1606) at io.c:1711
#9  0x00005555555876ed in setup_protocol (f_out=1605, f_in=1606) at compat.c:158
#10 0x0000555555576417 in client_run (f_in=1606, f_out=1605,
pid=24793, argc=1, argv=0x5555557d5240) at main.c:1128
#11 0x000055555557795e in start_client (argv=0x5555557d5240, argc=1)
at main.c:1423
#12 main (argc=2, argv=0x5555557d5240) at main.c:1651

It looks like a bug, but I'm not sure)

--
Boris

devzero at web.de

2017-Apr-16 06:43 UTC

head link

Aw: rsync buffer overflow detected

What's the value of "i" when this happens and what are the system
ulimit values for the user running that?

Roland


> Gesendet: Freitag, 14. April 2017 um 19:22 Uhr
> Von: "Boris Savelev via rsync" <rsync at lists.samba.org>
> An: rsync at lists.samba.org
> Betreff: rsync buffer overflow detected
>
> Hello!
> 
> I use rsync from python on my Debian Jessie amd64 and get this error:
> *** buffer overflow detected ***: <snip>/rsync terminated
> ======= Backtrace: ========>
/lib/x86_64-linux-gnu/libc.so.6(+0x731af)[0x7ffff78971af]
> /lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7ffff791caa7]
> /lib/x86_64-linux-gnu/libc.so.6(+0xf6cc0)[0x7ffff791acc0]
> /lib/x86_64-linux-gnu/libc.so.6(+0xf8a17)[0x7ffff791ca17]
> <snip>/rsync(+0x30c78)[0x555555584c78]
> <snip>/rsync(+0x31cfe)[0x555555585cfe]
> <snip>/rsync(+0x31ef6)[0x555555585ef6]
> <snip>/rsync(+0x336ed)[0x5555555876ed]
> <snip>/rsync(+0x22417)[0x555555576417]
> <snip>/rsync(+0x2395e)[0x55555557795e]
> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7ffff7845b45]
> <snip>/rsync(+0x7f89)[0x55555555bf89]
> 
> I guess that problem is about too many open fds
> STR for this is a small script on python:
> import os
> import subprocess
> 
> F = 'test'
> OPENS = 1600
> 
> cmd = [
>     #'gdb', '--args',
>     './rsync',
>     '-aviH',
>     '/etc/passwd',
>     '/tmp/passwd'
> ]
> 
> for i in xrange(OPENS):
>     fd = os.open(F, os.O_WRONLY | os.O_CREAT)
> print(cmd)
> subprocess.check_call(cmd)
> 
> I rebuild rsync-3.1.1 from Debian source with debug and -O1 and get bt from
gdb:
> (gdb) bt
> #0  0x00007ffff7859067 in __GI_raise (sig=sig at entry=6) at
> ../nptl/sysdeps/unix/sysv/linux/raise.c:56
> #1  0x00007ffff785a448 in __GI_abort () at abort.c:89
> #2  0x00007ffff78971b4 in __libc_message (do_abort=do_abort at entry=2,
> fmt=fmt at entry=0x7ffff7989cb3 "*** %s ***: %s terminated\n")
>     at ../sysdeps/posix/libc_fatal.c:175
> #3  0x00007ffff791caa7 in __GI___fortify_fail
> (msg=msg at entry=0x7ffff7989c4a "buffer overflow detected") at
> fortify_fail.c:31
> #4  0x00007ffff791acc0 in __GI___chk_fail () at chk_fail.c:28
> #5  0x00007ffff791ca17 in __fdelt_chk (d=d at entry=1606) at fdelt_chk.c:25
> #6  0x0000555555584c78 in safe_read (fd=fd at entry=1606,
> buf=buf at entry=0x7fffffffa810 "\037", len=len at entry=4) at
io.c:245
> #7  0x0000555555585cfe in read_buf (f=f at entry=1606,
> buf=buf at entry=0x7fffffffa810 "\037", len=len at entry=4) at
io.c:1815
> #8  0x0000555555585ef6 in read_int (f=f at entry=1606) at io.c:1711
> #9  0x00005555555876ed in setup_protocol (f_out=1605, f_in=1606) at
compat.c:158
> #10 0x0000555555576417 in client_run (f_in=1606, f_out=1605,
> pid=24793, argc=1, argv=0x5555557d5240) at main.c:1128
> #11 0x000055555557795e in start_client (argv=0x5555557d5240, argc=1)
> at main.c:1423
> #12 main (argc=2, argv=0x5555557d5240) at main.c:1651
> 
> It looks like a bug, but I'm not sure)
> 
> --
> Boris
> 
> -- 
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
>

Boris Savelev

2017-Apr-17 08:14 UTC

head link

rsync buffer overflow detected

ulimit:
time(seconds)        unlimited
file(blocks)         unlimited
data(kbytes)         unlimited
stack(kbytes)        8192
coredump(blocks)     0
memory(kbytes)       unlimited
locked memory(kbytes) 64
process              64098
nofiles              65536
vmemory(kbytes)      unlimited
locks                unlimited


Rsync call at the end, so i = 1600

modified test script:

---8<---
import os
import subprocess
#import psutil

F = 'test'
OPENS = 1600

cmd = [
    #'gdb', '--args',
    #'strace', '-f', '-olog',
    './rsync',
    '-aviH',
    '/etc/passwd',
    '/tmp/passwd'
]

for i in xrange(OPENS):
    fd = os.open(F, os.O_WRONLY | os.O_CREAT)

os.system('ulimit -a')
#proc = psutil.Process()
#print len(proc.open_files())
subprocess.check_call(cmd)
---8<---

strace:
https://pastebin.com/Xu9Pq4R9

If system limit excided it may fall with EMFILE, I think

2017-04-16 9:43 GMT+03:00  <devzero at web.de>:> What's the value of "i" when this happens and what are the
system ulimit values for the user running that?
>
> Roland
>
>
>
>> Gesendet: Freitag, 14. April 2017 um 19:22 Uhr
>> Von: "Boris Savelev via rsync" <rsync at
lists.samba.org>
>> An: rsync at lists.samba.org
>> Betreff: rsync buffer overflow detected
>>
>> Hello!
>>
>> I use rsync from python on my Debian Jessie amd64 and get this error:
>> *** buffer overflow detected ***: <snip>/rsync terminated
>> ======= Backtrace: ========>>
/lib/x86_64-linux-gnu/libc.so.6(+0x731af)[0x7ffff78971af]
>> /lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7ffff791caa7]
>> /lib/x86_64-linux-gnu/libc.so.6(+0xf6cc0)[0x7ffff791acc0]
>> /lib/x86_64-linux-gnu/libc.so.6(+0xf8a17)[0x7ffff791ca17]
>> <snip>/rsync(+0x30c78)[0x555555584c78]
>> <snip>/rsync(+0x31cfe)[0x555555585cfe]
>> <snip>/rsync(+0x31ef6)[0x555555585ef6]
>> <snip>/rsync(+0x336ed)[0x5555555876ed]
>> <snip>/rsync(+0x22417)[0x555555576417]
>> <snip>/rsync(+0x2395e)[0x55555557795e]
>> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7ffff7845b45]
>> <snip>/rsync(+0x7f89)[0x55555555bf89]
>>
>> I guess that problem is about too many open fds
>> STR for this is a small script on python:
>> import os
>> import subprocess
>>
>> F = 'test'
>> OPENS = 1600
>>
>> cmd = [
>>     #'gdb', '--args',
>>     './rsync',
>>     '-aviH',
>>     '/etc/passwd',
>>     '/tmp/passwd'
>> ]
>>
>> for i in xrange(OPENS):
>>     fd = os.open(F, os.O_WRONLY | os.O_CREAT)
>> print(cmd)
>> subprocess.check_call(cmd)
>>
>> I rebuild rsync-3.1.1 from Debian source with debug and -O1 and get bt
from gdb:
>> (gdb) bt
>> #0  0x00007ffff7859067 in __GI_raise (sig=sig at entry=6) at
>> ../nptl/sysdeps/unix/sysv/linux/raise.c:56
>> #1  0x00007ffff785a448 in __GI_abort () at abort.c:89
>> #2  0x00007ffff78971b4 in __libc_message (do_abort=do_abort at entry=2,
>> fmt=fmt at entry=0x7ffff7989cb3 "*** %s ***: %s
terminated\n")
>>     at ../sysdeps/posix/libc_fatal.c:175
>> #3  0x00007ffff791caa7 in __GI___fortify_fail
>> (msg=msg at entry=0x7ffff7989c4a "buffer overflow detected")
at
>> fortify_fail.c:31
>> #4  0x00007ffff791acc0 in __GI___chk_fail () at chk_fail.c:28
>> #5  0x00007ffff791ca17 in __fdelt_chk (d=d at entry=1606) at
fdelt_chk.c:25
>> #6  0x0000555555584c78 in safe_read (fd=fd at entry=1606,
>> buf=buf at entry=0x7fffffffa810 "\037", len=len at entry=4)
at io.c:245
>> #7  0x0000555555585cfe in read_buf (f=f at entry=1606,
>> buf=buf at entry=0x7fffffffa810 "\037", len=len at entry=4)
at io.c:1815
>> #8  0x0000555555585ef6 in read_int (f=f at entry=1606) at io.c:1711
>> #9  0x00005555555876ed in setup_protocol (f_out=1605, f_in=1606) at
compat.c:158
>> #10 0x0000555555576417 in client_run (f_in=1606, f_out=1605,
>> pid=24793, argc=1, argv=0x5555557d5240) at main.c:1128
>> #11 0x000055555557795e in start_client (argv=0x5555557d5240, argc=1)
>> at main.c:1423
>> #12 main (argc=2, argv=0x5555557d5240) at main.c:1651
>>
>> It looks like a bug, but I'm not sure)
>>
>> --
>> Boris
>>
>> --
>> Please use reply-all for most replies to avoid omitting the mailing
list.
>> To unsubscribe or change options:
https://lists.samba.org/mailman/listinfo/rsync
>> Before posting, read:
http://www.catb.org/~esr/faqs/smart-questions.html
>>


-- 
Boris

Wayne Davison

2017-Apr-29 21:40 UTC

head link

rsync buffer overflow detected

On Fri, Apr 14, 2017 at 10:22 AM, Boris Savelev via rsync <
rsync at lists.samba.org> wrote:
> #8  0x0000555555585ef6 in read_int (f=f at entry=1606) at io.c:1711
>#9  0x00005555555876ed in setup_protocol (f_out=1605, f_in=1606)
at> compat.c:158
>
Since rsync is just trying to setup the protocol, I'd imagine that your
shell is outputting extraneous characters. You can try running something
manually, such as "ssh host echo hello", which should output only
"hello",
and "ssh host rsync --server . ." (note the 2 trailing dot args) which
should output nothing, and require a Ctrl-C to kill it.  You might be able
to work around the issue with something like -e'ssh -q'.

..wayne..
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.samba.org/pipermail/rsync/attachments/20170429/6f2f9070/attachment.html>

Lars Ellenberg

2017-May-04 14:41 UTC

head link

rsync buffer overflow detected

On Fri, Apr 14, 2017 at 08:22:29PM +0300, Boris Savelev via rsync
wrote:> I use rsync from python on my Debian Jessie amd64 and get this error:
> *** buffer overflow detected ***: <snip>/rsync terminated
> I rebuild rsync-3.1.1 from Debian source with debug and -O1 and get bt from
gdb:
> (gdb) bt
> #5  0x00007ffff791ca17 in __fdelt_chk (d=d at entry=1606) at fdelt_chk.c:25
> #6  0x0000555555584c78 in safe_read (fd=fd at entry=1606, buf=buf at
entry=0x7fffffffa810 "\037", len=len at entry=4) at io.c:245
That is FD_SET(fd, &r_fds); with fd >= FD_SETSIZE, which is 1024.
You cannot use select with file descriptor numbers >= FD_SETSIZE (or < 0),
and glibc is catching that.

The "buffer" that would overflow is the fd_set.

Maybe rsync could simply close all inherited file descriptors,
first things first, before it does anything else,
possibly after making sure fds 0,1,2 are open to somewhere,
to avoid any output to "supposedly" stdout/stderr to clobber
fds opened only later.  Similar to what lvm tools do in their
_check_standard_fds() and _close_stray_fds()?

But of course rsync could also say: not my problem, *you* (whatever
entity was spawning rsync) leaked file descriptors, learn to use
O_CLOEXEC resp. set FD_CLOEXEC, so only 0,1,2 will be inherited.

quick and dirty workaround:
use a wrapper script, close all fds >= 3 "just in case",
then exec rsync.
> It looks like a bug, but I'm not sure)
Thanks,

    Lars Ellenberg

Reasonably Related Threads

Search for more possibly parallel threads

rsync - Apr 2017 - rsync buffer overflow detected

rsync buffer overflow detected

Aw: rsync buffer overflow detected

rsync buffer overflow detected

rsync buffer overflow detected

rsync buffer overflow detected

Reasonably Related Threads