rsync - Jun 2015 - [Bug 11338] New: Rsync Crash - Segmentation fault

https://bugzilla.samba.org/show_bug.cgi?id=11338

            Bug ID: 11338
           Summary: Rsync Crash - Segmentation fault
           Product: rsync
           Version: 3.1.1
          Hardware: x64
                OS: Linux
            Status: NEW
          Severity: normal
          Priority: P5
         Component: core
          Assignee: wayned at samba.org
          Reporter: bsc.chauhan at gmail.com
        QA Contact: rsync-qa at samba.org

Created attachment 11170
  --> https://bugzilla.samba.org/attachment.cgi?id=11170&action=edit
Core file generated at crash

Dear Team,

We are facing a crash in rsync on one of our production servers. We have
updated rsync to the latest version 3.1.1 but still crash is observed. Below
call stack is generated by the crash:
(gdb)
#0  0x0000003ad7e25433 in __gconv_transform_ascii_internal () from
/lib64/libc.so.6
#1  0x0000003ad7e20264 in __gconv () from /lib64/libc.so.6
#2  0x0000003ad7e1f83c in iconv () from /lib64/libc.so.6
#3  0x000000000040d252 in iconvbufs ()
#4  0x0000000000423bce in rwrite ()
#5  0x0000000000423505 in rprintf ()
#6  0x000000000042b911 in whine_about_eof ()
#7  0x000000000042f5e2 in read_buf ()
#8  0x000000000042fde1 in read_int ()
#9  0x000000000043055d in setup_protocol ()
#10 0x000000000041ffb8 in client_run ()
#11 0x00000000004209c6 in main ()

Please let us know if any additional information is needed for debugging.

Regards,
Balveer Singh

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=11338

Wayne Davison <wayned at samba.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED

--- Comment #1 from Wayne Davison <wayned at samba.org> ---
You need to compile rsync with debug symbols (or load an rpm/deb/etc that
provides debug symbols) to get useful info in your core's backtrace.  You
should be figuring out why the call to iconvbufs is crashing by checking its
args (change level to rwrite() and look at the vars). Rsync is exiting due to
an EOF in the protocol stream (the whine_about_eof() call), but the message
that it outputs shouldn't cause the iconv code any issues.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=11338

--- Comment #2 from James Stevenson <james at stev.org> ---

I seem to see the same crash. Here is some additional details with symbols. It
seems to happen during network timeouts.

#0  0x00007fcc2e167b05 in utf8_internal_loop (step=0x560710d73f90,
step_data=0x560710d740c0, irreversible=0x7ffccf327d28, outend=0x560710d7dad0
"", outptrp=<synthetic pointer>, 
    inend=0x7ffccf3287f8 "\n", inptrp=0x7ffccf327e80) at
../iconv/loop.c:325
#1  __gconv_transform_utf8_internal (step=0x560710d73f90,
data=data at entry=0x560710d740c0, inptrp=inptrp at entry=0x7ffccf327e80,
inend=inend at entry=0x7ffccf3287f8 "\n", 
    outbufstart=outbufstart at entry=0x0,
irreversible=irreversible at entry=0x7ffccf327df8, do_flush=do_flush at entry=0,
consume_incomplete=consume_incomplete at entry=0)
    at ../iconv/skeleton.c:613
#2  0x00007fcc2e162722 in __gconv (cd=0x560710d740b0,
inbuf=inbuf at entry=0x7ffccf327e80, inbufend=0x7ffccf3287f8 "\n",
outbuf=outbuf at entry=0x7ffccf327e98, outbufend=<optimized out>, 
    irreversible=irreversible at entry=0x7ffccf327df8) at gconv.c:79
#3  0x00007fcc2e161fa1 in iconv (cd=cd at entry=0x560710d740b0,
inbuf=inbuf at entry=0x7ffccf327e80, inbytesleft=inbytesleft at
entry=0x7ffccf327e88,
outbuf=outbuf at entry=0x7ffccf327e98, 
    outbytesleft=outbytesleft at entry=0x7ffccf327e90) at iconv.c:52
#4  0x000056070fa1954f in iconvbufs (ic=ic at entry=0x560710d740b0,
in=in at entry=0x7ffccf327f20, out=out at entry=0x7ffccf327f00, flags=0) at
rsync.c:208
#5  0x000056070fa30191 in rwrite (code=<optimized out>, code at
entry=FERROR,
buf=<optimized out>, 
    buf at entry=0x7ffccf3287b0 "rsync: connection unexpectedly closed (0
bytes
received so far) [sender]\n", len=<optimized out>,
is_utf8=<optimized out>,
is_utf8 at entry=0) at log.c:374
#6  0x000056070fa307e3 in rprintf (code=code at entry=FERROR,
format=format at entry=0x56070fa677b0 "rsync: connection unexpectedly
closed (%s
bytes received so far) [%s]\n") at log.c:433
#7  0x000056070fa39064 in whine_about_eof (allow_kluge=allow_kluge at entry=0)
at
io.c:231
#8  0x000056070fa3c992 in read_buf (f=f at entry=5, buf=buf at
entry=0x7ffccf329cd0
"\037", len=len at entry=4) at io.c:1834
#9  0x000056070fa3caa1 in read_int (f=f at entry=5) at io.c:1729
#10 0x000056070fa3e48c in setup_protocol (f_out=4, f_in=5) at compat.c:158
#11 0x000056070fa2d4b0 in client_run (f_in=5, f_out=4, pid=9022, argc=1,
argv=0x560710d74760) at main.c:1110
#12 0x000056070fa109a1 in start_client (argv=0x560710d74760, argc=1) at
main.c:1420
#13 main (argc=2, argv=0x560710d74760) at main.c:1648
(gdb) f 4
#4  0x000056070fa1954f in iconvbufs (ic=ic at entry=0x560710d740b0,
in=in at entry=0x7ffccf327f20, out=out at entry=0x7ffccf327f00, flags=0) at
rsync.c:208
208    in rsync.c
(gdb) p flags
$15 = 0
(gdb) p xbuf
Attempt to use a type name as an expression
(gdb) p in
$16 = (xbuf *) 0x7ffccf327f20
(gdb) p *in
$17 = {buf = 0x7ffccf3287b0 "rsync: connection unexpectedly closed (0 bytes
received so far) [sender]\n", pos = 18889, len = 18446744073709532799, size
18446744073709551615}
(gdb) p *out
$18 = {buf = 0x7ffccf327f40 "\"\023", pos = 0, len = 0, size =
1024}

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=11338

--- Comment #3 from Ralph Corderoy <ralph-sambabugz at inputplus.co.uk>
---
Hi Wayne,

I don't know if James's SEGV is the same root cause as Balveer's
that
was two years earlier, but I'm seeing SEGV like James's.  Both times, my
network connection was down when rsync was run by cron;  sounds similar
to James's `network timeouts'.

This is on Arch Linux with rsync 3.1.2-8 that depends on acl 2.2.52-4,
attr 2.4.47-3, glibc 2.26-8, and popt 1.16-9.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=11338

--- Comment #4 from Michal Ruprich <mruprich at redhat.com> ---
Hi,

we have encountered similar segfault a couple of years ago. As you can see here
in comment #2, the p *in shows that the input buffer's size and len are
unreasonably huge - len = 18446744073709532799, size = 18446744073709551615,
which leads to an assumption that there was a buffer overflow here. We were
never able to create a reliable reproducer for this but after some analysis, we
think that the problem is in the rwrite function:

#ifdef ICONV_CONST
    if (ic != (iconv_t)-1) {
        xbuf outbuf, inbuf;
        char convbuf[1024];
        int ierrno;

        INIT_CONST_XBUF(outbuf, convbuf);
        INIT_XBUF(inbuf, (char*)buf, len, (size_t)-1);

        while (inbuf.len) {
            iconvbufs(ic, &inbuf, &outbuf, inbuf.pos ? 0 : ICB_INIT);
            ierrno = errno;
            if (outbuf.len) {
                filtered_fwrite(f, convbuf, outbuf.len, 0); 
                outbuf.len = 0;
            }
            if (!ierrno || ierrno == E2BIG)
                continue;
            fprintf(f, "\\#%03o", CVAL(inbuf.buf, inbuf.pos++));
            inbuf.len--;     <============== no check for the buffer length
        }
    } else
#endif

The problem is(probably) that the input buffer length is changed in the
iconvbufs function. And the rwrite function is dependent on the iconvbufs
return values. I know that there have been changes in the iconvbufs code over
the years and without a reproducer it is hard to test this but we came up with
a patch to make the rsync code in rwite a little bit more robust. Adding the
patch.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=11338

--- Comment #5 from Michal Ruprich <mruprich at redhat.com> ---
Created attachment 14872
  --> https://bugzilla.samba.org/attachment.cgi?id=14872&action=edit
Patch for the rwrite function

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=11338

--- Comment #6 from Michal Ruprich <mruprich at redhat.com> ---
This has just freshly arrived yesterday. Same bug as before and as described
here. 

https://bugzilla.redhat.com/show_bug.cgi?id=1683737

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=11338

Wayne Davison <wayne at opencoder.net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |FIXED
             Status|ASSIGNED                    |RESOLVED

--- Comment #7 from Wayne Davison <wayne at opencoder.net> ---
Thanks for the patch, Michal!  That looks quite reasonable to me, so I have
committed it to git.  Reopen this bug if someone can still get the issue to
trigger with the patched source.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.

https://bugzilla.samba.org/show_bug.cgi?id=11338

--- Comment #8 from Michal Ruprich <mruprich at redhat.com> ---
Great, thanks Wayne.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.