openssh bugs - Jun 2003 - [Bug 585] sshd core dumping on IRIX 6.5.18 with VerifyReverseMapping enabled

http://bugzilla.mindrot.org/show_bug.cgi?id=585

           Summary: sshd core dumping on IRIX 6.5.18 with
                    VerifyReverseMapping enabled
           Product: Portable OpenSSH
           Version: -current
          Platform: MIPS
        OS/Version: IRIX
            Status: NEW
          Severity: major
          Priority: P2
         Component: sshd
        AssignedTo: openssh-bugs at mindrot.org
        ReportedBy: ktaylor at daac.gsfc.nasa.gov


** I'm re-opening this case (it was bug #574). I don't think it got
entered
correctly into the system **


Occasionally, we're noticing that sshd is core dumping on our IRIX 6.5.18f
machine.

The only time we've noticed it is when users are logging in with putty
from offsite (although this is not really a client issue).

The user manages to log in, sshd apparently core dumps, but the user is not
logged out, the privilege separated user is still running their own personal
sshd spawn, and the parent is 1, so the root owned sshd process is gone.

wtmp is not updated, so the only way you can tell the user is logged in is by
listing their processes.

The end user doesn't notice that anything happened...and this doesn't
ALWAYS
happen, but I can't correlate any system event and this. It will happen when
the
system is first started, and it will happen when it's busier.



First core:

   6 record_login(pid = 13759, ttyname = 0x1014a22c = "/dev/ttyq7",
user 0x101520d8 = "user1", uid = ####, host = 0x101522a8
"pcp01711145pcs.nrockv01.md.`omcast.net", addr = 0x7fff24b0, addrlen =
16)
["/usr/local/src/security/openssh-3.6.1p1/sshlogin.c":72, 0x1002be58]


Second core:

   6 record_login(pid = 182438, ttyname = 0x1014a22c = "/dev/ttyq39",
user 0x101520d8 = "user2", uid = ####, host = 0x10152358
"toronto-hse-ppp3760148.symp`tico.ca", addr = 0x7fff24b0, addrlen =
16)
["/usr/local/src/security/openssh-3.6.1p1/sshlogin.c":72, 0x1002be58]


For some reason, the 29th character of the hostname is messed up. The first
hostname should be .comcast.net, the second hostname should be sympatico.ca

After looking through the source code, the actual problem may lie in 
verify_reverse_mapping.

We had this option enabled in sshd_config, we disabled it and are currently
monitoring for the core dumps. If we don't see any, that may be the root of
this
problem....hopefully it will point someone in the right direction towards fixing
it.

After about 2 weeks, we have not had any core files, so it was definately this
option causing the crashing problem.



------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.mindrot.org/show_bug.cgi?id=585





------- Additional Comments From djm at mindrot.org  2003-06-03 23:02 -------
It looks like the hostnames are being scribbled over by something. Perhaps a bug
in getaddrinfo()?

Is Irix using our getaddrinfo() replacement? (check for HAVE_GETADDRINFO in
config.h)

I doubt that the bug is in our canohost.c file, as it is used on all platforms.

Also, did you compile in 64-bit mode?



------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.mindrot.org/show_bug.cgi?id=585





------- Additional Comments From ktaylor at daac.gsfc.nasa.gov  2003-06-03 23:05
-------

/* Define to 1 if you have the `getaddrinfo' function. */
#define HAVE_GETADDRINFO 1

We compile in n32 mode.





------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.mindrot.org/show_bug.cgi?id=585





------- Additional Comments From ktaylor at daac.gsfc.nasa.gov  2003-06-03 23:09
-------

This was also in our config.h

/* getaddrinfo is broken (if present) */
/* #undef BROKEN_GETADDRINFO */


I'm not sure if it matters much that we're using openssh-3.6.1p1, not
p2.




------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.mindrot.org/show_bug.cgi?id=585





------- Additional Comments From djm at mindrot.org  2003-06-04 08:41 -------
Well, that indicates that you are using the system getaddrinfo function. We have
encountered bugs on some platforms' versions of these, but never ones
leading to
crashes.



------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.mindrot.org/show_bug.cgi?id=585





------- Additional Comments From djm at mindrot.org  2003-06-04 18:59 -------
*** Bug 574 has been marked as a duplicate of this bug. ***



------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.mindrot.org/show_bug.cgi?id=585





------- Additional Comments From djm at mindrot.org  2003-06-04 19:03 -------
I just discovered your debugger output in bug #574 - this looks like things are
blowing up inside malloc(). This is usually an indication that memory has been
trashed before the call. 

Consider building against ElectricFence[1] or some other malloc debugging
library. This would likely show up the error at the time the corruption happens.

[1] ftp://ftp.perens.com/pub/ElectricFence/ (I have no idea whether or not it
works on Irix)



------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.mindrot.org/show_bug.cgi?id=585





------- Additional Comments From ktaylor at daac.gsfc.nasa.gov  2003-06-04 20:19
-------
not having any luck getting to sites with dmalloc tools. Unfortunately I'm
not
very experienced with source debugging, so hopefully these things are easy to
implement. 




------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.mindrot.org/show_bug.cgi?id=585





------- Additional Comments From djm at mindrot.org  2003-06-04 20:24 -------
I should also warn you that electricfence drives up memory usage considerably



------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.mindrot.org/show_bug.cgi?id=585





------- Additional Comments From ktaylor at daac.gsfc.nasa.gov  2003-06-04 20:29
-------
that could be a problem then, the system we're seeing the problems on may
run
into troubles with high memory usage from sshd. 

I may try forcing sshd to build using your getaddrinfo, and maybe that will
clear things up temporarily, although may not solve the actual problem.
Unfortunately we don't have a good test scenario that can generate this
problem.
It has to happen on our main production box.



------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.mindrot.org/show_bug.cgi?id=585





------- Additional Comments From ktaylor at daac.gsfc.nasa.gov  2003-06-11 01:16
-------
due to the security bug, we re-enabled verifyreversemapping, and immediately saw
core dumps again, so that just proves we're looking in the right spot.

Luckily the users are not inconvenienced by this.

Tomorrow, we're going to try using the sshd binary that uses the non-system
getaddrinfo function. (we rebuilt after unsetting  HAVE_GETADDRINFO in config.h)
Hopefully that's all we needed to do.




------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.mindrot.org/show_bug.cgi?id=585





------- Additional Comments From ktaylor at daac.gsfc.nasa.gov  2003-06-12 02:12
-------
Ok. After using the fake-getaddrinfo, sshd is still crashing. Here's the
latest
dbx output.

Is there anything else we can look at without resorting to memory debugging?

>  0 realfree(0x10165f80, 0x10151490, 0x10165f60, 0x73706561, 0x73706560,
0x7ffed420, 0x10, 0x0)
["/xlv86/patches/5015/work/irix/lib/libc/libc_n32_M4/gen/malloc.c":527,
0xfb2466c]
   1 cleanfree(0x0, 0x10151490, 0x10165f60, 0x73706561, 0x73706560, 0x7ffed420,
0x10, 0x0)
["/xlv86/patches/5015/work/irix/lib/libc/libc_n32_M4/gen/malloc.c":944,
0xfb24eac]
   2 __malloc(0x260, 0x10151490, 0x10165f60, 0x73706561, 0x73706560, 0x7ffed420,
0x10, 0x0)
["/xlv86/patches/5015/work/irix/lib/libc/libc_n32_M4/gen/malloc.c":230,
0xfb240e0]
   3 _malloc(0x0, 0x10151490, 0x10165f60, 0x73706561, 0x73706560, 0x7ffed420,
0x10, 0x0)
["/xlv86/patches/5015/work/irix/lib/libc/libc_n32_M4/gen/malloc.c":186,
0xfb23f4c]
   4 xmalloc(size = 608)
["/usr/local/src/security/openssh-3.6.1p1/xmalloc.c":28, 0x10065994]
   5 login_alloc_entry(pid = 20692179, username = 0x10151490 =
"asdfa", hostname
= 0x10165f60 = "dsl093-055-063.blt1.dsl.spe`keasy.net", line =
0x1014a27c "/dev/ttyq25")
["/usr/local/src/security/openssh-3.6.1p1/loginrec.c":325,
0x10048b00]
   6 record_login(pid = 20692179, ttyname = 0x1014a27c =
"/dev/ttyq25", user 0x10151490 = "asdf", uid = ####, host =
0x10165f60 "dsl093-055-063.blt1.dsl.spe`keasy.net", addr = 0x7ffed420,
addrlen = 16)
["/usr/local/src/security/openssh-3.6.1p1/sshlogin.c":72, 0x1002beb8]
   7 mm_record_login(s = 0x1014a248, pw = 0x1015dc08)
["/usr/local/src/security/openssh-3.6.1p1/monitor.c":1030, 0x10042c84]
   8 mm_answer_pty(socket = 6, m = 0x7ffed510)
["/usr/local/src/security/openssh-3.6.1p1/monitor.c":1080, 0x10042f2c]
   9 monitor_read(pmonitor = 0x101527c0, ent = 0x10137790, pent = (nil))
["/usr/local/src/security/openssh-3.6.1p1/monitor.c":371, 0x10040f54]
   10 monitor_child_postauth(pmonitor = 0x101527c0)
["/usr/local/src/security/openssh-3.6.1p1/monitor.c":334, 0x10040dac]
   11 privsep_postauth(authctxt = 0x101515b0)
["/usr/local/src/security/openssh-3.6.1p1/sshd.c":665, 0x10025f78]
   12 main(ac = 1, av = 0x7ffedf14)
["/usr/local/src/security/openssh-3.6.1p1/sshd.c":1533, 0x10028a88]
   13 __start()
["/xlv55/kudzu-apr12/work/irix/lib/libc/libc_n32_M4/csu/crt1text.s":177,
0x10024a48]



------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.mindrot.org/show_bug.cgi?id=585





------- Additional Comments From dtucker at zip.com.au  2003-06-12 20:41 -------
Out of curiousity, what is MAXHOSTNAMELEN defined as on IRIX?



------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.mindrot.org/show_bug.cgi?id=585





------- Additional Comments From ktaylor at daac.gsfc.nasa.gov  2003-06-12 20:50
-------
param.h:#define MAXHOSTNAMELEN 256              /* can't be longer than
SYS_NMLN
- 1 */



------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.mindrot.org/show_bug.cgi?id=585





------- Additional Comments From ktaylor at daac.gsfc.nasa.gov  2003-06-12 20:55
-------
FYI

utsname.h:#define _SYS_NMLN     257     /* 4.0 size of utsname elements.*/




------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.mindrot.org/show_bug.cgi?id=585





------- Additional Comments From djm at mindrot.org  2003-06-13 09:20 -------
If you aren't already, you may want to try a CVS snapshot to see if the
problem
has already been fixed there. Otherwise you will have to try malloc debugging -
the crash is definitely occurring inside malloc.



------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.