openssh unix dev - Jun 2002 - UsePrivilegeSeparation: "fatal: xrealloc: out of memory"

I just upgraded to OpenSSH 3.4p1 from 2.5.2p2 to take advantage of
privilege separation.  After installation, when a user tries to login
he gets dropped almost immediately.  In the server's
/var/log/messages:

Jun 26 20:15:04 sclp3 sshd[6433]: Accepted password for jason from
128.165.148.66 port 41871 ssh2
Jun 26 20:15:12 sclp3 jason[110]: sshd[6444]: fatal: xrealloc: out of memory
(new_size 5566464 bytes)

The server is running BSD/OS 4.0, whose mmap(2) seems to indicate that
it supports anonymous (MAP_ANON) memory mapping.  I've created
/var/empty and a sshd user and group.

Setting "UsePrivilegeSeparation no" in sshd_config clears up this
problem, but I'd rather not have to disable this.  Any ideas?

Thanks.

-- 
(http://tmda.sourceforge.net/)

"Jason R. Mastaler" wrote:
> Jun 26 20:15:04 sclp3 sshd[6433]: Accepted password for jason from
128.165.148.66 port 41871 ssh2
> Jun 26 20:15:12 sclp3 jason[110]: sshd[6444]: fatal: xrealloc: out of
memory (new_size 5566464 bytes)
> 
> Setting "UsePrivilegeSeparation no" in sshd_config clears up this
> problem, but I'd rather not have to disable this.  Any ideas?
Ulimits? Check with ulimit -a if the data size is <5.5M. If that's it
you'll need to increase it (eg "ulimit -d unlimited") and kill and
restart sshd.

-- 
Darren Tucker (dtucker at zip.com.au)
GPG Fingerprint D9A3 86E9 7EEE AF4B B2D4  37C9 C982 80C7 8FF4 FA69
    Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.

Hi,

On Wed, Jun 26, 2002 at 06:32:20PM -0600, Jason R. Mastaler
wrote:
> Jun 26 20:15:04 sclp3 sshd[6433]: Accepted password for jason from
128.165.148.66 port 41871 ssh2
> Jun 26 20:15:12 sclp3 jason[110]: sshd[6444]: fatal: xrealloc: out of
memory (new_size 5566464 bytes)
I've seen this on some old FreeBSD systems (2.2.7) when using "-2
-C".

Protocol 1 works with compression, Protocol 2 works without, but if you
use -2 -C, sshd will grow to about 25 Mbyte of memory, and then ulimit
will strike.

I'm not sure where the bug is, but something in the old system is
triggering it.

(There's another weirdness.  DSA operations on FreeBSD 2.2.x with x<7
reliably trigger a core dump crash, so no "-2" on these OSes at all).

gert
-- 
USENET is *not* the non-clickable part of WWW!
                                                           //www.muc.de/~gert/
Gert Doering - Munich, Germany                             gert at
greenie.muc.de
fax: +49-89-35655025                        gert.doering at
physik.tu-muenchen.de

Gert Doering wrote:
> On Thu, Jun 27, 2002 at 08:19:12PM +1000, Darren Tucker wrote:
> > Gert Doering gave an example of sshd using 25MB of memory (!) so you
> > might want try setting "ulimit -d unlimited" anyway.
> 
> This is a kind of a misunderstanding.  I do consider this a BUG - a sshd
> should never grow to a size larger than a few MB.
I did realise that. I only suggested it as a way of isolating the
problem.
Sorry for the misunderstanding.

-- 
Darren Tucker (dtucker at zip.com.au)
GPG Fingerprint D9A3 86E9 7EEE AF4B B2D4  37C9 C982 80C7 8FF4 FA69
    Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.