freebsd stable - Jul 2003 - malloc does not return null when out of memory

We have a little soekris box running freebsd that uses racoon for key
management.  It's used for setting up an ipsec tunnel.  I noticed that
one of these devices lost the tunnel this morning.  I looked in the
log and saw this

Jul 23 01:37:57 m0n0wall /kernel: pid 80 (racoon), uid 0, was killed: out of
swap space

I reproduced this problem using this code.

#include <stdlib.h>
int
main(int argc, char **argv)
{
        char *p;
        int cnt = 0;

        while (1) {
                p = malloc(100000);
                printf("%d %p\n", ++cnt, p);
                if (p)
                        memset(p, 0, 100000);
                else
                        break;
        }
        return 0;
}

Here is the tail end of the output.  It dies when trying to poke at
the memory using memset.  If I just malloc without the memset, it
never even dies.

210 0x94b5000
211 0x94ce000
212 0x94e7000
213 0x9500000
214 0x9519000
215 0x9532000
216 0x954b000
217 0x9564000
218 0x957d000
219 0x9596000
220 0x95af000
221 0x95c8000
Killed

The kernel for this has the NO_SWAPPING option enabled since there is
essentially no disk, it boots from a CF.

On Wed, 23 Jul 2003, Gabor wrote:
> We have a little soekris box running freebsd that uses racoon for key
> management.  It's used for setting up an ipsec tunnel.  I noticed that
> one of these devices lost the tunnel this morning.  I looked in the
> log and saw this
>
> Jul 23 01:37:57 m0n0wall /kernel: pid 80 (racoon), uid 0, was killed: out
of swap space
[...]
> Here is the tail end of the output.  It dies when trying to poke at
> the memory using memset.  If I just malloc without the memset, it
> never even dies.
That's normal. If you malloc() memory and never touch it then it is never
actually allocated.

Your problem is that you're running out of memory.  Try killing off
unneeded daemons and set maxusers to a low value (like 32).  Also try
reducing the size of your mfs partitions, if you're using rc.diskless2. I
don't think raccoon is much of a memory pig, unless you have a huge number
of connections.
> The kernel for this has the NO_SWAPPING option enabled since there is
> essentially no disk, it boots from a CF.
Do you have the 32MB or 64MB version? I have a net4501 (64MB) and I still
have 39MB free.  Its just running natd, sshd, named and dhcpd.

-- 
Doug White                    |  FreeBSD: The Power to Serve
dwhite@gumbysoft.com          |  www.FreeBSD.org

> The kernel for this has the NO_SWAPPING option enabled since there is
> essentially no disk, it boots from a CF.
man malloc.  In particular, see the MALLOC_OPTIONS you might place
in /etc/malloc.conf

On Wed, Jul 23, 2003 at 01:34:27PM -0400, Gabor wrote:
> Here is the tail end of the output.  It dies when trying to poke at
> the memory using memset.  If I just malloc without the memset, it
> never even dies.
Ah, the annual "memory overcommit" thread.  I thought we were overdue
for one.

Kris
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url :
http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20030723/913d12bd/attachment.bin

At 08:15 PM 7/23/2003 -0700, Kris Kennaway wrote:
>On Wed, Jul 23, 2003 at 01:34:27PM -0400, Gabor wrote:
>
> > Here is the tail end of the output.  It dies when trying to poke at
> > the memory using memset.  If I just malloc without the memset, it
> > never even dies.
>
>Ah, the annual "memory overcommit" thread.  I thought we were
overdue
>for one.

But why does the man page for malloc (3) say,

     If malloc() fails, a NULL pointer is returned.

         ---Mike
--------------------------------------------------------------------
Mike Tancsa,                          	          tel +1 519 651 3400
Sentex Communications,     			  mike@sentex.net
Providing Internet since 1994                    www.sentex.net
Cambridge, Ontario Canada			  www.sentex.net/mike

On Wed, Jul 23, 2003 at 11:44:11PM -0400, Mike Tancsa
wrote:
> At 08:15 PM 7/23/2003 -0700, Kris Kennaway wrote:
> >On Wed, Jul 23, 2003 at 01:34:27PM -0400, Gabor wrote:
> >
> >> Here is the tail end of the output.  It dies when trying to poke
at
> >> the memory using memset.  If I just malloc without the memset, it
> >> never even dies.
> >
> >Ah, the annual "memory overcommit" thread.  I thought we were
overdue
> >for one.
> 
> 
> But why does the man page for malloc (3) say,
> 
>     If malloc() fails, a NULL pointer is returned.
Words fail me.

Kris
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
Url :
http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20030724/599910c1/attachment.bin

The Single UNIX Specification extends malloc() function, a little.
Following URL is of the manual page of this function:
http://www.opengroup.org/onlinepubs/007904975/functions/malloc.html
Read "ERRORS" and "Issue 6" sections. Should FreeBSD be
conformed to
this standard?

__________________________________
Do you Yahoo!?
Yahoo! SiteBuilder - Free, easy-to-use web site design software
http://sitebuilder.yahoo.com

Rostislav Krasny wrote:
> The Single UNIX Specification extends malloc() function, a little.
> Following URL is of the manual page of this function:
> http://www.opengroup.org/onlinepubs/007904975/functions/malloc.html
> Read "ERRORS" and "Issue 6" sections. Should FreeBSD be
conformed to
> this standard?
FreeBSD malloc is POSIX/SUSv3 compliant.  It returns NULL and sets
errno to ENOMEM if insufficient storage space is available.

Yes, I thought briefly about something like this.
   Then I thought 'there's a race condition'.
   Then I realised that other processes might not link against this
   malloc.
   Then I realised the race condition doesn't even matter; processes will
   still be killed, as the kernel doesn't care that you're still in
   malloc() when the overcommitted memory is touched, it just knows
   you've touched it and there's no actual memory there. This will
result
   in far more processes being killed. I believe that's a bad thing.
   Any patch for this is going to have to be in kernelland not userland.
   Cheers,
   Matt

   From: "Chuck Swiger" <cswiger@mac.com>
   Date: 24/07/03 15:16
   I don't think the following is a particularly good idea, as the
   existing prezero
   ('Z') or junk ('J') options will also serve to reference
memory and
   prevent
   the "memory overcommit issue", but:
   <patch to touch memory just after it's sbrk'd>

> > The Single UNIX Specification extends malloc() function, a little.
> > Following URL is of the manual page of this function:
> > http://www.opengroup.org/onlinepubs/007904975/functions/malloc.html
> > Read "ERRORS" and "Issue 6" sections. Should
FreeBSD be conformed
to
> > this standard?
>
> FreeBSD malloc is POSIX/SUSv3 compliant.  It returns NULL and sets
> errno to ENOMEM if insufficient storage space is available.
Then why Gabor <gabor@vmunix.com> reported that it doesn't return NULL
when insufficient storage space is available? Am I misunderstand something?

__________________________________
Do you Yahoo!?
Yahoo! SiteBuilder - Free, easy-to-use web site design software
http://sitebuilder.yahoo.com

[hopefully I didn't mess up the attribution, replying to a digest sucks]

On Thu, 24 Jul 2003, Jamie Bowden wrote:
> On Wed, 23 Jul 2003, Don Lewis wrote:
>
> > On 23 Jul, Brooks Davis wrote:
>
> > > No there isn't.  Overcommit is a fundamental design feature
of the BSD
> > > VM.  If you don't like it, find an OS that doesn't do it.
The only one
> > > I can think of off the top of my head in Irix where I've
found it to be
> > > a serious pain in the ass.
>
> Irix allows for use of what it calls 'vswap' as well, which is just
a file
> that lives on a normal filesystem that the system can fall back on when
> actual RAM and swap are exhausted.  It allows for extremely large commits
> on memory limited systems while guaranteeing that it has somewhere to page
> out to if necessary.  The poster above may not like Irix, but I do.
> There's a lot of good stuff in there, but SGI aren't the best at
telling
> people it exists and how to use it.
Almost.  Having actually worked on the IRIX VM system, and on vswap
issues in particular...

Don is correct in that IRIX doesn't overcommit.  But that is only
the default swap configuration.

The amount of "vswap" you add to the system is the amount by which
the VM system is allowed to overcommit total memory (real swap + RAM).
If the system is overcommitted and runs into a genuine memory outage,
then processes will start to be killed (hopefully intelligently).

In particular "vswap" is not a file that lives in the normal
filesystem
that gets swapped to.  The somewhat ugly syntax for adding vswap
means that vswap does get configured along with a normal swap area
(file or device), but vswap itself has no backing store.

Clear as mud?

BTW, I've found that more people are confused by memory overcommitting
than the abscence thereof.  I've found an analogy to overbooking airline
flights to be useful in clarifying the situation.  That said, I think
overcommitting is the right thing to do for typical system usage, but
that this behavior should be configurable for the unusual cases.

Brent

-- 
Brent Casavant			http://www.angeltread.org/
44.9067N 93.0558N 907F		-.- -.. ..... . -- -...