freebsd stable - Feb 2007 - panic: kmem_malloc(16384): kmem_map too small: md-mounted /tmp filled up

The memory-mounted /tmp filled up on this 6.2-PRERELEASE system (as of Nov 7).

Unfortunately, instead of the process existing due to ENOSPC, the entire 
system paniced:

[...]
g_vfs_done():md0[WRITE(offset=982335488, length=131072)]error = 28
g_vfs_done():md0[WRITE(offset=982466560, length=131072)]error = 28
panic: kmem_malloc(16384): kmem_map too small: 259153920 total allocated
Uptime: 34d21h13m21s
Dumping 767 MB (2 chunks)
  chunk 0: 1MB (159 pages) ... ok
  chunk 1: 767MB (196288 pages) 751 735 719 703 687 671 655 639 623 607 591 
575 559 543 527 511 495 479 463 447 431 415 399 383 367 351 335 319 303 287 
271 255 239 223 207 191 175 159 143 127 111 95 79 63 47 31 15 ... ok

Dump complete
Automatic reboot in 15 seconds - press a key on the console to abort
Rebooting...

I don't think, such DoS-ing is normal -- a regular user shot the entire
system
in the knee...

Is anybody interested in the stack/etc.? Is this something, that's fixed in 
the more recent 6.2?

Machine has 3/4Gb RAM and ample swap:

	Device          1K-blocks     Used    Avail Capacity
	/dev/ad0s1b       3145728       32  3145696     0%
	/dev/ad2s1b       1048576       32  1048544     0%
	Total             4194304       64  4194240     0%

/tmp's space allocation (after reboot) is as follows:

	Filesystem 1K-blocks Used   Avail Capacity  Mounted on
	/dev/md0     2026030 3552 1860396     0%    /tmp

Note, that it is supposed to hold 2Gb, but was filled up and paniced holding 
about 300Mb... Probably, because it is created with ``-M'' by default --
but
it is not supposed to panic anyway!

Please, advise. Thanks!

	-mi

On Tue, Feb 27, 2007 at 10:59:08AM -0500, Mikhail Teterin
wrote:
> /tmp's space allocation (after reboot) is as follows:
> 
>   Filesystem 1K-blocks Used   Avail Capacity  Mounted on
>   /dev/md0     2026030 3552 1860396     0%    /tmp
> 
> Note, that it is supposed to hold 2Gb, but was filled up and paniced
> holding about 300Mb... Probably, because it is created with ``-M''
> by default -- but it is not supposed to panic anyway!
You may want to set -o reserve option. Or better, switch to a swap backed
md.


--
Adios

On Tue, Feb 27, 2007 at 02:48:11PM -0500, Mikhail Teterin
wrote:
> On Tuesday 27 February 2007 11:41, Alex Kozlov wrote:
> = On Tue, Feb 27, 2007 at 10:59:08AM -0500, Mikhail Teterin wrote:
> = > /tmp's space allocation (after reboot) is as follows:
> = > 
> = >   Filesystem 1K-blocks Used   Avail Capacity  Mounted on
> = >   /dev/md0     2026030 3552 1860396     0%    /tmp
> = > 
> = > Note, that it is supposed to hold 2Gb, but was filled up and paniced
> = > holding about 300Mb... Probably, because it is created with
``-M''
> = > by default -- but it is not supposed to panic anyway!
> = You may want to set -o reserve option. Or better, switch to a swap backed
> = md.
> 
> Yes, I switched to swap-backed md already. But the malloc-based variety is 
> currently the _default_ (see /etc/defaults/rc.conf)...
Bad default.
Anyway, you may wish to change tmpmfs_flags on something like that:
"-S -m0 -o async,nosymfollow,nosuid"
> Creation of a 2Gb malloc-based md should've failed on a machine with
768Mb of
> RAM, shouldn't it have?
Only if you set '-o reserve'. Memory for malloc-based md was allocated
dynamically.


--
Adios

Documented in the manpage, use swap backing or reserve enough space.

Kris

On Tue, Feb 27, 2007 at 10:59:08AM -0500, Mikhail Teterin
wrote:
> The memory-mounted /tmp filled up on this 6.2-PRERELEASE system (as of Nov
7).
> 
> Unfortunately, instead of the process existing due to ENOSPC, the entire 
> system paniced:
> 
> [...]
> g_vfs_done():md0[WRITE(offset=982335488, length=131072)]error = 28
> g_vfs_done():md0[WRITE(offset=982466560, length=131072)]error = 28
> panic: kmem_malloc(16384): kmem_map too small: 259153920 total allocated
> Uptime: 34d21h13m21s
> Dumping 767 MB (2 chunks)
>   chunk 0: 1MB (159 pages) ... ok
>   chunk 1: 767MB (196288 pages) 751 735 719 703 687 671 655 639 623 607 591
> 575 559 543 527 511 495 479 463 447 431 415 399 383 367 351 335 319 303 287
> 271 255 239 223 207 191 175 159 143 127 111 95 79 63 47 31 15 ... ok
> 
> Dump complete
> Automatic reboot in 15 seconds - press a key on the console to abort
> Rebooting...
> 
> I don't think, such DoS-ing is normal -- a regular user shot the entire
system
> in the knee...
> 
> Is anybody interested in the stack/etc.? Is this something, that's
fixed in
> the more recent 6.2?
> 
> Machine has 3/4Gb RAM and ample swap:
> 
> 	Device          1K-blocks     Used    Avail Capacity
> 	/dev/ad0s1b       3145728       32  3145696     0%
> 	/dev/ad2s1b       1048576       32  1048544     0%
> 	Total             4194304       64  4194240     0%
> 
> /tmp's space allocation (after reboot) is as follows:
> 
> 	Filesystem 1K-blocks Used   Avail Capacity  Mounted on
> 	/dev/md0     2026030 3552 1860396     0%    /tmp
> 
> Note, that it is supposed to hold 2Gb, but was filled up and paniced
holding
> about 300Mb... Probably, because it is created with ``-M'' by
default -- but
> it is not supposed to panic anyway!
> 
> Please, advise. Thanks!
> 
> 	-mi
> _______________________________________________
> freebsd-stable@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to
"freebsd-stable-unsubscribe@freebsd.org"
>

On Tuesday 27 February 2007 16:09, Kris Kennaway wrote:
= Documented in the manpage, use swap backing or reserve enough space.
= 
= Kris

The strings "panic" and "-o reserve" are mentioned in
neither mdmfs(8) nor in
rc.conf(5)... Is one supposed to look elsewhere?

Worse, the use of malloc-based mds is touted in rc.conf as something, that is 
supposed to help "system stability at low memory conditions".

All I did on this system, was set 

	tmpmfs="YES"
	tmpsize="2048m"

in the /etc/rc.conf.

Alex has already said, that using malloc-backed md is a bad default -- do you 
disagree?

	-mi

On Sun, Mar 04, 2007 at 10:59:46AM +0300, Yar Tikhiy
wrote:
> On Tue, Feb 27, 2007 at 04:03:30PM -0500, Mikhail Teterin wrote:
> > On Tuesday 27 February 2007 15:53, Alex Kozlov wrote:
> > = > Yes, I switched to swap-backed md already. But the malloc-based
variety is
> > = > currently the _default_ (see /etc/defaults/rc.conf)...
> > = Bad default.
> > 
> > Filing a PR.
> 
> Keep in mind that changing the default can break existing setups.
> Such setups are likely to be broken anyway, but...  E.g., if we
> drop the -M flag, it will break systems with tons of RAM but little
> swap using tmpmfs.  OTOH, if we add '-o reserve', that will break
> systems with a too large but mostly unused tmpmfs.  A sort of a loud
> announcement will be needed.
#sudo swapoff /dev/ad4s2b
#swapinfo
Device          1K-blocks     Used    Avail Capacity

#mdconfig -d -u 666 && \
mdconfig -a -t swap -s10000M -u 666 && \
bsdlabel -w /dev/md666 auto && \
newfs -U -f 512 -b 4096 -i 512 -m 0 /dev/md666a && \
mkdir /tmp/mdtest && \
mount /dev/md666a /tmp/mdtest
$dd if=/dev/zero of=/tmp/mdtest

#df /tmp/mdtest
Filesystem 1K-blocks Used    Avail Capacity  Mounted on
/dev/md666a 3561822 3561622 200 100% /tmp/mdtest
#dmesg
Mar  4 10:20:35 ravenloft kernel: pid 97529 (firefox-bin), uid 1001, was
killed: out of swap space
Mar  4 10:20:39 ravenloft kernel: pid 79658 (Xorg), uid 0, was killed: out
of swap space
Mar  4 10:20:40 ravenloft kernel: pid 519 (thunderbird-bin), uid 1001, was
killed: out of swap space

Ouch, but this is still better than panic.
> By the way, I seem to be the one responsible for the paragraph in
> rc.conf(5) advocating -M "for maximum performance and system stability
> at low memory conditions".  When I wrote it, I thought as follows:
> were the system low on memory, additional swap activity due to mfs
> would just make the system start thrashing sooner, while malloc'd
> mfs would just report ENOSPC.  I was unaware of the panic back then.
> Perhaps it was a misconception, either.  I haven't heard of any real
> pitfalls in swap-backed mfs, and getting ENOSPC on /tmp early is no
> good.
> 
> > = > Creation of a 2Gb malloc-based md should've failed on a
machine with
> > = > 768Mb of RAM, shouldn't it have?
> > = Only if you set '-o reserve'. Memory for malloc-based md was
allocated
> > = dynamically.
> > 
> > But malloc can only allocate from RAM, right? So the amount of RAM is
the hard
> > limit, which a malloc-based md can not exceed even in theory. This
means,
> > md-creation should've failed...
> > 
> > In fact, the limit should, of course, be even lower -- and mdmfs
should be
> > smart enough to substract the sizes of other kernel memory chunks from
the
> > maximum.
> > 
> > Since even that would still not be a guarantee against running out,
the system
> > should be able to recover gracefully instead of panicing. Do you
agree?
> 
> FWIW, some discussion of the panic is in the audit trail of kern/87255.
> Irrespective of tmpmfs defaults, this is a way to panic the system
> with stock tools and without doing something totally stupid like
> writing junk to /dev/mem.

--
Adios

in message <00e001c75fe2$0f3f4200$0c00a8c0@Artem>,
wrote Artem Kuchin thusly...

(something about SATA in IPMI thread, and previously, something
about IPMI in USB thread)


Artem,

Could you please stop hijacking threads?  If you need to start a new
thread, then please do not reply to a message of unrelated topic;
start a whole new message.  Or, at the very least, delete
In-reply-to: and References: headers, with subject line changed of
course.


  - Parv

--