Tim Chase
2009-Jun-04 15:05 UTC
kmem map too small panic after updating to STABLE-7 r192996
Hello,
I decided to give the new zfs code a try and upgraded my stable-7 system
and discovered a panic that I can reproduce at will.
This is an i386 system with 1.5GB of RAM and a single zfs pool:
appbuild# zpool status
pool: backup
state: ONLINE
status: The pool is formatted using an older on-disk format. The pool can
still be used, but some features are unavailable.
action: Upgrade the pool using 'zpool upgrade'. Once this is done,
the
pool will no longer be accessible on older software versions.
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
backup ONLINE 0 0 0
mirror ONLINE 0 0 0
ad4 ONLINE 0 0 0
ad6 ONLINE 0 0 0
errors: No known data errors
I had previously used the following zfs tuning:
vm.kmem_size="512M"
vm.kmem_size_max="512M"
vfs.zfs.arc_max="100M"
After getting this panic, I removed these tunables. The panic happens
in either case.
Reproducing the panic is easy: I keep a copy of the FreeBSD svn tree
in a compressed zfs file system (compression ratio runs around 1.35x).
An "svn update" (or the requisite "svn cleanup" following a
system crash)
in my checkout area will _always_ result in a "kmem map too small"
panic.
Here's a stack trace from my most recent panic:
(kgdb) bt
#0 doadump () at pcpu.h:196
#1 0xc052c963 in boot (howto=260) at
/root/stable-7/sys/kern/kern_shutdown.c:418
#2 0xc052cb9d in panic (fmt=Variable "fmt" is not available.
) at /root/stable-7/sys/kern/kern_shutdown.c:574
#3 0xc069fa2a in kmem_malloc (map=0xc105408c, size=131072, flags=2) at
/root/stable-7/sys/vm/vm_kern.c:305
#4 0xc0696587 in page_alloc (zone=0x0, bytes=131072, pflag=0xe6dbcb47
"\002", wait=2)
at /root/stable-7/sys/vm/uma_core.c:952
#5 0xc0699000 in uma_large_malloc (size=131072, wait=2) at
/root/stable-7/sys/vm/uma_core.c:2706
#6 0xc051af38 in malloc (size=131072, mtp=0xc094f060, flags=2) at
/root/stable-7/sys/kern/kern_malloc.c:393
#7 0xc085db00 in zfs_kmem_alloc (size=131072, kmflags=2)
at
/root/stable-7/sys/modules/zfs/../../cddl/compat/opensolaris/kern/opensolaris_kmem.c:74
#8 0xc08d1c79 in zio_buf_alloc (size=131072)
at
/root/stable-7/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:207
#9 0xc08bcc40 in vdev_queue_io_to_issue (vq=0xc492b334, pending_limit=Variable
"pending_limit" is not available.
)
at
/root/stable-7/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:227
#10 0xc08bce42 in vdev_queue_io_done (zio=0xd420a000)
at
/root/stable-7/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:313
#11 0xc08d4162 in zio_vdev_io_done (zio=0xd420a000)
at
/root/stable-7/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1847
#12 0xc08d2532 in zio_execute (zio=0xd420a000)
at
/root/stable-7/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:998
#13 0xc0862aa0 in taskq_thread (arg=0xc44a10cc)
at
/root/stable-7/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/os/taskq.c:854
#14 0xc0506816 in fork_exit (callout=0xc0862960 <taskq_thread>,
arg=0xc44a10cc, frame=0xe6dbcd38)
at /root/stable-7/sys/kern/kern_fork.c:811
#15 0xc06d4670 in fork_trampoline () at
/root/stable-7/sys/i386/i386/exception.s:264
(kgdb) print panicstr
$1 = 0xc0792320 "kmem_malloc(131072): kmem_map too small: 325656576 total
allocated"
The arc size is now auto-tuned as:
kstat.zfs.misc.arcstats.c_min: 26214400
kstat.zfs.misc.arcstats.c_max: 209715200
and while monitoring kstat.zfs.misc.arcstats.size I noticed it fluctuated
between around 140 and 150 million. Interestingly enough, immediately
before the last panic, It (arcstats.size) had just been reduced from
150038268 to 128790020.
I'm going to continue to poke around in my core dumps and see if I can
figure out what's going on. Any hints as to what to look for or monitor
while the system is running would be appreciated.
Thanks,
Tim
Angelo Turetta
2009-Jun-04 18:29 UTC
kmem map too small panic after updating to STABLE-7 r192996
Tim Chase wrote:> Hello, > > I decided to give the new zfs code a try and upgraded my stable-7 system > and discovered a panic that I can reproduce at will.I just had the same problem, and it turned out I was not diligent when I first set my zfs pool up :) To use vm.kmem_size="512M" you need to put options KVA_PAGES=512 in your kernel. I remember trying the loader.conf settings on a GENERIC kernel, found it worked and forgot to add KVA_PAGES. It seems that recently the memory allocated by a ZFS kernel on startup has become larger than the default KVA_PAGES in GENERIC. Hope this helps, Angelo Turetta
Ben Kelly
2009-Jun-04 19:15 UTC
kmem map too small panic after updating to STABLE-7 r192996
On Jun 4, 2009, at 10:48 AM, Tim Chase wrote:> vm.kmem_size="512M" > vm.kmem_size_max="512M" > vfs.zfs.arc_max="100M"> $1 = 0xc0792320 "kmem_malloc(131072): kmem_map too small: 325656576 > total allocated"It looks like you are suffering from fragmentation of your kmem address space. I've done some investigation of this on -current: http://docs.freebsd.org/cgi/getmsg.cgi?fetch=909067+0+archive/2009/freebsd-current/20090503.freebsd-current The patch linked from that mail allows me to run with an untuned ARC size with kmem of 512 MB. It does this, however, by more aggressively limiting how the ARC grows so it doesn't need to trigger its aggressive memory reclamation algorithm. I haven't tested this patch against -stable or with Kip's more recent changes so I don't know if it will apply cleanly for you. I've had some luck avoiding the fragmentation by changing ZFS to use a separate memory pool with a 128KB slab allocator, but this isn't really ready for general usage. Unfortunately I've been quite busy at work lately and haven't had time to work on it in a few weeks. I think with the current code base your only options are to increase your kmem maximum or limit your ARC even further. I hope that helps. - Ben
Kip Macy
2009-Jun-04 21:13 UTC
kmem map too small panic after updating to STABLE-7 r192996
As I mentioned in the initial e-mail, auto-tuning is only safe to rely on on amd64. -Kip On Thu, Jun 4, 2009 at 7:48 AM, Tim Chase <tim@chase2k.com> wrote:> Hello, > > I decided to give the new zfs code a try and upgraded my stable-7 system > and discovered a panic that I can reproduce at will. > > This is an i386 system with 1.5GB of RAM and a single zfs pool: > > ? ? ? ?appbuild# zpool status > ? ? ? ? ?pool: backup > ? ? ? ? state: ONLINE > ? ? ? ?status: The pool is formatted using an older on-disk format. ?The > pool can > ? ? ? ? ? ? ? ?still be used, but some features are unavailable. > ? ? ? ?action: Upgrade the pool using 'zpool upgrade'. ?Once this is done, > the > ? ? ? ? ? ? ? ?pool will no longer be accessible on older software versions. > ? ? ? ? scrub: none requested > ? ? ? ?config: > > ? ? ? ? ? ? ? ?NAME ? ? ? ?STATE ? ? READ WRITE CKSUM > ? ? ? ? ? ? ? ?backup ? ? ?ONLINE ? ? ? 0 ? ? 0 ? ? 0 > ? ? ? ? ? ? ? ? ?mirror ? ?ONLINE ? ? ? 0 ? ? 0 ? ? 0 > ? ? ? ? ? ? ? ? ? ?ad4 ? ? ONLINE ? ? ? 0 ? ? 0 ? ? 0 > ? ? ? ? ? ? ? ? ? ?ad6 ? ? ONLINE ? ? ? 0 ? ? 0 ? ? 0 > > ? ? ? ?errors: No known data errors > > I had previously used the following zfs tuning: > > ? ? ? ?vm.kmem_size="512M" > ? ? ? ?vm.kmem_size_max="512M" > ? ? ? ?vfs.zfs.arc_max="100M" > > After getting this panic, I removed these tunables. ?The panic happens > in either case. > > Reproducing the panic is easy: I keep a copy of the FreeBSD svn tree > in a compressed zfs file system (compression ratio runs around 1.35x). > An "svn update" (or the requisite "svn cleanup" following a system crash) > in my checkout area will _always_ result in a "kmem map too small" panic. > Here's a stack trace from my most recent panic: > > (kgdb) bt > #0 ?doadump () at pcpu.h:196 > #1 ?0xc052c963 in boot (howto=260) at > /root/stable-7/sys/kern/kern_shutdown.c:418 > #2 ?0xc052cb9d in panic (fmt=Variable "fmt" is not available. > ) at /root/stable-7/sys/kern/kern_shutdown.c:574 > #3 ?0xc069fa2a in kmem_malloc (map=0xc105408c, size=131072, flags=2) at > /root/stable-7/sys/vm/vm_kern.c:305 > #4 ?0xc0696587 in page_alloc (zone=0x0, bytes=131072, pflag=0xe6dbcb47 > "\002", wait=2) > ? ?at /root/stable-7/sys/vm/uma_core.c:952 > #5 ?0xc0699000 in uma_large_malloc (size=131072, wait=2) at > /root/stable-7/sys/vm/uma_core.c:2706 > #6 ?0xc051af38 in malloc (size=131072, mtp=0xc094f060, flags=2) at > /root/stable-7/sys/kern/kern_malloc.c:393 > #7 ?0xc085db00 in zfs_kmem_alloc (size=131072, kmflags=2) > ? ?at > /root/stable-7/sys/modules/zfs/../../cddl/compat/opensolaris/kern/opensolaris_kmem.c:74 > #8 ?0xc08d1c79 in zio_buf_alloc (size=131072) > ? ?at > /root/stable-7/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:207 > #9 ?0xc08bcc40 in vdev_queue_io_to_issue (vq=0xc492b334, > pending_limit=Variable "pending_limit" is not available. > ) > ? ?at > /root/stable-7/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:227 > #10 0xc08bce42 in vdev_queue_io_done (zio=0xd420a000) > ? ?at > /root/stable-7/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_queue.c:313 > #11 0xc08d4162 in zio_vdev_io_done (zio=0xd420a000) > ? ?at > /root/stable-7/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1847 > #12 0xc08d2532 in zio_execute (zio=0xd420a000) > ? ?at > /root/stable-7/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:998 > #13 0xc0862aa0 in taskq_thread (arg=0xc44a10cc) > ? ?at > /root/stable-7/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/os/taskq.c:854 > #14 0xc0506816 in fork_exit (callout=0xc0862960 <taskq_thread>, > arg=0xc44a10cc, frame=0xe6dbcd38) > ? ?at /root/stable-7/sys/kern/kern_fork.c:811 > #15 0xc06d4670 in fork_trampoline () at > /root/stable-7/sys/i386/i386/exception.s:264 > (kgdb) print panicstr > $1 = 0xc0792320 "kmem_malloc(131072): kmem_map too small: 325656576 total > allocated" > > > The arc size is now auto-tuned as: > > ? ? ? ?kstat.zfs.misc.arcstats.c_min: 26214400 > ? ? ? ?kstat.zfs.misc.arcstats.c_max: 209715200 > > and while monitoring kstat.zfs.misc.arcstats.size I noticed it fluctuated > between around 140 and 150 million. ?Interestingly enough, immediately > before the last panic, It (arcstats.size) had just been reduced from > 150038268 to 128790020. > > I'm going to continue to poke around in my core dumps and see if I can > figure out what's going on. ?Any hints as to what to look for or monitor > while the system is running would be appreciated. > > ? ? ? ?Thanks, > ? ? ? ?Tim > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" >-- When bad men combine, the good must associate; else they will fall one by one, an unpitied sacrifice in a contemptible struggle. Edmund Burke
Tim Chase
2009-Jun-04 22:46 UTC
kmem map too small panic after updating to STABLE-7 r192996
On Thu, 4 Jun 2009, Angelo Turetta wrote:> Tim Chase wrote: >> Hello, >> >> I decided to give the new zfs code a try and upgraded my stable-7 system >> and discovered a panic that I can reproduce at will. > > I just had the same problem, and it turned out I was not diligent when I > first set my zfs pool up :) > > To use vm.kmem_size="512M" you need to put > > options KVA_PAGES=512 >...Thanks for the reminder. I had forgotten about this one, too. With my kmem size @ 512M now and with KVA_PAGES at 512, my svn test case works just fine. I guess I should have paid closer attention to the actual panic message. - Tim
Alexandre "Sunny" Kovalenko
2009-Jun-08 00:27 UTC
kmem map too small panic after updating to STABLE-7 r192996
On Thu, 2009-06-04 at 20:28 +0200, Angelo Turetta wrote:> Tim Chase wrote: > > Hello, > > > > I decided to give the new zfs code a try and upgraded my stable-7 system > > and discovered a panic that I can reproduce at will. > > I just had the same problem, and it turned out I was not diligent when I > first set my zfs pool up :) > > To use vm.kmem_size="512M" you need to put > > options KVA_PAGES=512Are you sure this is the case? RabbitsDen# uname -a FreeBSD RabbitsDen.RabbitsLawn.verizon.net 7.2-STABLE FreeBSD 7.2-STABLE #0: Sun Jun 7 12:25:08 EDT 2009 root@RabbitsDen.RabbitsLawn.verizon.net:/usr/obj/usr/src/sys/TPX60 i386 RabbitsDen# grep KVA_PAGES /sys/i386/conf/TPX60 options KVA_PAGES=256 <========RabbitsDen# grep kmem /boot/loader.conf vm.kmem_size_max="512M" vm.kmem_size="512M" RabbitsDen# sysctl vm.kmem_size_max vm.kmem_size_max: 536870912 <========RabbitsDen# sysctl vm.kmem_size vm.kmem_size: 536870912 RabbitsDen# grep arc /boot/loader.conf vfs.zfs.arc_min="128M" vfs.zfs.arc_max="256M" -- Alexandre Kovalenko (????????? ?????????)