thr3ads.net - freebsd stable - 7.2 dies in midnight run again [Nov 2009]

If this information is useful, please help other people find it:
Share via:

Randy Bush

2009-Nov-20 00:45 UTC

7.2 dies in midnight run again

i think the issue is how to tune for zfs

i386 with 4G of RAM

RELENG_7 cvsupped Nov 18 02:42 GMT

    panic: kmem_malloc(65536): kmem_map too small: 535019520 total allocated
    cpuid = 0
    Uptime: 13h15m1s
    Physical memory: 3958 MB
    Dumping 637 MB: 622 606 590 574 558 542 526 510 494 478 462 446 430 414 398
382 366 350 334 318 302 286 270 254 238 222 206 190 174 158 142 126 110 94 78 62
46 30 14
    Dump complete
    Automatic reboot in 15 seconds - press a key on the console to abort

and it did not auto reboot

    # cat /boot/loader.conf.local
    ipfw_load=YES
    umass_load=YES
    zfs_load=YES
    vm.kmem_size=536870912
    vm.kmem_size_max=1073741824
    vfs.zfs.prefetch_disable=1

it has zfs 

    # zpool status
      pool: tank
     state: ONLINE
     scrub: none requested
    config:

	    NAME        STATE     READ WRITE CKSUM
	    tank        ONLINE       0     0     0
	      mirror    ONLINE       0     0     0
		twed1   ONLINE       0     0     0
		twed2   ONLINE       0     0     0

    errors: No known data errors

but boots and has root on ufs

    # df -H
    Filesystem        Size    Used   Avail Capacity  Mounted on
    /dev/twed0s1a     260M    199M     40M    83%    /
    devfs             1.0k    1.0k      0B   100%    /dev
    /dev/twed0s1h      65M    2.3M     57M     4%    /root
    procfs            4.1k    4.1k      0B   100%    /proc
    tank              147G     17M    147G     0%    /tank
    tank/usr          167G     20G    147G    12%    /usr
    tank/usr/home     216G     68G    147G    32%    /usr/home
    tank/var          149G    2.3G    147G     2%    /var
    tank/var/spool    148G    531M    147G     0%    /var/spool
    /dev/md0          130M     12k    119M     0%    /tmp

# kgdb kernel.debug /usr/home/crash/vmcore.20
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for
details.
This GDB was configured as "i386-marcel-freebsd"...

Unread portion of the kernel message buffer:
panic: kmem_malloc(65536): kmem_map too small: 535019520 total allocated
cpuid = 0
Uptime: 13h15m1s
Physical memory: 3958 MB
Dumping 637 MB: 622 606 590 574 558 542 526 510 494 478 462 446 430 414 398 382
366 350 334 318 302 286 270 254 238 222 206 190 174 158 142 126 110 94 78 62 46
30 14

Reading symbols from /boot/kernel/zfs.ko...Reading symbols from
/boot/kernel/zfs.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/zfs.ko
Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from
/boot/kernel/opensolaris.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/opensolaris.ko
Reading symbols from /boot/kernel/ipfw.ko...Reading symbols from
/boot/kernel/ipfw.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ipfw.ko
Reading symbols from /boot/kernel/umass.ko...Reading symbols from
/boot/kernel/umass.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/umass.ko
Reading symbols from /boot/kernel/cam.ko...Reading symbols from
/boot/kernel/cam.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/cam.ko
Reading symbols from /boot/kernel/usb.ko...Reading symbols from
/boot/kernel/usb.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/usb.ko
#0  doadump () at pcpu.h:196
196             __asm __volatile("movl %%fs:0,%0" : "=r"
(td));
(kgdb) back
#0  doadump () at pcpu.h:196
#1  0xc052b0b6 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:418
#2  0xc052b39e in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:574
#3  0xc06dfb54 in kmem_malloc (map=0xc107108c, size=65536, flags=2) at
/usr/src/sys/vm/vm_kern.c:305
#4  0xc06d6317 in page_alloc (zone=0x0, bytes=65536, pflag=0xf67ee4a7
"\002", wait=2) at /usr/src/sys/vm/uma_core.c:952
#5  0xc06d8e20 in uma_large_malloc (size=65536, wait=2) at
/usr/src/sys/vm/uma_core.c:2706
#6  0xc05189e8 in malloc (size=65536, mtp=0xc0989060, flags=2) at
/usr/src/sys/kern/kern_malloc.c:393
#7  0xc0897a61 in zfs_kmem_alloc (size=65536, kmflags=2) at
/usr/src/sys/modules/zfs/../../cddl/compat/opensolaris/kern/opensolaris_kmem.c:74
#8  0xc090bf4a in zio_buf_alloc (size=65536) at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:207
#9  0xc08f3472 in vdev_cache_read (zio=0xd39d0708) at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_cache.c:188
#10 0xc090c145 in zio_vdev_io_start (zio=0xd39d0708) at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:1816
#11 0xc090c7f0 in zio_execute (zio=0xd39d0708) at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:998
#12 0xc08f6bda in vdev_mirror_io_start (zio=0xd77e7708) at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_mirror.c:303
#13 0xc090c7f0 in zio_execute (zio=0xd77e7708) at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:998
#14 0xc08f6bda in vdev_mirror_io_start (zio=0xdbd22960) at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/vdev_mirror.c:303
#15 0xc090c7f0 in zio_execute (zio=0xdbd22960) at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zio.c:998
#16 0xc08ad39d in arc_read_nolock (pio=0xd1844960, spa=0xc7746000,
bp=0xcc548640, done=0xc08b0600 <dbuf_read_done>, private=0xd9c89e38,
priority=0, zio_flags=1, arc_flags=0xf67ee854, zb=0xf67ee834) at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:2762
#17 0xc08ad878 in arc_read (pio=0xd1844960, spa=0xc7746000, bp=0xcc548640,
pbuf=0xc9b29134, done=0xc08b0600 <dbuf_read_done>, private=0xd9c89e38,
priority=0, zio_flags=1, arc_flags=0xf67ee854, zb=0xf67ee834) at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:2507
#18 0xc08b0ada in dbuf_read (db=0xd9c89e38, zio=0xd1844960, flags=14) at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:521
#19 0xc08b1142 in dbuf_findbp (dn=0xcba92000, level=Variable "level"
is not available.
) at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:1381
#20 0xc08b1269 in dbuf_hold_impl (dn=0xcba92000, level=0 '\0', blkid=0,
fail_sparse=0, tag=0x0, dbp=0xf67ee8f0) at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:1617
#21 0xc08b2529 in dbuf_hold (dn=0xcba92000, blkid=0, tag=0x0) at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dbuf.c:1689
#22 0xc08b48cc in dmu_buf_hold (os=0xc774b3d0, object=167123, offset=0, tag=0x0,
dbp=0xf67ee95c) at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu.c:101
#23 0xc0900044 in zap_lockdir (os=0xc774b3d0, obj=167123, tx=0x0, lti=RW_READER,
fatreader=1, adding=0, zapp=0xf67eeba0) at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zap_micro.c:388
#24 0xc09009fd in zap_cursor_retrieve (zc=0xf67eeb9c, za=0xf67eea84) at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zap_micro.c:1004
#25 0xc0925a2b in zfs_freebsd_readdir (ap=0xf67eec00) at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:2156
#26 0xc07382f2 in VOP_READDIR_APV (vop=0xc098b560, a=0xf67eec00) at
vnode_if.c:1407
#27 0xc05becae in kern_getdirentries (td=0xd5a01000, fd=19, buf=0x29061000
<Address 0x29061000 out of bounds>, count=4096, basep=0xf67eec74) at
vnode_if.h:747
#28 0xc05beec1 in getdirentries (td=0xd5a01000, uap=0xf67eecfc) at
/usr/src/sys/kern/vfs_syscalls.c:3776
#29 0xc072cfc5 in syscall (frame=0xf67eed38) at
/usr/src/sys/i386/i386/trap.c:1101
#30 0xc0711380 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:262
#31 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)


randy

Stefan Esser

2009-Nov-20 10:38 UTC

head link

7.2 dies in midnight run again

Am 20.11.2009 01:45, schrieb Randy Bush:> i think the issue is how to tune for zfs
> 
> i386 with 4G of RAM
> 
> RELENG_7 cvsupped Nov 18 02:42 GMT
> 
>     panic: kmem_malloc(65536): kmem_map too small: 535019520 total
allocated
>     cpuid = 0
>     Uptime: 13h15m1s
>     Physical memory: 3958 MB
>     Dumping 637 MB: 622 606 590 574 558 542 526 510 494 478 462 446 430 414
398 382 366 350 334 318 302 286 270 254 238 222 206 190 174 158 142 126 110 94
78 62 46 30 14
>     Dump complete
>     Automatic reboot in 15 seconds - press a key on the console to abort
> 
> and it did not auto reboot
> 
>     # cat /boot/loader.conf.local
>     ipfw_load=YES
>     umass_load=YES
>     zfs_load=YES
>     vm.kmem_size=536870912
>     vm.kmem_size_max=1073741824
>     vfs.zfs.prefetch_disable=1
I'm using 8.0-i386 (but AFAIK with same ZFS and allocation behaviour)
on a system with 2GB RAM and a RAIDZ1 consisting of 3*1TB (plus
separate 10GB L2ARC cache on a separate disk, to become a SSD).

Besides increasing KVA_PAGES to cover some 2GB (options KVA_PAGES=512),
I use:

vm.kmem_size=1500M
vm.kmem_size_max=2G

This is the result of quite some tuning, since I also suffered from
kmem_map too small panics under high load.

BTW: I use auto-tuning of the ARC cache size:

vfs.zfs.arc_min: 122880000
vfs.zfs.arc_max: 983040000

Due to the extra ZFS cache drive, I could reduce arc_max to some 300MB
(or even lower, Though I did not test lower values, yet), without
noticeable impact (faster than with just a 700MB RAM cache in many
tests). And since L2ARC caches can be removed from pools at any time
and do not need redundancy, it is quite simple to test the effect ...

Regards, STefan

Randy Bush

2009-Nov-21 23:35 UTC

head link

7.2 dies in zfs

>> imiho, zfs can not be called production ready if it crashes if you
>> do not stand on your left leg, put your right hand in the air, and
>> burn some eye of newt.
> This is not a rant, but where do you read that on FreeBSD 7.2 ZFS has
> been marked as production ready.
> As far as i know, on FreeBSD 8.0 ZFS is called production ready.
whoops!  you are correct.  my apologies.
> Try running FreeBSD 7-Stable to get the latest ZFS version which on
> FreeBSD is 13
that is what i am running.  RELENG_7

randy

freebsd stable - Nov 2009 - 7.2 dies in midnight run again

7.2 dies in midnight run again

7.2 dies in midnight run again

7.2 dies in zfs