Jeremy Chadwick
2009-Oct-05 17:42 UTC
Still possible to panic RELENG_7 with ZFS (kmem exhaustion)?
(Note: please keep me CC'd, as I am not subscribed to freebsd-stable)
Is it still possible with ZFS to panic a RELENG_7 amd64 box (kernel/world
from recent[1] source) with "kmem map too small" or similar
conditions?
Why I ask:
Our production SQL/backup box kernel panic'd a couple days ago. Sadly,
the box also acts as a serial console server, so I don't have the
exact message spit back from the kernel prior to being dumped to DDB,
but the backtrace looks very much like the historic problem of the ZFS
ARC exhausting all kernel memory, so I'm betting the message prior to
being dumped to DDB was "kmem map too small":
db> bt
Tracing pid 40738 tid 100168 td 0xffffff001f078720
kdb_enter_why() at kdb_enter_why+0x3d
panic() at panic+0x176
kmem_malloc() at kmem_malloc+0x548
uma_large_malloc() at uma_large_malloc+0x3c
malloc() at malloc+0xc1
arc_get_data_buf() at arc_get_data_buf+0x1bb
arc_buf_alloc() at arc_buf_alloc+0xa1
arc_read_nolock() at arc_read_nolock+0xd1
arc_read() at arc_read+0x71
dbuf_prefetch() at dbuf_prefetch+0x135
dmu_zfetch_dofetch() at dmu_zfetch_dofetch+0xe3
dmu_zfetch() at dmu_zfetch+0xa58
dbuf_read() at dbuf_read+0x433
dmu_buf_hold_array_by_dnode() at dmu_buf_hold_array_by_dnode+0x119
dmu_buf_hold_array() at dmu_buf_hold_array+0x57
dmu_read_uio() at dmu_read_uio+0x3f
zfs_freebsd_read() at zfs_freebsd_read+0x55a
vn_read() at vn_read+0x1ef
dofileread() at dofileread+0x88
kern_readv() at kern_readv+0x43
read() at read+0x4d
syscall() at syscall+0x247
Xfast_syscall() at Xfast_syscall+0xab
The machine in question has absolutely no loader.conf tuning applied,
and kernel/world was built from RELENG_7 dated 2009/06/11. The ZFS pool
consisted of a single (entire) disk; nothing special. I do not have
sysctl counters from the box before it panic'd, but these are what are
active presently:
hw.physmem: 4286558208
vm.kmem_size_max: 329853485875
vm.kmem_size_min: 0
vm.kmem_size: 1381478400
With regards to the above counters: ZFS is not in use. I had to switch
back to UFS2 (zpool destroy + newfs -O2 -U...) because of stability
concerns relating to the question at hand.
If someone familiar with the FreeBSD ZFS internals, and/or the VM,
please make a statement I think it would beneficial to those who are
considering using/migrating to ZFS on FreeBSD.
The only "semi-official" statements I've read as of late are here:
http://lists.freebsd.org/pipermail/freebsd-stable/2009-September/051810.html
http://lists.freebsd.org/pipermail/freebsd-stable/2009-September/051830.html
And what's in src/UPDATING:
20090207:
ZFS users on amd64 machines with 4GB or more of RAM should
reevaluate their need for setting vm.kmem_size_max and
vm.kmem_size manually. In fact, after recent changes to the
kernel, the default value of vm.kmem_size is larger than the
suggested manual setting in most ZFS/FreeBSD tuning guides.
Thanks!
[1]: "Recent" means post-February 2009, specifically after Alan
Cox's
commits listed here:
http://svn.freebsd.org/changeset/base/188291
http://svn.freebsd.org/changeset/base/187523
http://svn.freebsd.org/changeset/base/187522
http://svn.freebsd.org/changeset/base/187520
http://svn.freebsd.org/changeset/base/187485
http://svn.freebsd.org/changeset/base/187466
http://svn.freebsd.org/changeset/base/187465
http://svn.freebsd.org/changeset/base/187464
http://svn.freebsd.org/changeset/base/187458
http://svn.freebsd.org/changeset/base/187428
http://svn.freebsd.org/changeset/base/187425
http://svn.freebsd.org/changeset/base/187420
http://svn.freebsd.org/changeset/base/187419
http://svn.freebsd.org/changeset/base/187416
http://svn.freebsd.org/changeset/base/187414
http://svn.freebsd.org/changeset/base/187408
http://svn.freebsd.org/changeset/base/187407
http://svn.freebsd.org/changeset/base/187404
http://svn.freebsd.org/changeset/base/187400
--
| Jeremy Chadwick jdc@parodius.com |
| Parodius Networking http://www.parodius.com/ |
| UNIX Systems Administrator Mountain View, CA, USA |
| Making life hard for others since 1977. PGP: 4BD6C0CB |