dennis berger
2013-Dec-10 12:09 UTC
freebsd 9.1 kernel panic / kmem_malloc(2887680): kmem _map too small: 28092231680 total allocated
Dear list,
in the last days our storage system crashed (kernel panic savecore: reboot after
panic: kmem_malloc(2887680): kmem
_map too small: 28092231680 total allocated) two times. After the first time
I've activated a crash device in order to get kernel core dump, which I got
the second time it crashed.
Both times the crash happened at night around 3 o'clock when we clean up old
snapshots, backup the filesystems and locatedb gets updated.
Additionally I recognized that one device name is too long and geom was
complaining about it.
I?ve renamed this to a shorter one today, but it?s unclear if that was the
cause. In any case I?ve attached the error message as well.
Have you a hint for what the cause could be and if it?s fixed in recent stable?
best,
-dennis
= OS
* FreeBSD 9.1 release plus a patched opensolaris.ko/zfs.ko which includes one
fix for a zfs deadlock in arc handler.
= Software
* kernel nfsv3 with 16 kernel threads.
* istgt accessing zvols
* samba3 accessing zfs filesystems directly.
= Attachments
== opensolaris.ko patch mentioned above =
--- a/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c
+++ b/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c
@@ -3720,8 +3720,16 @@ arc_lowmem(void *arg __unused, int
howto __unused)
mutex_enter(&arc_reclaim_thr_lock);
needfree = 1;
cv_signal(&arc_reclaim_thr_cv);
- while (needfree)
- msleep(&needfree, &arc_reclaim_thr_lock, 0, "zfs:lowmem",
0);
+
+ /*
+ * It is unsafe to block here in arbitrary threads, because
we can come
+ * here from ARC itself and may hold ARC locks and thus risk
a deadlock
+ * with ARC reclaim thread.
+ */
+ if (curproc == pageproc) {
+ while (needfree)
+ msleep(&needfree, &arc_reclaim_thr_lock, 0, "zfs:lowmem",
0);
+ }
mutex_exit(&arc_reclaim_thr_lock);
mutex_exit(&arc_lowmem_lock);
== Geom error message ( hundreds of them ) ==
Dec 10 11:09:02 freenas kernel: g_dev_taste: make_dev_p() failed
(gp->name=zvol/vmpool/XXXXXXXXXXXXXXXXXX at hourly-2013-12-10_11.09.01--1ds2,
error=63)
== Kernel debugger + backtrace =
# kgdb /boot/kernel/kernel /var/crash/vmcore.0
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for
details.
This GDB was configured as "amd64-marcel-freebsd"...
Unread portion of the kernel message buffer:
panic: kmem_malloc(2887680): kmem_map too small: 28092231680 total allocated
cpuid = 6
KDB: stack backtrace:
#0 0xffffffff809208a6 at kdb_backtrace+0x66
#1 0xffffffff808ea8be at panic+0x1ce
#2 0xffffffff80b44930 at vm_map_locked+0
#3 0xffffffff80b3b41a at uma_large_malloc+0x4a
#4 0xffffffff808d5a69 at malloc+0xd9
#5 0xffffffff80928931 at sbuf_extend+0x51
#6 0xffffffff809289ef at sbuf_put_byte+0x6f
#7 0xffffffff809235d3 at kvprintf+0x83
#8 0xffffffff80928b96 at sbuf_vprintf+0x26
#9 0xffffffff80928c10 at sbuf_printf+0x60
#10 0xffffffff8085cad8 at g_part_dumpconf+0x98
#11 0xffffffff80854359 at g_conf_specific+0x349
#12 0xffffffff80854e4c at g_run_events+0x1ec
#13 0xffffffff808bb9ef at fork_exit+0x11f
#14 0xffffffff80bc368e at fork_trampoline+0xe
Uptime: 4d23h40m31s
Dumping 17627 out of 49123 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%
(kgdb) bt
#0 doadump (textdump=Variable "textdump" is not available.
) at pcpu.h:224
#1 0xffffffff808ea3a1 in kern_reboot (howto=260)
at /usr/src/sys/kern/kern_shutdown.c:448
#2 0xffffffff808ea897 in panic (fmt=0x1 <Address 0x1 out of bounds>)
at /usr/src/sys/kern/kern_shutdown.c:636
#3 0xffffffff80b44930 in kmem_malloc (map=Variable "map" is not
available.
) at /usr/src/sys/vm/vm_kern.c:304
#4 0xffffffff80b3b41a in uma_large_malloc (size=2887680, wait=2)
at /usr/src/sys/vm/uma_core.c:3067
#5 0xffffffff808d5a69 in malloc (size=2887680, mtp=0xffffffff811a22c0, flags=2)
at /usr/src/sys/kern/kern_malloc.c:492
#6 0xffffffff80928931 in sbuf_extend (s=0xfffffe091201d100, addlen=Variable
"addlen" is not available.
)
at /usr/src/sys/kern/subr_sbuf.c:159
#7 0xffffffff809289ef in sbuf_put_byte (s=0xfffffe091201d100, c=116)
at /usr/src/sys/kern/subr_sbuf.c:369
#8 0xffffffff809235d3 in kvprintf (fmt=0xffffffff80e19c63 ">\n",
func=0xffffffff80928a20 <sbuf_putc_func>, arg=0xfffffe091201d100,
radix=10,
ap=Variable "ap" is not available.
) at /usr/src/sys/kern/subr_prf.c:737
#9 0xffffffff80928b96 in sbuf_vprintf (s=0xfffffe091201d100, fmt=Variable
"fmt" is not available.
)
at /usr/src/sys/kern/subr_sbuf.c:544
#10 0xffffffff80928c10 in sbuf_printf (s=Variable "s" is not
available.
) at /usr/src/sys/kern/subr_sbuf.c:626
#11 0xffffffff8085cad8 in g_part_dumpconf (sb=0xfffffe091201d100,
indent=0xffffffff80e18965 "\t ", gp=Variable "gp" is
not available.
)
at /usr/src/sys/geom/part/g_part.c:1993
---Type <return> to continue, or q <return> to quit---
#12 0xffffffff80854359 in g_conf_specific (sb=0xfffffe091201d100, mp=0x0,
gp=0x0, pp=0x0, cp=0x0) at /usr/src/sys/geom/geom_dump.c:216
#13 0xffffffff80854e4c in g_run_events () at /usr/src/sys/geom/geom_event.c:259
#14 0xffffffff808bb9ef in fork_exit (
callout=0xffffffff80856380 <g_event_procbody>, arg=0x0,
frame=0xffffff80003b1c40) at /usr/src/sys/kern/kern_fork.c:992
#15 0xffffffff80bc368e in fork_trampoline ()
at /usr/src/sys/amd64/amd64/exception.S:602
#16 0x0000000000000000 in ?? ()
#17 0x0000000000000000 in ?? ()
#18 0x0000000000000001 in ?? ()
#19 0x0000000000000000 in ?? ()
#20 0x0000000000000000 in ?? ()
#21 0x0000000000000000 in ?? ()
#22 0x0000000000000000 in ?? ()
#23 0x0000000000000000 in ?? ()
#24 0x0000000000000000 in ?? ()
#25 0x0000000000000000 in ?? ()
#26 0x0000000000000000 in ?? ()
#27 0x0000000000000000 in ?? ()
#28 0x0000000000000000 in ?? ()
#29 0x0000000000000000 in ?? ()
#30 0x0000000000000000 in ?? ()
---Type <return> to continue, or q <return> to quit---
#31 0x0000000000000000 in ?? ()
#32 0x0000000000000000 in ?? ()
#33 0x0000000000000000 in ?? ()
#34 0x0000000000000000 in ?? ()
#35 0x0000000000000000 in ?? ()
#36 0x0000000000000000 in ?? ()
#37 0x0000000000000000 in ?? ()
#38 0x0000000000000000 in ?? ()
#39 0x0000000000000000 in ?? ()
#40 0x000000000000000a in ?? ()
#41 0xffffffff81248000 in tdq_cpu ()
#42 0xfffffe00139c38e0 in ?? ()
#43 0x0000000000000000 in ?? ()
#44 0xffffff80003b1600 in ?? ()
#45 0xffffff80003b15a8 in ?? ()
#46 0xfffffe00139a38e0 in ?? ()
#47 0xffffffff8091352e in sched_switch (td=0x0, newtd=0x0, flags=Variable
"flags" is not available.
)
at /usr/src/sys/kern/sched_ule.c:1921
Previous frame inner to this frame (corrupt stack?)
(kgdb)
(kgdb)
== /var/log/messages =
Dec 10 03:23:27 freenas kernel: g_dev_taste: make_dev_p() failed
(gp->name=zvol/
vmpool/XXXXXXXXXXXXXXXX at hourly-2013-12-09_12.09.03--1ds2, error=63)
Dec 10 03:23:27 freenas kernel: WARNING: /var/crash: GJOURNAL flag on fs but no
gjournal provider below
Dec 10 03:23:27 freenas kernel: ix3: Could not setup receive structures
Dec 10 03:23:27 freenas last message repeated 3 times
Dec 10 03:23:27 freenas savecore: reboot after panic: kmem_malloc(2887680): kmem
_map too small: 28092231680 total allocated
Dec 10 03:23:27 freenas savecore: writing core to vmcore.0
Dec 10 03:26:29 freenas kernel: pid 954 (fstat), uid 0, was killed: out of swap
space
= Information about the system right before the crash
== top -b =
last pid: 87181; load averages: 0.28, 0.30, 0.16 up 4+23:32:13 03:09:00
55 processes: 1 running, 54 sleeping
Mem: 13M Active, 595M Inact, 41G Wired, 9852K Cache, 4922M Buf, 4780M Free
Swap: 50G Total, 50G Free
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
1099 root 4 20 0 136M 6704K select 3 58:20 8.15%
bacula-fd
87164 root 1 22 0 14008K 7844K zio->i 1 0:04 6.05% find
1009 root 32 20 0 9912K 1000K rpcsvc 14 36:28 0.00% nfsd
1164 root 1 20 0 88868K 10748K select 7 22:22 0.00% bsnmpd
1071 root 20 20 0 138M 36704K uwait 8 15:14 0.00% istgt
1023 root 1 20 0 14132K 500K rpcsvc 7 1:27 0.00%
rpc.lockd
1058 root 1 20 0 51384K 1844K select 13 0:14 0.00% ntpd
1085 root 1 20 0 55876K 2472K select 6 0:07 0.00% nmbd
1109 root 1 20 0 47044K 2624K select 11 0:05 0.00% sendmail
1089 root 1 20 0 80728K 3692K select 14 0:02 0.00% winbindd
1087 root 1 20 0 83032K 3456K select 12 0:02 0.00% smbd
22262 root 1 20 0 91636K 7532K select 7 0:01 0.00% smbd
990 root 1 20 0 47040K 1696K select 10 0:01 0.00% rpcbind
1116 root 1 20 0 47036K 1752K nanslp 2 0:01 0.00% cron
1141 root 1 20 0 55824K 2484K select 0 0:01 0.00% sshd
942 root 1 20 0 44960K 1472K select 15 0:01 0.00% syslogd
1094 root 1 20 0 80652K 3732K select 0 0:01 0.00% winbindd
1242 root 1 20 0 80652K 3636K select 3 0:01 0.00% winbindd
== vmstat -m ==
Type InUse MemUse HighUse Requests Size(s)
CAM dev queue 9 2K - 9 128
md_sii_data 0 0K - 4953 512
CAM XPT 523 606K - 5079 16,32,64,128,256,1024,2048
SCSI SES 1 2K - 1 2048
mps 35 459K - 125 16,32,64,128,256,512,1024,4096
isadev 7 1K - 7 128
UART 6 4K - 6 16,512,1024
acpiintr 1 1K - 1 64
cdev 8 2K - 8 256
acpica 6649 733K - 794997 16,32,64,128,256,512,1024,2048,4096
sigio 1 1K - 1 64
filedesc 89 51K - 88905 16,32,64,128,256,512,1024,2048,4096
kenv 82 11K - 1702 16,32,64,128
kqueue 14 16K - 12776 256,2048
proc-args 45 3K - 144142 16,32,64,128,256
hhook 2 1K - 2 128
acpitask 1 2K - 1 2048
ithread 332 53K - 332 32,128,256
CAM queue 70 44K - 6035 16,32,64,128,256,512,1024,2048,4096
KTRACE 100 13K - 100 128
linker 218 808K - 310 16,32,64,128,256,512,2048,4096
lockf 74 9K - 13384132 64,128
loginclass 3 1K - 2514 64
ip6ndp 22 2K - 27 64,128
temp 103 15K - 13035118 16,32,64,128,256,512,1024,2048,4096
devbuf 22477 81023K - 23254 16,32,64,128,256,512,1024,2048,4096
module 488 61K - 488 128
mtx_pool 2 16K - 2
USBdev 42 9K - 42 64,128,512,4096
osd 35 1K - 36 16,64
USB 66 36K - 68 16,32,64,128,256,2048,4096
pmchooks 1 1K - 1 128
subproc 954 743K - 88061 512,4096
proc 2 16K - 2
session 32 4K - 4047 128
pgrp 32 4K - 4190 128
cred 120 19K - 39895700 64,256
uidinfo 4 3K - 571713 128,2048
plimit 17 5K - 32899 256
sysctltmp 0 0K - 6635 16,32,64,128,256,4096
sysctloid 12804 632K - 13098 16,32,64,128
sysctl 0 0K - 727908 16,32,64
tidhash 1 16K - 1
callout 15 7680K - 15
umtx 6240 780K - 6240 128
p1003.1b 1 1K - 1 16
SWAP 2 7649K - 10 64
bus-sc 218 506K - 14625 16,32,64,128,256,512,1024,2048,4096
bus 1674 162K - 29317 16,32,64,128,256,512,1024,2048
devstat 1564 3153K - 1564 32,4096
eventhandler 95 8K - 95 64,128
acpisem 17 3K - 17 128
kobj 336 1344K - 35336 4096
CAM SIM 9 3K - 9 256
Per-cpu 1 1K - 1 32
CAM periph 46 12K - 562 16,32,64,128,256
rman 378 43K - 857 16,32,128
sbuf 1 1K - 384622502
16,32,64,128,256,512,1024,2048,4096
entropy 1024 64K - 1024 64
ctlmem 5062 10113K - 5062 128,2048
stack 0 0K - 716 256
taskqueue 251 23K - 86171 16,32,64,128,1024
Unitno 13 1K - 151273 32,64
iov 0 0K - 11645047 16,64,128,256,512
select 822 103K - 822 128
ioctlops 0 0K - 125364 16,32,64,128,256,512,1024,2048
msg 4 30K - 4 2048,4096
sem 4 106K - 4 2048,4096
shm 1 20K - 1
tty 21 21K - 32 1024,2048
pts 0 0K - 9 256
mbuf_tag 0 0K - 12892 32,128
shmfd 1 8K - 1
pcb 54 158K - 12638 16,32,128,1024,2048,4096
soname 11 2K - 328995 16,32,128
acl 0 0K - 2234889 4096
vfscache 1 8192K - 1
cl_savebuf 0 0K - 141 64
export_host 6 2K - 18 256
vfs_hash 1 4096K - 1
DEVFS1 4571 2286K - 5088 512
vnodes 2 1K - 2 256
DEVFS3 4605 1152K - 4697 256
mount 197 15K - 3908 16,32,64,128,256,512
vnodemarker 0 0K - 386295 512
BPF 15 2K - 15 128
ether_multi 180 10K - 216 16,32,64
ifaddr 133 34K - 133 32,64,128,256,512,4096
ifnet 16 31K - 16 128,2048
clone 6 24K - 8 16,128,4096
arpcom 6 1K - 6 16
lltable 53 21K - 387 256,512
DEVFS 22 1K - 23 16,128
DEVFSP 1 1K - 1 64
routetbl 81 10K - 10162 32,64,128,256,512
igmp 15 4K - 15 256
in_multi 7 2K - 7 256
sctp_iter 0 0K - 9 256
sctp_ifn 7 1K - 7 128
sctp_ifa 15 2K - 15 128
sctp_vrf 1 1K - 1 64
sctp_a_it 0 0K - 9 16
hostcache 1 28K - 1
syncache 1 96K - 1
ip6_moptions 2 1K - 2 32,256
in6_multi 67 10K - 67 32,256
in6_mfilter 1 1K - 1 1024
mld 15 2K - 15 128
NLM 1 2K - 30641 32,256,2048
rpc 98 13K - 48635539 16,32,64,128,256,512,1024,2048
audit_evclass 179 6K - 218 32
freework 1 1K - 1 16
newblk 1 1024K - 1
bmsafemap 1 8K - 1
inodedep 1 4096K - 1
pagedep 1 1024K - 1
ufs_dirhash 180 57K - 1271 16,32,64,128,256,512
ufs_mount 6 25K - 11 512,1024,4096
vm_pgdata 2 129K - 6 128
UMAHash 3 14K - 12 512,1024,2048,4096
memdesc 1 4K - 1 4096
NFSD srvcache 0 0K - 17222411 128
atkbddev 2 1K - 2 64
pfs_nodes 21 6K - 21 256
ctlblk 200 1600K - 200
GEOM 30540 5905K - 227528 16,32,64,128,256,512,1024,2048,4096
ramdisk 1 4096K - 1
acpidev 67 5K - 67 64
ctlpool 532 142K - 532 32,512
kbdmux 7 18K - 7 16,512,1024,2048
LED 28 2K - 28 16,128
apmdev 1 1K - 1 128
madt_table 0 0K - 1 4096
feeder 7 1K - 7 32
scsi_da 0 0K - 840 16
pci_link 16 2K - 16 64,128
acpi_perf 16 4K - 16 256
qpidrv 2 1K - 2 16
raid_data 0 0K - 29718 32,128,256
io_apic 3 6K - 3 2048
MCA 33 5K - 33 128
msi 56 7K - 56 128
nexusdev 4 1K - 4 16
md_nvidia_data 0 0K - 4953 512
ipmi 0 0K - 6 128
solaris 6052422 34083371K - 855712754
16,32,64,128,256,512,1024,2048,4096
kstat_data 4 1K - 4 64
== vmstat -z =
ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP
UMA Kegs: 208, 0, 189, 15, 189, 0, 0
UMA Zones: 2432, 0, 189, 0, 189, 0, 0
UMA Slabs: 568, 0, 1028022, 1205552,67578210, 0, 0
UMA RCntSlabs: 568, 0, 64686, 1835, 263367, 0, 0
UMA Hash: 256, 0, 78, 12, 81, 0, 0
16 Bucket: 152, 0, 52, 173, 242, 0, 0
32 Bucket: 280, 0, 155, 223, 421, 1, 0
64 Bucket: 536, 0, 290, 116, 871, 137, 0
128 Bucket: 1048, 0, 5602, 2, 895393,3679513, 0
VM OBJECT: 232, 0, 296742, 183210, 7303072, 0, 0
MAP: 232, 0, 7, 25, 7, 0, 0
KMAP ENTRY: 120, 1549473, 403200, 230967,164135153, 0, 0
MAP ENTRY: 120, 0, 2746, 4911, 3839047, 0, 0
fakepg: 120, 0, 0, 0, 0, 0, 0
mt_zone: 4112, 0, 331, 42, 331, 0, 0
16: 16, 0, 1607034, 421230,145940609, 0, 0
32: 32, 0, 913848, 91809,148582956, 0, 0
64: 64, 0, 301580, 1848260,634878743, 0, 0
128: 128, 0, 256324, 493819,132936793, 0, 0
256: 256, 0, 114915, 126540,82219080, 0, 0
512: 512, 0, 1932913, 347085,122393860, 0, 0
1024: 1024, 0, 2354, 7230, 6335288, 0, 0
2048: 2048, 0, 8300, 3358,36143800, 0, 0
4096: 4096, 0, 142416, 10073,14056966, 0, 0
Files: 80, 0, 353, 4102,10473433, 0, 0
TURNSTILE: 136, 0, 3121, 359, 3121, 0, 0
umtx pi: 96, 0, 0, 0, 0, 0, 0
MAC labels: 40, 0, 0, 0, 0, 0, 0
PROC: 1184, 0, 75, 801, 87183, 0, 0
THREAD: 1128, 0, 2714, 406, 46141, 0, 0
SLEEPQUEUE: 80, 0, 3121, 446, 3121, 0, 0
VMSPACE: 392, 0, 56, 2124, 87165, 0, 0
cpuset: 72, 0, 1839, 661, 45300, 0, 0
audit_record: 960, 0, 0, 0, 0, 0, 0
mbuf_packet: 256, 0, 18805, 1419,253572421, 0, 0
mbuf: 256, 0, 49286, 3975,1345906286, 0, 0
mbuf_cluster: 2048, 262144, 20224, 2512,19208068, 0, 0
mbuf_jumbo_page: 4096, 131072, 40, 1536,382314578, 0, 0
mbuf_jumbo_9k: 9216, 65536, 49136, 2606,46897363, 4, 0
mbuf_jumbo_16k: 16384, 32768, 0, 0, 0, 0, 0
mbuf_ext_refcnt: 4, 0, 0, 0, 0, 0, 0
g_bio: 232, 0, 8, 7560,334229483, 0, 0
ttyinq: 160, 0, 120, 480, 930, 0, 0
ttyoutq: 256, 0, 64, 356, 489, 0, 0
ata_request: 328, 0, 0, 0, 0, 0, 0
ata_composite: 336, 0, 0, 0, 0, 0, 0
VNODE: 480, 0, 454870, 34098,26814245, 0, 0
VNODEPOLL: 112, 0, 0, 0, 0, 0, 0
NAMEI: 1024, 0, 0, 1040,62691796, 0, 0
S VFS Cache: 108, 0, 417293, 36094,23514596, 0, 0
STS VFS Cache: 148, 0, 0, 0, 0, 0, 0
L VFS Cache: 328, 0, 69668, 26056, 3432753, 0, 0
LTS VFS Cache: 368, 0, 0, 0, 0, 0, 0
DIRHASH: 1024, 0, 169, 1655, 11763, 0, 0
NCLNODE: 568, 0, 0, 0, 0, 0, 0
Mountpoints: 792, 0, 15, 65, 16, 0, 0
pipe: 728, 0, 37, 1868, 61363, 0, 0
AIO: 208, 0, 0, 0, 0, 0, 0
AIOP: 32, 0, 0, 0, 0, 0, 0
AIOCB: 480, 0, 0, 0, 0, 0, 0
AIOL: 128, 0, 0, 0, 0, 0, 0
AIOLIO: 272, 0, 0, 0, 0, 0, 0
ksiginfo: 112, 0, 865, 2798, 140833, 0, 0
itimer: 344, 0, 1, 21, 1, 0, 0
KNOTE: 128, 0, 21, 675, 7613312, 0, 0
socket: 680, 262146, 134, 1126, 61267, 0, 0
unpcb: 240, 262144, 30, 946, 18662, 0, 0
ipq: 56, 8253, 0, 0, 0, 0, 0
udp_inpcb: 392, 262150, 63, 267, 32514, 0, 0
udpcb: 16, 262248, 63, 2121, 32514, 0, 0
tcp_inpcb: 392, 262150, 40, 520, 10074, 0, 0
tcpcb: 976, 262144, 39, 497, 10074, 0, 0
tcptw: 72, 27800, 1, 899, 1735, 0, 0
syncache: 152, 15375, 0, 550, 7380, 0, 0
hostcache: 136, 15372, 3, 557, 90, 0, 0
tcpreass: 40, 16464, 0, 1764, 2394, 0, 0
sackhole: 32, 0, 0, 2727, 92869, 0, 0
sctp_ep: 1376, 262144, 0, 0, 0, 0, 0
sctp_asoc: 2288, 40000, 0, 0, 0, 0, 0
sctp_laddr: 48, 80064, 0, 576, 14, 0, 0
sctp_raddr: 704, 80000, 0, 0, 0, 0, 0
sctp_chunk: 136, 400008, 0, 0, 0, 0, 0
sctp_readq: 104, 400032, 0, 0, 0, 0, 0
sctp_stream_msg_out: 112, 400026, 0, 0, 0, 0, 0
sctp_asconf: 40, 400008, 0, 0, 0, 0, 0
sctp_asconf_ack: 48, 400032, 0, 0, 0, 0, 0
ripcb: 392, 262150, 0, 100, 10, 0, 0
rtentry: 200, 0, 47, 200, 58, 0, 0
selfd: 56, 0, 1333, 2636,105220327, 0, 0
SWAPMETA: 288, 116519, 0, 0, 0, 0, 0
FFS inode: 168, 0, 211949, 2397, 1336134, 0, 0
FFS1 dinode: 128, 0, 0, 0, 0, 0, 0
FFS2 dinode: 256, 0, 211949, 2356, 1336134, 0, 0
taskq_zone: 48, 0, 0, 2880, 125840, 0, 0
zio_cache: 880, 0, 10, 22154,375343426, 0, 0
zio_link_cache: 48, 0, 5, 29731,364012142, 0, 0
zio_buf_512: 512, 0, 0, 0, 0, 0, 0
zio_data_buf_512: 512, 0, 0, 0, 0, 0, 0
zio_buf_1024: 1024, 0, 0, 0, 0, 0, 0
zio_data_buf_1024: 1024, 0, 0, 0, 0, 0, 0
zio_buf_1536: 1536, 0, 0, 0, 0, 0, 0
zio_data_buf_1536: 1536, 0, 0, 0, 0, 0, 0
zio_buf_2048: 2048, 0, 0, 0, 0, 0, 0
zio_data_buf_2048: 2048, 0, 0, 0, 0, 0, 0
zio_buf_2560: 2560, 0, 0, 0, 0, 0, 0
zio_data_buf_2560: 2560, 0, 0, 0, 0, 0, 0
zio_buf_3072: 3072, 0, 0, 0, 0, 0, 0
zio_data_buf_3072: 3072, 0, 0, 0, 0, 0, 0
zio_buf_3584: 3584, 0, 0, 0, 0, 0, 0
zio_data_buf_3584: 3584, 0, 0, 0, 0, 0, 0
zio_buf_4096: 4096, 0, 0, 0, 0, 0, 0
zio_data_buf_4096: 4096, 0, 0, 0, 0, 0, 0
zio_buf_5120: 5120, 0, 0, 0, 0, 0, 0
zio_data_buf_5120: 5120, 0, 0, 0, 0, 0, 0
zio_buf_6144: 6144, 0, 0, 0, 0, 0, 0
zio_data_buf_6144: 6144, 0, 0, 0, 0, 0, 0
zio_buf_7168: 7168, 0, 0, 0, 0, 0, 0
zio_data_buf_7168: 7168, 0, 0, 0, 0, 0, 0
zio_buf_8192: 8192, 0, 0, 0, 0, 0, 0
zio_data_buf_8192: 8192, 0, 0, 0, 0, 0, 0
zio_buf_10240: 10240, 0, 0, 0, 0, 0, 0
zio_data_buf_10240: 10240, 0, 0, 0, 0, 0, 0
zio_buf_12288: 12288, 0, 0, 0, 0, 0, 0
zio_data_buf_12288: 12288, 0, 0, 0, 0, 0, 0
zio_buf_14336: 14336, 0, 0, 0, 0, 0, 0
zio_data_buf_14336: 14336, 0, 0, 0, 0, 0, 0
zio_buf_16384: 16384, 0, 0, 0, 0, 0, 0
zio_data_buf_16384: 16384, 0, 0, 0, 0, 0, 0
zio_buf_20480: 20480, 0, 0, 0, 0, 0, 0
zio_data_buf_20480: 20480, 0, 0, 0, 0, 0, 0
zio_buf_24576: 24576, 0, 0, 0, 0, 0, 0
zio_data_buf_24576: 24576, 0, 0, 0, 0, 0, 0
zio_buf_28672: 28672, 0, 0, 0, 0, 0, 0
zio_data_buf_28672: 28672, 0, 0, 0, 0, 0, 0
zio_buf_32768: 32768, 0, 0, 0, 0, 0, 0
zio_data_buf_32768: 32768, 0, 0, 0, 0, 0, 0
zio_buf_36864: 36864, 0, 0, 0, 0, 0, 0
zio_data_buf_36864: 36864, 0, 0, 0, 0, 0, 0
zio_buf_40960: 40960, 0, 0, 0, 0, 0, 0
zio_data_buf_40960: 40960, 0, 0, 0, 0, 0, 0
zio_buf_45056: 45056, 0, 0, 0, 0, 0, 0
zio_data_buf_45056: 45056, 0, 0, 0, 0, 0, 0
zio_buf_49152: 49152, 0, 0, 0, 0, 0, 0
zio_data_buf_49152: 49152, 0, 0, 0, 0, 0, 0
zio_buf_53248: 53248, 0, 0, 0, 0, 0, 0
zio_data_buf_53248: 53248, 0, 0, 0, 0, 0, 0
zio_buf_57344: 57344, 0, 0, 0, 0, 0, 0
zio_data_buf_57344: 57344, 0, 0, 0, 0, 0, 0
zio_buf_61440: 61440, 0, 0, 0, 0, 0, 0
zio_data_buf_61440: 61440, 0, 0, 0, 0, 0, 0
zio_buf_65536: 65536, 0, 0, 0, 0, 0, 0
zio_data_buf_65536: 65536, 0, 0, 0, 0, 0, 0
zio_buf_69632: 69632, 0, 0, 0, 0, 0, 0
zio_data_buf_69632: 69632, 0, 0, 0, 0, 0, 0
zio_buf_73728: 73728, 0, 0, 0, 0, 0, 0
zio_data_buf_73728: 73728, 0, 0, 0, 0, 0, 0
zio_buf_77824: 77824, 0, 0, 0, 0, 0, 0
zio_data_buf_77824: 77824, 0, 0, 0, 0, 0, 0
zio_buf_81920: 81920, 0, 0, 0, 0, 0, 0
zio_data_buf_81920: 81920, 0, 0, 0, 0, 0, 0
zio_buf_86016: 86016, 0, 0, 0, 0, 0, 0
zio_data_buf_86016: 86016, 0, 0, 0, 0, 0, 0
zio_buf_90112: 90112, 0, 0, 0, 0, 0, 0
zio_data_buf_90112: 90112, 0, 0, 0, 0, 0, 0
zio_buf_94208: 94208, 0, 0, 0, 0, 0, 0
zio_data_buf_94208: 94208, 0, 0, 0, 0, 0, 0
zio_buf_98304: 98304, 0, 0, 0, 0, 0, 0
zio_data_buf_98304: 98304, 0, 0, 0, 0, 0, 0
zio_buf_102400: 102400, 0, 0, 0, 0, 0, 0
zio_data_buf_102400: 102400, 0, 0, 0, 0, 0, 0
zio_buf_106496: 106496, 0, 0, 0, 0, 0, 0
zio_data_buf_106496: 106496, 0, 0, 0, 0, 0, 0
zio_buf_110592: 110592, 0, 0, 0, 0, 0, 0
zio_data_buf_110592: 110592, 0, 0, 0, 0, 0, 0
zio_buf_114688: 114688, 0, 0, 0, 0, 0, 0
zio_data_buf_114688: 114688, 0, 0, 0, 0, 0, 0
zio_buf_118784: 118784, 0, 0, 0, 0, 0, 0
zio_data_buf_118784: 118784, 0, 0, 0, 0, 0, 0
zio_buf_122880: 122880, 0, 0, 0, 0, 0, 0
zio_data_buf_122880: 122880, 0, 0, 0, 0, 0, 0
zio_buf_126976: 126976, 0, 0, 0, 0, 0, 0
zio_data_buf_126976: 126976, 0, 0, 0, 0, 0, 0
zio_buf_131072: 131072, 0, 0, 0, 0, 0, 0
zio_data_buf_131072: 131072, 0, 0, 0, 0, 0, 0
sa_cache: 80, 0, 242861, 39649,25477393, 0, 0
dnode_t: 856, 0, 1961556, 68320, 8022408, 0, 0
dmu_buf_impl_t: 224, 0, 2407116, 802807,36856756, 0, 0
arc_buf_hdr_t: 216, 0, 3108123, 1528605,58028612, 0, 0
arc_buf_t: 104, 0, 894075, 1915545,64612624, 0, 0
zil_lwb_cache: 192, 0, 3, 2657, 49153, 0, 0
zfs_znode_cache: 400, 0, 242861, 39640,25477393, 0, 0
== /usr/local/bin/zfs-stats -a =
------------------------------------------------------------------------
ZFS Subsystem Report Tue Dec 10 03:09:00 2013
------------------------------------------------------------------------
System Information:
Kernel Version: 901000 (osreldate)
Hardware Platform: amd64
Processor Architecture: amd64
ZFS Storage pool Version: 28
ZFS Filesystem Version: 5
FreeBSD 9.1-RELEASE #0 r243825: Tue Dec 4 09:23:10 UTC 2012 root
3:09AM up 4 days, 23:32, 0 users, load averages: 0.28, 0.30, 0.16
------------------------------------------------------------------------
System Memory:
0.03% 12.86 MiB Active, 1.25% 594.87 MiB Inact
88.67% 41.26 GiB Wired, 0.02% 9.62 MiB Cache
10.03% 4.67 GiB Free, 0.00% 352.00 KiB Gap
Real Installed: 48.00 GiB
Real Available: 99.94% 47.97 GiB
Real Managed: 97.00% 46.53 GiB
Logical Total: 48.00 GiB
Logical Used: 89.05% 42.74 GiB
Logical Free: 10.95% 5.26 GiB
Kernel Memory: 32.67 GiB
Data: 99.94% 32.65 GiB
Text: 0.06% 20.53 MiB
Kernel Memory Map: 30.99 GiB
Size: 100.00% 30.99 GiB
Free: 0.00% 568.00 KiB
------------------------------------------------------------------------
ARC Summary: (HEALTHY)
Memory Throttle Count: 0
ARC Misc:
Deleted: 53.77m
Recycle Misses: 5.11m
Mutex Misses: 554.83k
Evict Skips: 6.12m
ARC Size: 75.01% 34.15 GiB
Target Size: (Adaptive) 75.01% 34.16 GiB
Min Size (Hard Limit): 12.50% 5.69 GiB
Max Size (High Water): 8:1 45.53 GiB
ARC Size Breakdown:
Recently Used Cache Size: 61.07% 20.86 GiB
Frequently Used Cache Size: 38.93% 13.30 GiB
ARC Hash Breakdown:
Elements Max: 5.29m
Elements Current: 58.78% 3.11m
Collisions: 63.24m
Chain Max: 21
Chains: 739.52k
------------------------------------------------------------------------
ARC Efficiency: 238.62m
Cache Hit Ratio: 74.10% 176.82m
Cache Miss Ratio: 25.90% 61.80m
Actual Hit Ratio: 62.27% 148.58m
Data Demand Efficiency: 93.96% 76.52m
Data Prefetch Efficiency: 4.52% 54.66m
CACHE HITS BY CACHE LIST:
Anonymously Used: 12.32% 21.78m
Most Recently Used: 39.13% 69.19m
Most Frequently Used: 44.90% 79.39m
Most Recently Used Ghost: 0.37% 658.56k
Most Frequently Used Ghost: 3.28% 5.80m
CACHE HITS BY DATA TYPE:
Demand Data: 40.66% 71.90m
Prefetch Data: 1.40% 2.47m
Demand Metadata: 43.35% 76.65m
Prefetch Metadata: 14.59% 25.80m
CACHE MISSES BY DATA TYPE:
Demand Data: 7.47% 4.62m
Prefetch Data: 84.45% 52.19m
Demand Metadata: 6.68% 4.13m
Prefetch Metadata: 1.39% 861.23k
------------------------------------------------------------------------
L2 ARC Summary: (HEALTHY)
Passed Headroom: 19.14m
Tried Lock Failures: 3.41m
IO In Progress: 17
Low Memory Aborts: 209
Free on Write: 33.24k
Writes While Full: 39.98k
R/W Clashes: 193
Bad Checksums: 0
IO Errors: 0
SPA Mismatch: 15.80b
L2 ARC Size: (Adaptive) 30.67 GiB
Header Size: 0.17% 52.26 MiB
L2 ARC Evicts:
Lock Retries: 126
Upon Reading: 0
L2 ARC Breakdown: 61.80m
Hit Ratio: 2.96% 1.83m
Miss Ratio: 97.04% 59.97m
Feeds: 466.09k
L2 ARC Buffer:
Bytes Scanned: 321.97 TiB
Buffer Iterations: 466.09k
List Iterations: 29.11m
NULL List Iterations: 1.49m
L2 ARC Writes:
Writes Sent: 100.00% 92.12k
------------------------------------------------------------------------
File-Level Prefetch: (HEALTHY)
DMU Efficiency: 221.74m
Hit Ratio: 57.51% 127.53m
Miss Ratio: 42.49% 94.21m
Colinear: 94.21m
Hit Ratio: 0.01% 11.87k
Miss Ratio: 99.99% 94.19m
Stride: 111.20m
Hit Ratio: 99.92% 111.11m
Miss Ratio: 0.08% 90.41k
DMU Misc:
Reclaim: 94.19m
Successes: 0.64% 606.42k
Failures: 99.36% 93.59m
Streams: 16.43m
+Resets: 0.03% 4.43k
-Resets: 99.97% 16.43m
Bogus: 0
------------------------------------------------------------------------
VDEV cache is disabled
------------------------------------------------------------------------
ZFS Tunables (sysctl):
kern.maxusers 384
vm.kmem_size 49964781568
vm.kmem_size_scale 1
vm.kmem_size_min 0
vm.kmem_size_max 329853485875
vfs.zfs.l2c_only_size 24372193280
vfs.zfs.mfu_ghost_data_lsize 14966604288
vfs.zfs.mfu_ghost_metadata_lsize 2825756672
vfs.zfs.mfu_ghost_size 17792360960
vfs.zfs.mfu_data_lsize 15835021312
vfs.zfs.mfu_metadata_lsize 23130112
vfs.zfs.mfu_size 16063756288
vfs.zfs.mru_ghost_data_lsize 11579461632
vfs.zfs.mru_ghost_metadata_lsize 7082113536
vfs.zfs.mru_ghost_size 18661575168
vfs.zfs.mru_data_lsize 7865216000
vfs.zfs.mru_metadata_lsize 8403897344
vfs.zfs.mru_size 17009602560
vfs.zfs.anon_data_lsize 0
vfs.zfs.anon_metadata_lsize 0
vfs.zfs.anon_size 8093696
vfs.zfs.l2arc_norw 1
vfs.zfs.l2arc_feed_again 1
vfs.zfs.l2arc_noprefetch 1
vfs.zfs.l2arc_feed_min_ms 200
vfs.zfs.l2arc_feed_secs 1
vfs.zfs.l2arc_headroom 2
vfs.zfs.l2arc_write_boost 8388608
vfs.zfs.l2arc_write_max 8388608
vfs.zfs.arc_meta_limit 12222759936
vfs.zfs.arc_meta_used 12973058504
vfs.zfs.arc_min 6111379968
vfs.zfs.arc_max 48891039744
vfs.zfs.dedup.prefetch 1
vfs.zfs.mdcomp_disable 0
vfs.zfs.write_limit_override 0
vfs.zfs.write_limit_inflated 154529181696
vfs.zfs.write_limit_max 6438715904
vfs.zfs.write_limit_min 33554432
vfs.zfs.write_limit_shift 3
vfs.zfs.no_write_throttle 0
vfs.zfs.zfetch.array_rd_sz 1048576
vfs.zfs.zfetch.block_cap 256
vfs.zfs.zfetch.min_sec_reap 2
vfs.zfs.zfetch.max_streams 8
vfs.zfs.prefetch_disable 0
vfs.zfs.mg_alloc_failures 24
vfs.zfs.check_hostid 1
vfs.zfs.recover 0
vfs.zfs.txg.synctime_ms 1000
vfs.zfs.txg.timeout 5
vfs.zfs.vdev.cache.bshift 16
vfs.zfs.vdev.cache.size 0
vfs.zfs.vdev.cache.max 16384
vfs.zfs.vdev.write_gap_limit 4096
vfs.zfs.vdev.read_gap_limit 32768
vfs.zfs.vdev.aggregation_limit 131072
vfs.zfs.vdev.ramp_rate 2
vfs.zfs.vdev.time_shift 6
vfs.zfs.vdev.min_pending 4
vfs.zfs.vdev.max_pending 10
vfs.zfs.vdev.bio_flush_disable 0
vfs.zfs.cache_flush_disable 0
vfs.zfs.zil_replay_disable 0
vfs.zfs.zio.use_uma 0
vfs.zfs.snapshot_list_prefetch 0
vfs.zfs.version.zpl 5
vfs.zfs.version.spa 28
vfs.zfs.version.acl 1
vfs.zfs.debug 0
vfs.zfs.super_owner 0
------------------------------------------------------------------------
Dipl.-Inform. (FH)
Dennis Berger
email: db at bsdsystems.de
mobile: +491791231509
fon: +494054001817
GPG Fingerprint: 313E A4AD 72CB 2617 1BA9 7410 F954 07C0 DC3E 9578