freebsd stable - Dec 2012 - FreeBSD 9.1-RELEASE crashes almost daily; backtraces always list zfs routines

Please help, I reported this issue on
http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/174372 but the crashes
are unbearable since they happen regularly at night, most of the time
when periodic.daily is called (3am) but there are exceptions. It seems
like it can be triggered by any heavy disk activity. In many of the
crash dumps the current process is find command, but of course that's
does not always is the cause.

When calling scrub, it appears to pass successfully. Smartmon tools is
not reporting any disk errors. I tested memory using memtest86 about a
month ago and it passed tests successfully.

I never had this type of issue on 9.0, and not much changed in my
kernel config besides installing WiFi card.

System:
FreeBSD chinatsu.takeda.tk 9.1-RELEASE FreeBSD 9.1-RELEASE #2 r244482: Wed Dec
19 23:28:15 PST 2012     root at
chinatsu.takeda.tk:/usr/obj/usr/src/sys/CHINATSU  amd64
It's compiled from releng/9.1 branch.

Thank you for any help,
Derek


Example crash messages:

Today:
panic: general protection fault
cpuid = 1
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
kdb_backtrace() at kdb_backtrace+0x37
panic() at panic+0x1cd
trap_fatal() at trap_fatal+0x285
trap() at trap+0x23e
calltrap() at calltrap+0x8
--- trap 0x9, rip = 0xffffffff8101f5ac, rsp = 0xffffff8230ac18c0, rbp =
0xffffff8230ac18d0 ---
list_prev() at list_prev+0xc
arc_evict() at arc_evict+0x194
arc_adjust() at arc_adjust+0x1a1
arc_reclaim_thread() at arc_reclaim_thread+0x1a6
fork_exit() at fork_exit+0x11c
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8230ac1bb0, rbp = 0 ---
Uptime: 20h44m51s
Dumping 1157 out of 8072 MB:..2%..12%..21%..31%..41%..52%..61%..71%..81%..92%


Yesterday:
Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0xffffffffffffffe8
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff8102d1e7
stack pointer           = 0x28:0xffffff82315ec190
frame pointer           = 0x28:0xffffff82315ec280
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 34744 (find)
trap number             = 12
panic: page fault
cpuid = 0
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
kdb_backtrace() at kdb_backtrace+0x37
panic() at panic+0x1cd
trap_fatal() at trap_fatal+0x285
trap_pfault() at trap_pfault+0x216
trap() at trap+0x363
calltrap() at calltrap+0x8
--- trap 0xc, rip = 0xffffffff8102d1e7, rsp = 0xffffff82315ec190, rbp =
0xffffff82315ec280 ---
arc_evict() at arc_evict+0x1e7
arc_get_data_buf() at arc_get_data_buf+0x1d5
arc_read_nolock() at arc_read_nolock+0x1ec
arc_read() at arc_read+0x93
dbuf_read() at dbuf_read+0x452
dmu_buf_hold() at dmu_buf_hold+0xe0
zap_lockdir() at zap_lockdir+0x58
zap_cursor_retrieve() at zap_cursor_retrieve+0x19b
zfs_freebsd_readdir() at zfs_freebsd_readdir+0x2ee
VOP_READDIR_APV() at VOP_READDIR_APV+0x4a
kern_getdirentries() at kern_getdirentries+0x225
sys_getdirentries() at sys_getdirentries+0x23
amd64_syscall() at amd64_syscall+0x5d8
Xfast_syscall() at Xfast_syscall+0xf7
--- syscall (196, FreeBSD ELF64, sys_getdirentries), rip = 0x80089b29c, rsp =
0x7fffffffd8a8, rbp = 0x1 ---
Uptime: 2d3h49m3s
Dumping 3701 out of 8072 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%

  

-- 
Best regards,
 Derek                          mailto:takeda at takeda.tk

on 24/12/2012 00:23 Derek Kulinski said the following:
> Dumping 3701 out of 8072
MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%
So do you have the crash dump(s)?

-- 
Andriy Gapon