freebsd stable - Dec 2015 - Periodic jobs triggering panics in 10.1 and 10.2

I have multiple machines that have had occasional panics occur while the daily
and weekly periodic scripts run. The panic is always "Fatal trap 18:
integer divide fault while in kernel mode". I've appended a kgdb trace
below with more detail.

A notable common factor is that all affected systems have zfs-based startup
disks, and 20-40 jails. Each jail has it's own filesystem that was created
by cloning the boot filesystem. I suspect this is a zfs bug that is triggered by
the access patterns in the periodic scripts. There is significant load on the
system when the scheduled processes start, because all jails execute the same
scripts at the same time.

I've been able to alleviate this problem by disabling the security scans
within the jails, but leave it enabled on the root host. If this is not a known
issue in FreeBSD 10.2, I'll file a PR on it.

	- .Dustin Wenz



Logged error:

Dec  5 04:16:47 <kern.crit> svr-033-08 kernel: 
Dec  5 04:16:47 <kern.crit> svr-033-08 kernel: 
Dec  5 04:16:47 <kern.crit> svr-033-08 kernel: Fatal trap 18: integer
divide fault while in kernel mode
Dec  5 04:16:47 <kern.crit> svr-033-08 kernel: cpuid = 19; apic id = 27
Dec  5 04:16:47 <kern.crit> svr-033-08 kernel: instruction pointer	=
0x20:0xffffffff819f54d4
Dec  5 04:16:47 <kern.crit> svr-033-08 kernel: stack pointer	        =
0x28:0xfffffe085fec76f0
Dec  5 04:16:47 <kern.crit> svr-033-08 kernel: frame pointer	        =
0x28:0xfffffe085fec7740
Dec  5 04:23:18 <kern.info> svr-033-08 syslogd: kernel boot file is
/boot/kernel/kernel
Dec  5 04:23:18 <kern.crit> svr-033-08 kernel: code segment		= base 0x0,
limit 0xfffff, type 0x1b
Dec  5 04:23:18 <kern.crit> svr-033-08 kernel: = DPL 0, pres 1, long 1,
def32 0, gran 1
Dec  5 04:23:18 <kern.crit> svr-033-08 kernel: processor eflags	=
interrupt enabled, resume, IOPL = 0
Dec  5 04:23:18 <kern.crit> svr-033-08 kernel: current process		= 20355
(find)
Dec  5 04:23:18 <kern.crit> svr-033-08 kernel: trap number		= 18
Dec  5 04:23:18 <kern.crit> svr-033-08 kernel: panic: integer divide fault
Dec  5 04:23:18 <kern.crit> svr-033-08 kernel: cpuid = 19



kgdb trace:

Unread portion of the kernel message buffer:
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 20355 (find)
trap number		= 18
panic: integer divide fault
cpuid = 19
KDB: stack backtrace:
#0 0xffffffff80986710 at kdb_backtrace+0x60
#1 0xffffffff80949e76 at vpanic+0x126
#2 0xffffffff80949d43 at panic+0x43
#3 0xffffffff80d5d3db at trap_fatal+0x36b
#4 0xffffffff80d5d05c at trap+0x75c
#5 0xffffffff80d42f12 at calltrap+0x8
#6 0xffffffff819f4fc8 at dmu_tx_assign+0xf8
#7 0xffffffff81a7a887 at zfs_inactive+0x157
#8 0xffffffff81a8369d at zfs_freebsd_inactive+0xd
#9 0xffffffff80e85ed7 at VOP_INACTIVE_APV+0xa7
#10 0xffffffff809ed182 at vinactive+0x102
#11 0xffffffff809ed572 at vputx+0x272
#12 0xffffffff809f40ea at sys_fchdir+0x2aa
#13 0xffffffff80d5dcf7 at amd64_syscall+0x357
#14 0xffffffff80d431fb at Xfast_syscall+0xfb

On Tue, 2015-12-08 at 16:31 -0600, Dustin Wenz wrote:
> I suspect this is a zfs bug that is triggered by the access patterns
> in the periodic scripts. There is significant load on the system when
> the scheduled processes start, because all jails execute the same
> scripts at the same time.
> 
> I've been able to alleviate this problem by disabling the security
> scans within the jails, but leave it enabled on the root host.
To avoid the problem of jails all starting things at the same time, use
the cron(8) flags -j and -J to set a 'jitter' which will cause cron to
sleep for a random period of specified duration (60 sec max). Cron
flags can be set using the rc.conf variable 'cron_flags'.