freebsd stable - Nov 2016 - VNET + pf => panic [Was: Panic when tearing down a VNET jail; pf mentioned in stack trace]

On 20 Nov 2016, at 11:10, Kamila Sou?kov? wrote:

Hi,
> I had another panic, this time not related to stopping a jail, but
> also mentioning pf in the stack trace. Therefore the previously
> mentioned correlation was not causation -- this crash must be
> connected with simply combining pf and VNET, not with tearing down the
> VNET interface.
>
> I had two crashes in about one hour. I then put `set skip on vnet0:1`
> in pf.conf, and have not seen a panic since then, even with relatively
> heavy network traffic (continous 60Mbit/s since my last email, and an
> evening of 1Gbit/s).
I might have missed this;  which version of FreeBSD are you running?

/bz

Hi,
> I might have missed this;  which version of FreeBSD are you running?
Sorry, I forgot to mention this. 11.0-RELEASE-p3 (both on the host and
in jails).

Further exposure to crashes revealed that is probably connected to
tearing down the jail after all, but with a few seconds delay -- the
system is stable when I don't tear down jails. The iocage docs
actually mention a problem with this and they have code for "warm
restarts" in place, but I have not been able to find out more about
this issue.

Just wondering, is the -stable mailing list a good place, or should
this be moved somewhere else?

Here are some more stack traces:

Nov 20 17:00:34 oresme kernel: Fatal trap 12: page fault while in kernel mode
Nov 20 17:00:34 oresme kernel: cpuid = 6; apic id = 06
Nov 20 17:00:34 oresme kernel: fault virtual address    = 0x1800
Nov 20 17:00:34 oresme kernel: fault code               = supervisor
write data, page not present
Nov 20 17:00:34 oresme kernel: instruction pointer      0x20:0xffffffff8263eaa1
Nov 20 17:00:34 oresme kernel: stack pointer            0x28:0xfffffe085348b870
Nov 20 17:00:34 oresme kernel: frame pointer            0x28:0xfffffe085348b8e0
Nov 20 17:00:34 oresme kernel: code segment             = base 0x0,
limit 0xfffff, type 0x1b
Nov 20 17:00:34 oresme kernel: = DPL 0, pres 1, long 1, def32 0, gran 1
Nov 20 17:00:34 oresme kernel: processor eflags = interrupt enabled,
resume, IOPL = 0
Nov 20 17:00:34 oresme kernel: current process          = 473 (pf purge)
Nov 20 17:00:34 oresme kernel: trap number              = 12
Nov 20 17:00:34 oresme kernel: panic: page fault
Nov 20 17:00:34 oresme kernel: cpuid = 6
Nov 20 17:00:34 oresme kernel: KDB: stack backtrace:
Nov 20 17:00:34 oresme kernel: #0 0xffffffff80aa8787 at kdb_backtrace+0x67
Nov 20 17:00:34 oresme kernel: #1 0xffffffff80a5d632 at vpanic+0x182
Nov 20 17:00:34 oresme kernel: #2 0xffffffff80a5d4a3 at panic+0x43
Nov 20 17:00:34 oresme kernel: #3 0xffffffff80f3cd51 at trap_fatal+0x351
Nov 20 17:00:34 oresme kernel: #4 0xffffffff80f3cf43 at trap_pfault+0x1e3
Nov 20 17:00:34 oresme kernel: #5 0xffffffff80f3c4ec at trap+0x26c
Nov 20 17:00:34 oresme kernel: #6 0xffffffff80f1f521 at calltrap+0x8
Nov 20 17:00:34 oresme kernel: #7 0xffffffff8263e32d at
pf_purge_expired_states+0x12d
Nov 20 17:00:34 oresme kernel: #8 0xffffffff8263e1bb at pf_purge_thread+0x13b
Nov 20 17:00:34 oresme kernel: #9 0xffffffff80a13e85 at fork_exit+0x85
Nov 20 17:00:34 oresme kernel: #10 0xffffffff80f1fa5e at fork_trampoline+0xe

Nov 20 17:53:34 oresme kernel: Fatal trap 12: page fault while in kernel mode
Nov 20 17:53:34 oresme kernel: cpuid = 1; apic id = 01
Nov 20 17:53:34 oresme kernel: fault virtual address    = 0x400
Nov 20 17:53:34 oresme kernel: fault code               = supervisor
write data, page not present
Nov 20 17:53:34 oresme kernel: instruction pointer      0x20:0xffffffff8263eaa1
Nov 20 17:53:34 oresme kernel: stack pointer            0x28:0xfffffe0853157870
Nov 20 17:53:34 oresme kernel: frame pointer            0x28:0xfffffe08531578e0
Nov 20 17:53:34 oresme kernel: code segment             = base 0x0,
limit 0xfffff, type 0x1b
Nov 20 17:53:34 oresme kernel: = DPL 0, pres 1, long 1, def32 0, gran 1
Nov 20 17:53:34 oresme kernel: processor eflags = interrupt enabled,
resume, IOPL = 0
Nov 20 17:53:34 oresme kernel: current process          = 477 (pf purge)
Nov 20 17:53:34 oresme kernel: trap number              = 12
Nov 20 17:53:34 oresme kernel: panic: page fault
Nov 20 17:53:34 oresme kernel: cpuid = 1
Nov 20 17:53:34 oresme kernel: KDB: stack backtrace:
Nov 20 17:53:34 oresme kernel: #0 0xffffffff80aa8787 at kdb_backtrace+0x67
Nov 20 17:53:34 oresme kernel: #1 0xffffffff80a5d632 at vpanic+0x182
Nov 20 17:53:34 oresme kernel: #2 0xffffffff80a5d4a3 at panic+0x43
Nov 20 17:53:34 oresme kernel: #3 0xffffffff80f3cd51 at trap_fatal+0x351
Nov 20 17:53:34 oresme kernel: #4 0xffffffff80f3cf43 at trap_pfault+0x1e3
Nov 20 17:53:34 oresme kernel: #5 0xffffffff80f3c4ec at trap+0x26c
Nov 20 17:53:34 oresme kernel: #6 0xffffffff80f1f521 at calltrap+0x8
Nov 20 17:53:34 oresme kernel: #7 0xffffffff8263e32d at
pf_purge_expired_states+0x12d
Nov 20 17:53:34 oresme kernel: #8 0xffffffff8263e1bb at pf_purge_thread+0x13b
Nov 20 17:53:34 oresme kernel: #9 0xffffffff80a13e85 at fork_exit+0x85
Nov 20 17:53:34 oresme kernel: #10 0xffffffff80f1fa5e at fork_trampoline+0xe

I will try to reproduce this in a VM and look at it (but I am new to
all things FreeBSD and rather busy with my studies, so I can't promise
anything). Any pointers would be welcome.

Thanks!

Kamila