freebsd stable - Jun 2017 - Can't boot (zfs-on-root) with "options EARLY_AP_STARTUP" and "device pf"

Good afternoon,

I tried to update the kernel on my Lenovo x230 (stable/11), and I would get
crashes on boot. *I am running a non-GENERIC kernel.* Bisecting the changes
from last-known-good, I found that I couldn't boot on any kernel past after
r318752. The next change to the kernel, r318763 [1], turned on
EARLY_AP_STARTUP by default.

I've narrowed it down to what appears to be a conflict with 'options
EARLY_AP_STARTUP' and 'device pf'. I can build a farily vanilla
kernel with
these set, and it crashes on boot. (I have a few other options set to
enable root-on-zfs-on-geli.)

Any suggestions, other than 'don't do that?',
 - Eric

[1] https://svnweb.freebsd.org/base?view=revision&revision=318763

FWIW, it appears to be crashing for an integer divide fault in
pf_purge_thread in the 'pf_purge' process. If i turn on INVARIAINTS and
INVARIANT_SUPPORT, it boots. Perhaps a race?

 - Eric

On Fri, Jun 9, 2017 at 5:04 PM, Eric A. Borisch <eborisch at gmail.com>
wrote:
> Good afternoon,
>
> I tried to update the kernel on my Lenovo x230 (stable/11), and I would
> get crashes on boot. *I am running a non-GENERIC kernel.* Bisecting the
> changes from last-known-good, I found that I couldn't boot on any
kernel
> past after r318752. The next change to the kernel, r318763 [1], turned on
> EARLY_AP_STARTUP by default.
>
> I've narrowed it down to what appears to be a conflict with
'options
> EARLY_AP_STARTUP' and 'device pf'. I can build a farily vanilla
kernel with
> these set, and it crashes on boot. (I have a few other options set to
> enable root-on-zfs-on-geli.)
>
> Any suggestions, other than 'don't do that?',
>  - Eric
>
> [1] https://svnweb.freebsd.org/base?view=revision&revision=318763
>