freebsd stable - Jul 2015 - Interesting Boot failure on HEAD with a large number of IGB nics

On Wed, Jul 22, 2015 at 12:11:09PM -0400, Mark Saad
wrote:
> All
>  I am wondering if anyone has run into this issue before , and if there is
> a fix.
> I have a Scalable Informatics siRouter with 24 Intel I350 igb nics and 8
> intel 82599ES ixgbe/ix nic . The SiRouter is a Supermicro X9DRX+-F with a
> bunch of intel nics.
> 
> On Head I run into a boot panic out of the box while plumbing the usb
> controller . The box crashes with
> 
> ehci0: <Intel Patsburg USB 2.0 controller> mem 0xdd923000-0xdd9233ff
irq 16
> at device 26.0 on pci0
> panic: Couldn't find an APIC vector for IRQ 16
> 
> However it has only probed 8 of the 24 igb when is crashes. Here is the
> complete boot up with some additional info.
> 
> http://pastebin.com/MfbEwwg4
> 
> Now the interesting thing.
> 
>  DragonFly 4.2x boots fine  see the dmesg here
> http://dmesgd.nycbug.org/index.cgi?action=dmesgd&do=view&id=2778
> 
> OpenBSD boots and finds the nics as well see dmesg here
> http://dmesgd.nycbug.org/index.cgi?action=dmesgd&do=view&id=2777
> 
> NetBSD Pukes while probing something out of the box and I cant get the
> serial console to work.
> 
> Linux 2.6 CentOS 6.6 works
> 
> OmniOS / Illumos-gate and Illumos-joyent work out of the box
> 
> 
> 
> So my goal here is to make the box boot FreeBSD . It had been used prior
> running OmniOS and CentOS and well now I want to make it work. :)
> 
> Any ideas here ?

I'm no expert, but you may want to try setting

hw.igb.num_queues=1

and maybe

hw.ixgbe.num_queues=1

in the boot loader and trying that.

Regards,

Gary

P.S. freebsd-net@ may be a better list than either -stable or -hackers.

> On Jul 22, 2015, at 10:14, Gary Palmer <gpalmer at freebsd.org>
wrote:
...
> I'm no expert, but you may want to try setting
> 
> hw.igb.num_queues=1
> 
> and maybe
> 
> hw.ixgbe.num_queues=1
> 
> in the boot loader and trying that.
There was another discussion that took place around June on current that might
be helpful. I think the solution boiled down to what Gary described above,
because the driver auto tuning was broken...
Thanks!
-NGie