On 07/08/2019 07:15, Greg Rivers wrote:> These Intel NICs have been performing very well for me:
>
> ix0 at pci0:7:0:0: class=0x020000 card=0x17d3103c chip=0x10fb8086 rev=0x01
hdr=0x00
> vendor = 'Intel Corporation'
> device = '82599ES 10-Gigabit SFI/SFP+ Network Connection'
> class = network
> subclass = ethernet
>
> On 11.2-RELEASE, the cards probe like this:
>
> ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version -
3.2.12-k> port 0x7000-0x701f mem 0xf7f00000-0xf7ffffff,0xf7ef0000-0xf7ef3fff
irq 28 at device 0.0 numa-domain 0 on pci1
> ix0: Using MSI-X interrupts with 9 vectors
> ix0: Ethernet address: 8c:dc:d4:af:8a:e0
> ix0: PCI Express Bus: Speed 5.0GT/s Width x8
> ix0: netmap queues/slots: TX 8/2048, RX 8/2048
>
> As expected, all 8 queues are enabled on the NIC, with 8 corresponding
interrupt threads pinned to 8 different CPUs
>
> But on 12.0-RELEASE, the cards probe like this:
>
> ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver> port
0x4020-0x403f mem 0xe2900000-0xe29fffff,0xe2a04000-0xe2a07fff at device 0.0
numa-domain 0 on pci4
> ix0: using 2048 tx descriptors and 2048 rx descriptors
> ix0: msix_init qsets capped at 16
> ix0: Unable to map MSIX table
> ix0: Using an MSI interrupt
> ix0: allocated for 1 queues
> ix0: allocated for 1 rx queues
> ix0: Ethernet address: 48:df:37:62:be:38
> ix0: PCI Express Bus: Speed 5.0GT/s Width x8
> ix0: netmap queues/slots: TX 1/2048, RX 1/2048
> ix0: link state changed to UP
>
> MSI-X is disabled, and only one queue is enabled. I'm sure this is
going to perform poorly under high network loads. Does anyone know why this is?
Don't know if it could help somehow, but I've got two identical boxes
with similar NICs side by side, one running 11.2 and the other one 12.0
and I don't have such problems.
If someone points me where to look, I could help collecting more data to
know how my system is different from Greg's.
$ freebsd-version
12.0-RELEASE-p7
$ pciconf -lvc ix0
ix0 at pci0:17:0:0: class=0x020000 card=0x7a128086 chip=0x10fb8086
rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = '82599ES 10-Gigabit SFI/SFP+ Network Connection'
class = network
subclass = ethernet
cap 01[40] = powerspec 3 supports D0 D3 current D0
cap 05[50] = MSI supports 1 message, 64 bit, vector masks
cap 11[70] = MSI-X supports 64 messages, enabled
Table in map 0x20[0x0], PBA in map 0x20[0x2000]
cap 10[a0] = PCI-Express 2 endpoint max data 256(512) FLR RO NS
link x8(x8) speed 5.0(5.0) ASPM disabled(L0s)
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
ecap 0003[140] = Serial 1 90e2baffff809b08
ecap 000e[150] = ARI 1
ecap 0010[160] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI
disabled
0 VFs configured out of 64 supported
First VF RID Offset 0x0180, VF RID Stride 0x0002
VF Device ID 0x10ed
Page Sizes: 4096 (enabled), 8192, 65536, 262144,
1048576, 4194304
$ dmesg
ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver> port 0x2fc0-0x2fdf
mem 0x91200000-0x912fffff,0x910fc000-0x910fffff irq 32 at device 0.0 on pci2
ix0: using 2048 tx descriptors and 2048 rx descriptors
ix0: msix_init qsets capped at 16
ix0: pxm cpus: 8 queue msgs: 63 admincnt: 1
ix0: using 8 rx queues 8 tx queues
ix0: Using MSIX interrupts with 9 vectors
ix0: allocated for 8 queues
ix0: allocated for 8 rx queues
ix0: Ethernet address: 90:e2:ba:80:9b:08
ix0: PCI Express Bus: Speed 5.0GT/s Width x8
ix0: netmap queues/slots: TX 8/2048, RX 8/2048
=========$ freebsd-version
11.2-RELEASE-p3
$ dmesg
ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.2.12-k>
port 0x2fc0-0x2fdf mem 0x91200000-0x912fffff,0x910fc000-0x910fffff irq
32 at device 0.0 numa-domain 0 on pci2
ix0: Using MSI-X interrupts with 9 vectors
ix0: Ethernet address: 90:e2:ba:80:99:d0
ix0: PCI Express Bus: Speed 5.0GT/s Width x8
ix0: netmap queues/slots: TX 8/2048, RX 8/2048