Damian Danielecki
2014-May-17 12:40 UTC
[9.3 PRE] Intel i210AT ethernet adapter MSI-X problems (igb driver)
I am receiving as many as 30.000-40.000 interrupts per second under no
big nfs traffic.
Avg traffic is 20Mb/s, avg interrupts number on igb device is 13.000/s.
IMHO. I could compare this to similar igb adapters on other FreeBSD 9
servers and in this case should be less interrupts (lightweight
traffic conditions).
Additionally all of these interrupts occupies only one IRQ.
The interested thing is CPU usage caused by %interrupts is very small
(under 1%) so this is reason I am still able to use this ethernet
card.
# uname -a
FreeBSD nfsd.xxx.pl 9.3-PRERELEASE FreeBSD 9.3-PRERELEASE #5: Fri May
16 15:41:36 CEST 2014
root at nfsd.xxx.pl:/usr/obj/usr/src/sys/FREEBSD9 amd64
This is custom minimalist kernel.
I see in the sources of igb driver that i210 is generally supported:
/usr/src/sys/dev/e1000 # grep 'I210' * |wc -l
64
I guess mine adapter (onboard quad gigabit I210T) is not correctly
handled in the sources of the driver but system recognizes it as igb.
# dmidecode
(...)
Manufacturer: Supermicro
Product Name: X10SLM+-LN4F
(...)
# dmesg |grep igb0
igb0: <Intel(R) PRO/1000 Network Connection version - 2.3.10> port
0xc000-0xc01f mem 0xf7400000-0xf747ffff,0xf7480000-0xf7483fff irq 18
at device 0.0 on pci4
igb0: Using MSIX interrupts with 5 vectors
igb0: Ethernet address: 0c:c4:7a:01:e3:50
igb0: Bound queue 0 to cpu 0
igb0: Bound queue 1 to cpu 1
igb0: Bound queue 2 to cpu 2
igb0: Bound queue 3 to cpu 3
igb0: link state changed to UP
No device description:
#pciconf -vl
igb0 at pci0:4:0:0: class=0x020000 card=0x153315d9 chip=0x15338086
rev=0x03 hdr=0x00
vendor = 'Intel Corporation'
class = network
subclass = ethernet
Number of interrupts taken by device since system startup:
# vmstat -i
irq269: igb0:que 0 501886805 13364
irq270: igb0:que 1 40477 1
irq271: igb0:que 2 40417 1
irq272: igb0:que 3 7526720 200
irq273: igb0:link 12 0
Sample current number of interrupts with not big NFS traffic:
# systat -vm1
Interrupts
34352 total (!!!)
(...)
29937 igb0:que 0
1 igb0:que 1
1 igb0:que 2
1 igb0:que 3
1 igb0:link
As you can see above there are abnormal high numbers of interrupts and
all of them sits on que0.
sysctls untouched by me:
hw.igb.rx_process_limit: 100
hw.igb.num_queues: 4
hw.igb.header_split: 0
hw.igb.buf_ring_size: 4096
hw.igb.max_interrupt_rate: 8000
hw.igb.enable_msix: 1
hw.igb.enable_aim: 1
hw.igb.txd: 1024
hw.igb.rxd: 1024
For example this is properly working IRQ balancing on I350 Gigabit
ethernet adaptor with FreeBSD 9.2-RELEASE:
otherserver# vmstat -i
irq264: igb0:que 0 14586306636 962
irq265: igb0:que 1 12313136472 812
irq266: igb0:que 2 12400518935 818
irq267: igb0:que 3 12230694398 807
irq268: igb0:que 4 12624900681 833
irq269: igb0:que 5 12311037080 812
irq270: igb0:que 6 31682657476 2090
irq271: igb0:que 7 12203814868 805
irq272: igb0:link 77 0
Regards,
DD
Harald Schmalzbauer
2014-May-18 10:31 UTC
[9.3 PRE] Intel i210AT ethernet adapter MSI-X problems (igb driver)
Bez?glich Damian Danielecki's Nachricht vom 17.05.2014 14:40 (localtime):> I am receiving as many as 30.000-40.000 interrupts per second under no > big nfs traffic. > Avg traffic is 20Mb/s, avg interrupts number on igb device is 13.000/s. > IMHO. I could compare this to similar igb adapters on other FreeBSD 9 > servers and in this case should be less interrupts (lightweight > traffic conditions). > Additionally all of these interrupts occupies only one IRQ. > The interested thing is CPU usage caused by %interrupts is very small > (under 1%) so this is reason I am still able to use this ethernet > card. > > # uname -a > FreeBSD nfsd.xxx.pl 9.3-PRERELEASE FreeBSD 9.3-PRERELEASE #5: Fri May > 16 15:41:36 CEST 2014 > root at nfsd.xxx.pl:/usr/obj/usr/src/sys/FREEBSD9 amd64 > This is custom minimalist kernel. > > > I see in the sources of igb driver that i210 is generally supported: > /usr/src/sys/dev/e1000 # grep 'I210' * |wc -l > 64 > > I guess mine adapter (onboard quad gigabit I210T) is not correctly > handled in the sources of the driver but system recognizes it as igb. > > # dmidecode > (...) > Manufacturer: Supermicro > Product Name: X10SLM+-LN4F > (...) > > # dmesg |grep igb0 > igb0: <Intel(R) PRO/1000 Network Connection version - 2.3.10> port > 0xc000-0xc01f mem 0xf7400000-0xf747ffff,0xf7480000-0xf7483fff irq 18 > at device 0.0 on pci4 > igb0: Using MSIX interrupts with 5 vectors > igb0: Ethernet address: 0c:c4:7a:01:e3:50 > igb0: Bound queue 0 to cpu 0 > igb0: Bound queue 1 to cpu 1 > igb0: Bound queue 2 to cpu 2 > igb0: Bound queue 3 to cpu 3 > igb0: link state changed to UP > > > No device description: > #pciconf -vl > igb0 at pci0:4:0:0: class=0x020000 card=0x153315d9 chip=0x15338086 > rev=0x03 hdr=0x00 > vendor = 'Intel Corporation' > class = network > subclass = ethernet > > > Number of interrupts taken by device since system startup: > # vmstat -i > irq269: igb0:que 0 501886805 13364 > irq270: igb0:que 1 40477 1 > irq271: igb0:que 2 40417 1 > irq272: igb0:que 3 7526720 200 > irq273: igb0:link 12 0 > > Sample current number of interrupts with not big NFS traffic: > # systat -vm1 > Interrupts > 34352 total (!!!) > (...) > 29937 igb0:que 0 > 1 igb0:que 1 > 1 igb0:que 2 > 1 igb0:que 3 > 1 igb0:linkAFAIK i210 doesn't support multiple queues. I had to add 'hw.igb.num_queues=1' to my loader.conf when I did some tests with i210. In my case it was PCIe-passthrough, so probably different story, but in case you might want to see what results you get with limiting queues to 1. -Harry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 196 bytes Desc: OpenPGP digital signature URL: <http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20140518/eb6be249/attachment.sig>