Hello list,
I have a problem with interrupts, network cards, and PF performance.
We have 2 firewalls running FreeBSD 8.0 for the current master and
FreeBSD 8.1 for the backup host, which I upgraded just yesterday.
The servers use CARP for redundancy.
These are rather busy boxes which run PF and nginx as a reverse proxy.
As you will see below, we're getting a "high" %interrupt CPU
usage,
which seems to come mostly from the NICs.
I'm wondering if there is any way to optimize the box's performance and
reduce the interrupts rate or the CPU usage ?
Also, we've noticed a sharp drop in CPU usage since we've disabled
pfsync, but we'd rather keep it now wouldn't we ?
Last, we seem to get input errors on the NICs, although the switch ports
report not a single layer 2 error in over a year.
I'm wondering what counts as a NIC input error ?
Hardware is as follows:
CPU
--
CPU: Intel(R) Xeon(R) CPU E5420 @ 2.50GHz (2496.25-MHz
K8-class CPU)
Origin = "GenuineIntel" Id = 0x10676 Stepping = 6
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0xce3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1>
AMD Features=0x20100800<SYSCALL,NX,LM>
AMD Features2=0x1<LAHF>
TSC: P-state invariant
ACPI APIC Table: <DELL PE_SC3 >
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
cpu2 (AP): APIC ID: 2
cpu3 (AP): APIC ID: 3
MEM
--
real memory = 2147483648 (2048 MB)
avail memory = 2057293824 (1961 MB)
NICs
--
bce0: <Broadcom NetXtreme II BCM5708 1000Base-T (B2)> mem
0xf4000000-0xf5ffffff irq 16 at device 0.0 on pci7
bce1: <Broadcom NetXtreme II BCM5708 1000Base-T (B2)> mem
0xf8000000-0xf9ffffff irq 16 at device 0.0 on pci3
igb0: <Intel(R) PRO/1000 Network Connection version - 1.7.3> port
0xdce0-0xdcff mem
0xfd0e0000-0xfd0fffff,0xfce00000-0xfcffffff,0xfd0dc000-0xfd0dffff irq 18
at device 0.0 on pci14
igb0: Using MSIX interrupts with 3 vectors
Find below different outputs from the current master running FreeBSD
8.0-RELEASE-p2
systat -v
---
3 users Load 0.41 0.31 0.29 Jan 26 18:59
Mem:KB REAL VIRTUAL VN PAGER SWAP
PAGER
Tot Share Tot Share Free in out in
out
Act 143036 8152 836392 11188 1262556 count
All 168224 10420 1074653k 31172 pages
Proc: Interrupts
r p d s w Csw Trp Sys Int Sof Flt cow 36163 total
47 105k 76 2077 28k 223 zfod
ata0 irq14
ozfod
mfi0 irq16
4.3%Sys 28.1%Intr 3.0%User 0.0%Nice 64.7%Idle %ozfod
uhci0 uhci
| | | | | | | | | | | daefr 1998
cpu0: time
==++++++++++++++>> prcfr 9428
bce0 256
33 dtbuf totfr 12931
igb0 257
Namei Name-cache Dir-cache 100000 desvn react 5791
igb0 258
Calls hits % hits % 70448 numvn pdwak
igb0 259
24988 frevn pdpgs
igb1 260
intrn 1
igb1 261
Disks mfid0 372392 wire
igb1 262
KB/t 0.00 62336 act 20
bce1 269
tps 0 323720 inact 1998
cpu1: time
MB/s 0.00 292 cache 1998
cpu2: time
%busy 0 1262264 free 1998
cpu3: time
218272 buf
vmstat -i
---
interrupt total rate
irq14: ata0 36 0
irq16: mfi0 353244 1
irq21: uhci0 uhci+ 461504 1
cpu0: timer 615183815 1996
irq256: bce0 1015412475 3295
irq257: igb0 1067318584 3464
irq258: igb0 695648752 2258
irq259: igb0 2 0
irq260: igb1 11503857 37
irq261: igb1 506598 1
irq262: igb1 69 0
irq269: bce1 790820 2
cpu1: timer 615183757 1996
cpu2: timer 615197165 1996
cpu3: timer 615197165 1996
Total 5252757843 17050
pf status (159 filter rules, 17 nat/rdr rules)
---
# pfctl -si
Status: Enabled for 3 days 13:34:56 Debug: Urgent
Interface Stats for igb0 IPv4 IPv6
Bytes In 487209136643 384
Bytes Out 687158173727 0
Packets In
Passed 1967249106 0
Blocked 6183860 6
Packets Out
Passed 2018192359 0
Blocked 686901 0
State Table Total Rate
current entries 25428
searches 9006187476 29231.8/s
inserts 679746853 2206.3/s
removals 679721425 2206.2/s
Counters
match 686988143 2229.8/s
bad-offset 0 0.0/s
fragment 56 0.0/s
short 0 0.0/s
normalize 171 0.0/s
memory 0 0.0/s
bad-timestamp 0 0.0/s
congestion 0 0.0/s
ip-option 1 0.0/s
proto-cksum 13916 0.0/s
state-mismatch 220169 0.7/s
state-insert 0 0.0/s
state-limit 0 0.0/s
src-limit 1812 0.0/s
synproxy 0 0.0/s
Regards,
--
dfl
W dniu 2011-01-27 10:57, Damien Fleuriot pisze:> Hello list, > > I have a problem with interrupts, network cards, and PF performance. >I think you should try with polling(4) enabled and probably increase kernel.hz i sysctl.conf :) -- Bartosz Stec
On 1/27/11 11:03 AM, Bartosz Stec wrote:> W dniu 2011-01-27 10:57, Damien Fleuriot pisze: >> Hello list, >> >> I have a problem with interrupts, network cards, and PF performance. >> > I think you should try with polling(4) enabled and probably increase > kernel.hz i sysctl.conf :) >As a matter of fact, we tried polling on the backup firewall yesterday with the following kernel options: options DEVICE_POLLING options HZ=1000 This had disastrous results. First, our LAN and DMZ interfaces (bce0 and 1) do not support polling, so no change here. Second, the WAN interface (igb0) supports polling but that caused problems with carp0 and the physical interface resetting itself for god knows what reason: carp0: link state changed to DOWN carp0: INIT -> BACKUP igb0: link state changed to UP carp0: link state changed to DOWN carp0: link state changed to UP carp0: MASTER -> BACKUP (more frequent advertisement received) carp0: link state changed to DOWN carp0: link state changed to UP igb0: Watchdog timeout -- resetting igb0: Queue(1) tdh = 57, hw tdt = 57 igb0: TX(1) desc avail = 967,Next TX to Clean = 0 igb0: link state changed to DOWN carp0: link state changed to DOWN carp0: INIT -> BACKUP igb0: link state changed to UP carp0: link state changed to DOWN carp0: link state changed to UP carp0: link state changed to DOWN igb0: Watchdog timeout -- resetting igb0: Queue(3) tdh = 5, hw tdt = 5 igb0: TX(3) desc avail = 1019,Next TX to Clean = 0 igb0: link state changed to DOWN igb0: link state changed to UP igb0: Watchdog timeout -- resetting igb0: Queue(2) tdh = 53, hw tdt = 53 igb0: TX(2) desc avail = 971,Next TX to Clean = 0 igb0: link state changed to DOWN igb0: link state changed to UP igb0: Watchdog timeout -- resetting igb0: Queue(2) tdh = 19, hw tdt = 19 igb0: TX(2) desc avail = 1005,Next TX to Clean = 0 igb0: link state changed to DOWN igb0: link state changed to UP
On Thu, Jan 27, 2011 at 10:57:14AM +0100, Damien Fleuriot wrote:> Hello list, > > I have a problem with interrupts, network cards, and PF performance. > > We have 2 firewalls running FreeBSD 8.0 for the current master and > FreeBSD 8.1 for the backup host, which I upgraded just yesterday. > > [...] > > vmstat -i > --- > interrupt total rate > irq14: ata0 36 0 > irq16: mfi0 353244 1 > irq21: uhci0 uhci+ 461504 1 > cpu0: timer 615183815 1996 > irq256: bce0 1015412475 3295 > irq257: igb0 1067318584 3464 > irq258: igb0 695648752 2258 > irq259: igb0 2 0 > irq260: igb1 11503857 37 > irq261: igb1 506598 1 > irq262: igb1 69 0 > irq269: bce1 790820 2 > cpu1: timer 615183757 1996 > cpu2: timer 615197165 1996 > cpu3: timer 615197165 1996 > Total 5252757843 17050There are changes to the igb(4) driver which are in RELENG_8 (8-STABLE), and some which will be in the upcoming 8.2-RELEASE, which may address this. Jack Vogel of Intel would be able to confirm for sure; CC'ing him here. Could you please provide output from the following commands? * pciconf -lvcb (only include igbX entries, thanks) * sysctl -a | grep msi Thanks. I can't help with the CARP-related issues or other stuff you're experiencing. These issues may all be separate problems, hard to say. -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP 4BD6C0CB |
On 1/27/11 7:46 PM, Sergey Lobanov wrote:> ? ????????? ?? ??????? 28 ?????? 2011 00:55:35 ????? Damien Fleuriot ???????: >> On 1/27/11 6:41 PM, Vogel, Jack wrote: >>> Jeremy is right, if you have a problem the first step is to try the >>> latest code. >>> >>> However, when I look at the interrupts below I don't see what the problem >>> is? The Broadcom seems to have about the same rate, it just doesn't have >>> MSIX (multiple vectors). >>> >>> Jack >> >> My main concern is that the CPU %interrupt is quite high, also, we seem >> to be experiencing input errors on the interfaces. > Would you show igb tuning which is done in loader.conf and output of sysctl > dev.igb.0? > Did you rise number of igb descriptors such as: > hw.igb.rxd=4096 > hw.igb.txd=4096 ?There is no tuning at all on our part in the loader's conf. Find below the sysctls: # sysctl -a |grep igb dev.igb.0.%desc: Intel(R) PRO/1000 Network Connection version - 1.7.3 dev.igb.0.%driver: igb dev.igb.0.%location: slot=0 function=0 dev.igb.0.%pnpinfo: vendor=0x8086 device=0x10d6 subvendor=0x8086 subdevice=0x145a class=0x020000 dev.igb.0.%parent: pci14 dev.igb.0.debug: -1 dev.igb.0.stats: -1 dev.igb.0.flow_control: 3 dev.igb.0.enable_aim: 1 dev.igb.0.low_latency: 128 dev.igb.0.ave_latency: 450 dev.igb.0.bulk_latency: 1200 dev.igb.0.rx_processing_limit: 100 dev.igb.1.%desc: Intel(R) PRO/1000 Network Connection version - 1.7.3 dev.igb.1.%driver: igb dev.igb.1.%location: slot=0 function=1 dev.igb.1.%pnpinfo: vendor=0x8086 device=0x10d6 subvendor=0x8086 subdevice=0x145a class=0x020000 dev.igb.1.%parent: pci14 dev.igb.1.debug: -1 dev.igb.1.stats: -1 dev.igb.1.flow_control: 3 dev.igb.1.enable_aim: 1 dev.igb.1.low_latency: 128 dev.igb.1.ave_latency: 450 dev.igb.1.bulk_latency: 1200 dev.igb.1.rx_processing_limit: 100