thr3ads.net - freebsd stable - em very slow, shared irq... on 6.3p8 [Feb 2008]

If this information is useful, please help other people find it:
Share via:

Holger Kipp

2008-Feb-27 11:01 UTC

em very slow, shared irq... on 6.3p8

Hello,

I updated a system with 12 dc-interfaces to a new hardware
with 14 em-interfaces. Yes, it is a firewall.
New System is 6.2-RELEASE-p8.

What I now experience between two internal networks (100MBit/s each) 
is the following:
1318 packets transmitted, 1317 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.387/246.153/2441.392/324.142 ms

tcpdump on the firewall shows similar delays (on the outgoing 
interface).

tcpdump on the system I ping however shows very quick responses
for incoming packages (ie usually less than a millisecond).

I therefore assume that the problem is between receiving the
irq from em<x> and getting the data from the interface on the firewall
itself.

My first option would be to activate polling on em-interfaces - but as
I did not experience this sort of notieceable slowdown with the old
dc-based firewall (without polling), maybe someone can shed some light
on this strange behaviour or has other suggestions as well?

More details and dmesg below.

Help and suggestions welcome :-)

Best regards,
Holger Kipp

PS: Regarding the MPSAFE-issue - I need to use ipsec, so don't have a choice
here. 




On the firewall I see many irqs (sometimes more than 5000) very often
on one of the em-interfaces using sysctl/vmstat:

--------------------------------------------------------------------------------
    2 users    Load  0.00  0.00  0.04                  Feb 27 12:20

Mem:KB    REAL            VIRTUAL                     VN PAGER  SWAP PAGER
        Tot   Share      Tot    Share    Free         in  out     in  out
Act   14804    3724    32800     3800 1327728 count
All  722324    5248 76723364     5696         pages
                                                                 Interrupts
Proc:r  p  d  s  w    Csw  Trp  Sys  Int  Sof  Flt        cow    7606 total
             27      7862    1  462 4105           158840 wire        1: atkb
                                                    14988 act         6: fdc0
 0.2%Sys   3.0%Intr  0.0%User  0.0%Nice 96.8%Idl   547868 inact       12: psm
|    |    |    |    |    |    |    |    |    |       1244 cache       14: ata
++                                                1326484 free    359 16: em4
                                                          daefr  2761 17: em5
Namei         Name-cache    Dir-cache                     prcfr       28: twe
    Calls     hits    %     hits    %                     react   306 52: em0
                                                          pdwak     1 53: em1
                                          zfod            pdpgs   177 55: em3
Disks twed0                               ozfod           intrn  2001 cpu0: time
KB/t  21.33                               %slo-z   114464 buf    2001 cpu1: time
tps       1                             1 tfree        10 dirtybuf
MB/s   0.01                                        100000 desiredvnodes
% busy    0                                         66594 numvnodes
                                                    19758 freevnodes
--------------------------------------------------------------------------------

dmesg

Copyright (c) 1992-2007 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 6.2-RELEASE-p8 #1: Thu Oct  4 16:07:31 CEST 2007
WARNING: debug.mpsafenet forced to 0 as ipsec requires Giant
WARNING: MPSAFE network stack disabled, expect reduced performance.
MPTable: <INTEL    MUKILTEO    >
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Core(TM)2 CPU          6420  @ 2.13GHz (2128.01-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0x6f6  Stepping = 6
 
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
 
Features2=0xe3bd<SSE3,RSVD2,MON,DS_CPL,VMX,EST,TM2,<b9>,CX16,<b14>,<b15>>
  AMD Features=0x20100000<NX,LM>
  AMD Features2=0x1<LAHF>
  Cores per package: 2
real memory  = 2146349056 (2046 MB)
avail memory = 2095296512 (1998 MB)
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
ioapic0: Assuming intbase of 0
ioapic1: Assuming intbase of 24
ioapic2: Assuming intbase of 48
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
ioapic2 <Version 2.0> irqs 48-71 on motherboard
kbd1 at kbdmux0
cpu0 on motherboard
cpu1 on motherboard
pcib0: <MPTable Host-PCI bridge> pcibus 0 on motherboard
pci0: <PCI bus> on pcib0
pcib0: unable to route slot 28 INTB
pcib1: <PCI-PCI bridge> irq 16 at device 1.0 on pci0
pci1: <PCI bus> on pcib1
pcib2: <MPTable PCI-PCI bridge> at device 0.0 on pci1
pci2: <PCI bus> on pcib2
twe0: <3ware Storage Controller. Driver version 1.50.01.002> port
0x4000-0x400f mem 0xe0100000-0xe010000f,0xe0800000-0xe0ffffff irq 28 at device
2.0 on pci2
twe0: [GIANT-LOCKED]
twe0: 2 ports, Firmware FE8S 1.05.00.068, BIOS BE7X 1.08.00.048
pci1: <base peripheral, interrupt controller> at device 0.1 (no driver
attached)
pcib3: <PCI-PCI bridge> at device 0.2 on pci1
pci3: <PCI bus> on pcib3
pcib4: <MPTable PCI-PCI bridge> at device 3.0 on pci3
pci4: <PCI bus> on pcib4
em0: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port
0x5000-0x503f mem 0xe1000000-0xe101ffff irq 52 at device 4.0 on pci4
em0: Ethernet address: 00:0e:0c:c6:cc:54
em0: [GIANT-LOCKED]
em1: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port
0x5040-0x507f mem 0xe1020000-0xe103ffff irq 53 at device 4.1 on pci4
em1: Ethernet address: 00:0e:0c:c6:cc:55
em1: [GIANT-LOCKED]
em2: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port
0x5080-0x50bf mem 0xe1040000-0xe105ffff irq 54 at device 6.0 on pci4
em2: Ethernet address: 00:0e:0c:c6:cc:56
em2: [GIANT-LOCKED]
em3: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port
0x50c0-0x50ff mem 0xe1060000-0xe107ffff irq 55 at device 6.1 on pci4
em3: Ethernet address: 00:0e:0c:c6:cc:57
em3: [GIANT-LOCKED]
pci1: <base peripheral, interrupt controller> at device 0.3 (no driver
attached)
pcib5: <PCI-PCI bridge> irq 16 at device 3.0 on pci0
pci10: <PCI bus> on pcib5
pcib6: <PCI-PCI bridge> at device 0.0 on pci10
pci11: <PCI bus> on pcib6
pcib7: <MPTable PCI-PCI bridge> at device 0.0 on pci11
pci12: <PCI bus> on pcib7
em4: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port
0x6000-0x601f mem 0xe1120000-0xe113ffff,0xe1100000-0xe111ffff irq 16 at device
0.0 on pci12
em4: Memory Access and/or Bus Master bits were not set!
em4: Ethernet address: 00:15:17:25:48:88
em4: [GIANT-LOCKED]
em5: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port
0x6020-0x603f mem 0xe1160000-0xe117ffff,0xe1140000-0xe115ffff irq 17 at device
0.1 on pci12
em5: Memory Access and/or Bus Master bits were not set!
em5: Ethernet address: 00:15:17:25:48:89
em5: [GIANT-LOCKED]
pcib8: <MPTable PCI-PCI bridge> at device 1.0 on pci11
pci13: <PCI bus> on pcib8
em6: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port
0x7000-0x701f mem 0xe1220000-0xe123ffff,0xe1200000-0xe121ffff irq 16 at device
0.0 on pci13
em6: Memory Access and/or Bus Master bits were not set!
em6: Ethernet address: 00:15:17:25:48:8a
em6: [GIANT-LOCKED]
em7: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port
0x7020-0x703f mem 0xe1260000-0xe127ffff,0xe1240000-0xe125ffff irq 17 at device
0.1 on pci13
em7: Memory Access and/or Bus Master bits were not set!
em7: Ethernet address: 00:15:17:25:48:8b
em7: [GIANT-LOCKED]
pcib9: <PCI-PCI bridge> irq 17 at device 28.0 on pci0
pci14: <PCI bus> on pcib9
pcib10: <PCI-PCI bridge> at device 0.0 on pci14
pci15: <PCI bus> on pcib10
pcib11: <MPTable PCI-PCI bridge> at device 0.0 on pci15
pci16: <PCI bus> on pcib11
em8: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port
0x8000-0x801f mem 0xe1320000-0xe133ffff,0xe1300000-0xe131ffff irq 16 at device
0.0 on pci16
em8: Memory Access and/or Bus Master bits were not set!
em8: Ethernet address: 00:15:17:25:48:64
em8: [GIANT-LOCKED]
em9: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port
0x8020-0x803f mem 0xe1360000-0xe137ffff,0xe1340000-0xe135ffff irq 17 at device
0.1 on pci16
em9: Memory Access and/or Bus Master bits were not set!
em9: Ethernet address: 00:15:17:25:48:65
em9: [GIANT-LOCKED]
pcib12: <MPTable PCI-PCI bridge> at device 1.0 on pci15
pci17: <PCI bus> on pcib12
em10: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port
0x9000-0x901f mem 0xe1420000-0xe143ffff,0xe1400000-0xe141ffff irq 16 at device
0.0 on pci17
em10: Memory Access and/or Bus Master bits were not set!
em10: Ethernet address: 00:15:17:25:48:66
em10: [GIANT-LOCKED]
em11: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port
0x9020-0x903f mem 0xe1460000-0xe147ffff,0xe1440000-0xe145ffff irq 17 at device
0.1 on pci17
em11: Memory Access and/or Bus Master bits were not set!
em11: Ethernet address: 00:15:17:25:48:67
em11: [GIANT-LOCKED]
pcib13: <MPTable PCI-PCI bridge> irq 17 at device 28.4 on pci0
pci18: <PCI bus> on pcib13
em12: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port
0xa000-0xa01f mem 0xe1500000-0xe151ffff irq 16 at device 0.0 on pci18
em12: Ethernet address: 00:30:48:90:86:7a
em12: [GIANT-LOCKED]
pcib14: <MPTable PCI-PCI bridge> irq 7 at device 28.5 on pci0
pci19: <PCI bus> on pcib14
em13: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port
0xb000-0xb01f mem 0xe1600000-0xe161ffff irq 17 at device 0.0 on pci19
em13: Ethernet address: 00:30:48:90:86:7b
em13: [GIANT-LOCKED]
pcib15: <MPTable PCI-PCI bridge> at device 30.0 on pci0
pci20: <PCI bus> on pcib15
pci20: <display, VGA> at device 4.0 (no driver attached)
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel ICH7 UDMA100 controller> port
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x3020-0x302f at device 31.1 on pci0
ata0: <ATA channel 0> on atapci0
ata1: <ATA channel 1> on atapci0
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
pmtimer0 on isa0
orm0: <ISA Option ROMs> at iomem
0xc0000-0xcafff,0xcb000-0xcbfff,0xcc000-0xccfff,0xcd000-0xcdfff on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: model MouseMan+, device ID 0
fdc0: <Enhanced floppy controller> at port 0x3f0-0x3f5,0x3f7 irq 6 drq 2
on isa0
fdc0: [FAST]
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
ppc0: parallel port not found.
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
sio0: type 16550A
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
unknown: <PNP0303> can't assign resources (port)
unknown: <INT0800> can't assign resources (memory)
unknown: <PNP0c02> can't assign resources (memory)
unknown: <PNP0f13> can't assign resources (irq)
unknown: <PNP0501> can't assign resources (port)
unknown: <PNP0700> can't assign resources (port)
Timecounters tick every 1.000 msec
IPsec: Initialized Security Association Processing.
IP Filter: v4.1.13 initialized.  Default = pass all, Logging = enabled
acd0: DVDROM <HL-DT-STDVD-ROM GDR8164B/0L06> at ata0-slave UDMA33
twed0: <Unit 0, TwinStor, Normal> on twe0
twed0: 76318MB (156299440 sectors)
SMP: AP CPU #1 Launched!
Trying to mount root from ufs:/dev/twed0s1a
em9: link state changed to UP
em8: link state changed to UP
em10: link state changed to UP
em0: link state changed to UP
em3: link state changed to UP
[...]

JoaoBR

2008-Feb-27 13:08 UTC

head link

em very slow, shared irq... on 6.3p8

On Wednesday 27 February 2008 07:49:42 Holger Kipp
wrote:> Hello,
>
> I updated a system with 12 dc-interfaces to a new hardware
> with 14 em-interfaces. Yes, it is a firewall.
> New System is 6.2-RELEASE-p8.
>
> What I now experience between two internal networks (100MBit/s each)
> is the following:
> 1318 packets transmitted, 1317 packets received, 0% packet loss
> round-trip min/avg/max/stddev = 0.387/246.153/2441.392/324.142 ms
>
> tcpdump on the firewall shows similar delays (on the outgoing
> interface).
>
> tcpdump on the system I ping however shows very quick responses
> for incoming packages (ie usually less than a millisecond).
>
> I therefore assume that the problem is between receiving the
> irq from em<x> and getting the data from the interface on the
firewall
> itself.
>
> My first option would be to activate polling on em-interfaces - but as
> I did not experience this sort of notieceable slowdown with the old
> dc-based firewall (without polling), maybe someone can shed some light
> on this strange behaviour or has other suggestions as well?


I had a setup with 4 (dlink 4port + 1 nic nfe onboard) which run extremely 
stable 6.3

I upgraded the hardware (S939 -> AM2) and used two em 2port cards, without 
polling it was certanly unusable but with polling I got very good performance

without polling normally in a day or less the machine hung, no msg, simply 
freezed

with polling it stands some days up to two weeks when it freeze again

I upgraded to 7.0 same result

check your setup with vmstat -i and if you see two nics on the same interrupt 
I guess you get the same result as I got

actual I am running one em 2port and two single port pci cards what seems to 
be stable

anyway, similare setup on Tyan MBs do not have this problem where I also have 
8 nics on one system

I also do not have this problems on S939 boards only on AM2, so I am not sure 
where the problem is exactly but seems in a certain way hardware related

You could try changing your PS unit because you might be short on power with 
lots of em cards and SATA disks




-- 

Jo?o







A mensagem foi scaneada pelo sistema de e-mail e pode ser considerada segura.
Service fornecido pelo Datacenter Matik  https://datacenter.matik.com.br

Mike Tancsa

2008-Feb-27 14:52 UTC

head link

em very slow, shared irq... on 6.3p8

At 05:49 AM 2/27/2008, Holger Kipp wrote:
>I therefore assume that the problem is between receiving the
>irq from em<x> and getting the data from the interface on the firewall
>itself.
I would try upgrading to 6.3R (there are several em driver bug fixes) 
and then try the box with
% cat /boot/loader.conf
hw.pci.enable_msi=1

...if the cards support msi.

I think pciconf -lvc should tell you if the cards and slots support it or not.

Also, if you dont need IPV6, use FAST_IPSEC. It does not need 
mpsafe.  If you do need IPSEC and IPV6, 7.0R got rid of that restriction.

         ---Mike

freebsd stable - Feb 2008 - em very slow, shared irq... on 6.3p8

em very slow, shared irq... on 6.3p8

em very slow, shared irq... on 6.3p8

em very slow, shared irq... on 6.3p8