Rudy wrote:> I am having watchdog timeout issues with my Intel 82573 Pro/1000 ...
> http://lists.freebsd.org/pipermail/freebsd-net/2008-May/018075.html
>
> link to dcgdis.zip didn't work. Do you have a copy?
>
Thanks, Jack. Got the file and flashed -- no upgrade needed.
So, while the router was offline, I flashed the motherboards bios
(Supermicro X7Sbi), upgraded to 7.2-STABLE, and downloaded the 6.9.12
version of the em driver. Still, watchdog timeouts. Sigh.
Will the Intel Gigabit ET Quad Port Adapter make my the timeouts go away???
Should I be using amd64???
Should tx_int_delay=0?
Summary:
2 Nics on Motherboard + quad card in PCIe slot.
Watchdog timeouts on motherboard nics and on quad card nic when
bandwidth > 10Mbps
There is minimal (bgp session) TCP to the box... it only forwards
packets between interfaces.
# uname -r -m
7.2-STABLE i386
# dmesg | grep ^em
em0: <Intel(R) PRO/1000 Network Connection 6.9.12> port 0x2000-0x201f
mem 0xd0220000-0xd023ffff,0xd0200000-0xd021ffff irq 16 at device 0.0 on pci5
em0: Using MSI interrupt
em0: [FILTER]
em0: Ethernet address: 00:15:17:78:99:70
em1: <Intel(R) PRO/1000 Network Connection 6.9.12> port 0x2020-0x203f
mem 0xd0260000-0xd027ffff,0xd0240000-0xd025ffff irq 17 at device 0.1 on pci5
em1: Using MSI interrupt
em1: [FILTER]
em1: Ethernet address: 00:15:17:78:99:71
em2: <Intel(R) PRO/1000 Network Connection 6.9.12> port 0x3000-0x301f
mem 0xd0320000-0xd033ffff,0xd0300000-0xd031ffff irq 17 at device 0.0 on pci6
em2: Using MSI interrupt
em2: [FILTER]
em2: Ethernet address: 00:15:17:78:99:72
em3: <Intel(R) PRO/1000 Network Connection 6.9.12> port 0x3020-0x303f
mem 0xd0360000-0xd037ffff,0xd0340000-0xd035ffff irq 18 at device 0.1 on pci6
em3: Using MSI interrupt
em3: [FILTER]
em3: Ethernet address: 00:15:17:78:99:73
em4: <Intel(R) PRO/1000 Network Connection 6.9.12> port 0x4000-0x401f
mem 0xd0400000-0xd041ffff irq 16 at device 0.0 on pci13
em4: Using MSI interrupt
em4: [FILTER]
em4: Ethernet address: 00:30:48:67:14:50
em5: <Intel(R) PRO/1000 Network Connection 6.9.12> port 0x5000-0x501f
mem 0xd0500000-0xd051ffff irq 17 at device 0.0 on pci15
em5: Using MSI interrupt
em5: [FILTER]
em5: Ethernet address: 00:30:48:67:14:51
# vmstat -i
interrupt total rate
irq1: atkbd0 710 0
irq4: sio0 3 0
irq23: atapci0 14943 0
cpu0: timer 929753417 2000
irq256: em0 702754836 1511
irq257: em1 2 0
irq260: em4 469338728 1009
irq261: em5 78605337 169
cpu1: timer 929753403 2000
Total 3110221379 6690
# sysctl dev.em.0.stats=1
Sep 30 01:08:20 mango kernel: em0: Excessive collisions = 0
Sep 30 01:08:20 mango kernel: em0: Sequence errors = 0
Sep 30 01:08:20 mango kernel: em0: Defer count = 0
Sep 30 01:08:20 mango kernel: em0: Missed Packets = 101469
Sep 30 01:08:20 mango kernel: em0: Receive No Buffers = 0
Sep 30 01:08:20 mango kernel: em0: Receive Length Errors = 0
Sep 30 01:08:20 mango kernel: em0: Receive errors = 0
Sep 30 01:08:20 mango kernel: em0: Crc errors = 0
Sep 30 01:08:20 mango kernel: em0: Alignment errors = 0
Sep 30 01:08:20 mango kernel: em0: Collision/Carrier extension errors = 0
Sep 30 01:08:20 mango kernel: em0: RX overruns = 0
Sep 30 01:08:20 mango kernel: em0: watchdog timeouts = 15
Sep 30 01:08:20 mango kernel: em0: RX MSIX IRQ = 0 TX MSIX IRQ = 0 LINK
MSIX IRQ = 0
Sep 30 01:08:20 mango kernel: em0: XON Rcvd = 0
Sep 30 01:08:20 mango kernel: em0: XON Xmtd = 0
Sep 30 01:08:20 mango kernel: em0: XOFF Rcvd = 0
Sep 30 01:08:20 mango kernel: em0: XOFF Xmtd = 0
Sep 30 01:08:20 mango kernel: em0: Good Packets Rcvd = 1056196797
Sep 30 01:08:20 mango kernel: em0: Good Packets Xmtd = 1088726903
Sep 30 01:08:20 mango kernel: em0: TSO Contexts Xmtd = 4088
Sep 30 01:08:20 mango kernel: em0: TSO Contexts Failed = 0
# sysctl dev.em.0.debug=1
Sep 30 01:34:59 mango kernel: em0: Adapter hardware address = 0xc5159420
Sep 30 01:34:59 mango kernel: em0: CTRL = 0x401c0241 RCTL = 0x8002
Sep 30 01:34:59 mango kernel: em0: Packet buffer = Tx=16k Rx=32k
Sep 30 01:34:59 mango kernel: em0: Flow control watermarks high = 30720
low = 29220
Sep 30 01:34:59 mango kernel: em0: tx_int_delay = 66, tx_abs_int_delay = 66
Sep 30 01:34:59 mango kernel: em0: rx_int_delay = 0, rx_abs_int_delay = 66
Sep 30 01:34:59 mango kernel: em0: fifo workaround = 0, fifo_reset_count = 0
Sep 30 01:34:59 mango kernel: em0: hw tdh = 980, hw tdt = 980
Sep 30 01:34:59 mango kernel: em0: hw rdh = 203, hw rdt = 202
Sep 30 01:34:59 mango kernel: em0: Num Tx descriptors avail = 1024
Sep 30 01:34:59 mango kernel: em0: Tx Descriptors not avail1 = 0
Sep 30 01:34:59 mango kernel: em0: Tx Descriptors not avail2 = 0
Sep 30 01:34:59 mango kernel: em0: Std mbuf failed = 0
Sep 30 01:34:59 mango kernel: em0: Std mbuf cluster failed = 0
Sep 30 01:34:59 mango kernel: em0: Driver dropped packets = 0
Sep 30 01:34:59 mango kernel: em0: Driver tx dma failure in encap = 0
# grep watchdog /var/log/messages
Sep 28 19:52:24 mango kernel: em0: watchdog timeout -- resetting
Sep 28 20:09:56 mango kernel: em4: watchdog timeout -- resetting
Sep 28 22:43:15 mango kernel: em0: watchdog timeout -- resetting
Sep 29 02:20:12 mango kernel: em0: watchdog timeout -- resetting
Sep 29 02:35:59 mango kernel: em4: watchdog timeout -- resetting
Sep 29 03:46:10 mango kernel: em0: watchdog timeout -- resetting
Sep 29 04:16:11 mango kernel: em0: watchdog timeout -- resetting
Sep 29 05:30:29 mango kernel: em0: watchdog timeout -- resetting
Sep 29 06:29:04 mango kernel: em0: watchdog timeout -- resetting
Sep 29 09:50:38 mango kernel: em0: watchdog timeout -- resetting
Sep 29 11:23:19 mango kernel: em0: watchdog timeout -- resetting
Sep 29 11:28:31 mango kernel: em0: watchdog timeout -- resetting
Sep 29 13:11:21 mango kernel: em0: watchdog timeout -- resetting
Sep 29 13:26:08 mango kernel: em0: watchdog timeout -- resetting
Sep 29 14:54:37 mango kernel: em0: watchdog timeout -- resetting
Sep 29 15:28:28 mango kernel: em0: watchdog timeout -- resetting
Sep 29 15:34:50 mango kernel: em0: watchdog timeout -- resetting
I did reconfigure the network to move 20Mbps off of em0 (which is now
doing about 40Mbps) and I haven't had a watchdog for 6 hours.
# cat /etc/sysctl.conf
net.inet.ip.fastforwarding=1
net.inet.ip.intr_queue_maxlen=92
net.inet.icmp.drop_redirect=1
dev.em.0.rx_processing_limit=200
dev.em.1.rx_processing_limit=200
dev.em.2.rx_processing_limit=200
dev.em.4.rx_processing_limit=200
dev.em.5.rx_processing_limit=200
# cat /boot/loader.conf
### dynamic module to get latest driver
if_em_load="YES"
### add more descriptors to em devices.
hw.em.rxd=1024
hw.em.txd=1024
Rudy