Hoping someone else has seen this before.
I have a few dozen Dell R610 systems with CentOS 5.2 that are
using kernels from 5.3 and 5.4 (2.6.18-128.1.10.el5 & 2.6.18-164.6.1.el5),
that at random lose layer 2 network connectivity either partially
or totally. Running tcpdump on the interface reveals only ARP
broadcasts, no responses. Switch reports no packets being
received on the interface.
Systems can run for days/weeks or even months without an issue then
drop off the network. At first I thought it was the Dell switches
which we had lots of problems with but it has happened on two other
brands of switches as well(Cisco and Extreme), so I no longer believe
it's the switch but rather the systems.
The workaround is to restart the network on the system. I have even
configured the bonding driver to do ARP requests and fail over to
the backup link in the event that fails but wasn't successful there
either as both links can go down, and/or the system can go into
"degraded" state where it can reach some systems but not others.
I have ESXi systems running on the same hardware and to-date have not
seen any of them drop off the same way.
System can be under high traffic load at the time or completely
idle, it doesn't seem to make a difference. No log entries indicating
what might be going on.
I have a case open with Dell but am not expecting a whole lot from
them, maybe I'll get lucky though. They asked me to upgrade the NIC
firmware which I did on a batch of systems to no avail(the release
notes for the firmware said nothing about any fixes that sounded
like my issue).
Driver versions:
ESXi (vSphere):
Broadcom NetXtreme II Gigabit Ethernet Driver bnx2 v1.6.9 (December 8, 2007)
Most linux systems(5.3 kernel):
Broadcom NetXtreme II Gigabit Ethernet Driver bnx2 v1.7.9-1 (July 18, 2008)
Some linux systems(5.4 kernel):
Broadcom NetXtreme II Gigabit Ethernet Driver bnx2 v1.9.3 (March 17, 2009)
Happens across at least a dozen systems spread over 4 data centers.
Never seen this sort of behavior before in the hundreds and hundreds
of systems I've run. These systems are all new, the R610 hardware
was released around May 2009, and we've been having issues since
day 1, but only recently have been able to rule the switches out as
the cause.
The latest driver on Broadcom's site is 1.9.20b which seems odd since
CentOS 5.4 seems to come with 1.9.3(the date on the Broadcom site is
more recent than the date on the linux kernel driver in 5.4) Most of
the fixes in the recent driver versions seem to focus around iSCSI,
which I'm not using.
lspci says:
02:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709
Gigabit Ethernet (rev 20)
Subsystem: Dell Unknown device 0236
Flags: bus master, fast devsel, latency 0, IRQ 114
Memory at dc000000 (64-bit, non-prefetchable) [size=32M]
Capabilities: [48] Power Management version 3
Capabilities: [50] Vital Product Data
Capabilities: [58] Message Signalled Interrupts: 64bit+ Queue=0/4
Enable-
Capabilities: [a0] MSI-X: Enable- Mask- TabSize=9
Capabilities: [ac] Express Endpoint IRQ 0
Capabilities: [100] Device Serial Number c9-dc-93-fe-ff-9b-21-00
Capabilities: [110] Advanced Error Reporting
Capabilities: [150] Power Budgeting
Capabilities: [160] Virtual Channel
I suppose I could go build the latest driver from their site and see
how it goes..
thanks
nate