Lustre discuss - Aug 2010 - New 10Gbe adapter on lustre client brought down switch due to broadcast traffic

Hi Guys,

I had a situation yesterday where I had a 10Gbe adapter on my lustre client
configured but not active (the cable was plugged in and had link but the
port was down) and this actually brought down our cisco switch.  This is a
new port that I am setting up.  The reason why the switch went down was due
to the amount of broadcast traffic coming from this port even though it was
shutdown:

..snip..
TenGigabitEthernet7/4 is down, line protocol is down (notconnect)
  Hardware is C6k 10000Mb 802.3, address is 001d.4577.c693 (bia
001d.4577.c693)
  MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 10Gb/s
  input flow-control is off, output flow-control is off
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 09:26:18, output hang never
  Last clearing of "show interface" counters 14:46:54
  Input queue: 0/2000/4149536364/0 (size/max/drops/flushes); Total output
drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 0 bits/sec, 0 packets/sec
     212613091560 packets input, 13607237844800 bytes, 0 no buffer
     Received 212613090908 broadcasts (2159693404 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 4149536364 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     154150900 packets output, 232610834033 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out
..snip..

All my lustre traffic should be going through my ib interface based on my
modprobe.conf.local:

options lnet networks="o2ib3(ib0)"

Just wanted to put this out there and see if anyone can point out something
that I might be missing.  These broadcasts should not be lustre related,
right?  Also, it does not make sense that this port is sending out broadcast
traffic when the port is not even active.

Any comments/assistance would be grately appreciated.

Thanks,
-J
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100802/d182d941/attachment.html

On Mon, 2010-08-02 at 10:24 -0700, Jagga Soorma wrote:
> Hi Guys,
Hi,
> I had a situation yesterday where I had a 10Gbe adapter on my lustre
> client
> configured but not active (the cable was plugged in and had link but
> the
> port was down) and this actually brought down our cisco switch.
Hrm.  A switch that can be taken down by a single NIC sending any amount
of broadcast traffic it''s capable of sending sounds like a broken
switch
to me.

Put another way, if I found a switch in my network that could be taken
out in any way at all by a single NIC, I would be replacing it
immediately either by another of the same switch (assuming it''s just a
broken piece of equipment) or another make/model (assuming it''s
defective by design).

b.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part
Url :
http://lists.lustre.org/pipermail/lustre-discuss/attachments/20100803/f6550d4b/attachment.bin