freebsd stable - Jul 2010 - packet loss on ixgbe using vlans and ipv6

Hi,

I have a Dell T710 with 4 X 10G ethernet interfaces (2 X Dual port Intel
82599 cards). It is running FreeBSD RELENG_8 last updated on July 13.

What I see is packet loss (0 - 40%) on IPv6 packets in vlans, when the
machine is not the originator of the packets.

Let me try to describe a little more. If a neigbouring machine ping6 it,
there will be packet loss. If it act as a router for ipv6, there will be
packet loss. This happen even when the network is pretty idle and with
different switches (Nortel and Cisco equipment). The packet loss is
very fluctuating. Pinging 1000 packets might loose 1% one time and the
next time 30%. Looking with tcpdump, I can see the packets arriving and
going out, but the packet never arrive at the next machine. (My feeling is
that they get lost inside the card.) The error counters on the switch
does not increment.

I do not see packet loss if the machine originate the packets, for example
ping6 from the machine. Also ipv4 packets do not have any packets loss. If
I do not use vlans, I don't see packet loss with ipv6 either.

pciconf -l of the ethernet cards:

ix0@pci0:129:0:0:       class=0x020000 card=0x00038086 chip=0x10fb8086 rev=0x01
hdr=0x00
ix1@pci0:129:0:1:       class=0x020000 card=0x00038086 chip=0x10fb8086 rev=0x01
hdr=0x00
ix2@pci0:131:0:0:       class=0x020000 card=0x00038086 chip=0x10fb8086 rev=0x01
hdr=0x00
ix3@pci0:131:0:1:       class=0x020000 card=0x00038086 chip=0x10fb8086 rev=0x01
hdr=0x00

dmesg of the first ethernet card. The others look the same, except for the
memory and irq values:

ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.2.1> port
0xdcc
0-0xdcdf mem 0xd4100000-0xd417ffff,0xd40f8000-0xd40fbfff irq 64 at device 0.0 on
pci129
ix0: Using MSIX interrupts with 17 vectors
ix0: [ITHREAD]
...
ix0: Ethernet address: 00:1b:21:57:b4:20
ix0: PCI Express Bus: Speed 5.0Gb/s Width x8

If anybody need more info, please ask.

John
-- 
John Hay -- jhay@meraka.csir.co.za / jhay@FreeBSD.org

On Mon, Jul 19, 2010 at 10:25:42PM +0200, John Hay
wrote:
> I have a Dell T710 with 4 X 10G ethernet interfaces (2 X Dual port Intel
> 82599 cards). It is running FreeBSD RELENG_8 last updated on July 13.
> 
> What I see is packet loss (0 - 40%) on IPv6 packets in vlans, when the
> machine is not the originator of the packets.
> 
> Let me try to describe a little more. If a neigbouring machine ping6 it,
> there will be packet loss. If it act as a router for ipv6, there will be
> packet loss. This happen even when the network is pretty idle and with
> different switches (Nortel and Cisco equipment). The packet loss is
> very fluctuating. Pinging 1000 packets might loose 1% one time and the
> next time 30%. Looking with tcpdump, I can see the packets arriving and
> going out, but the packet never arrive at the next machine. (My feeling is
> that they get lost inside the card.) The error counters on the switch
> does not increment.
> 
> I do not see packet loss if the machine originate the packets, for example
> ping6 from the machine. Also ipv4 packets do not have any packets loss. If
> I do not use vlans, I don't see packet loss with ipv6 either.
> 
> pciconf -l of the ethernet cards:
>
> ix0@pci0:129:0:0:       class=0x020000 card=0x00038086 chip=0x10fb8086
rev=0x01 hdr=0x00
> ix1@pci0:129:0:1:       class=0x020000 card=0x00038086 chip=0x10fb8086
rev=0x01 hdr=0x00
> ix2@pci0:131:0:0:       class=0x020000 card=0x00038086 chip=0x10fb8086
rev=0x01 hdr=0x00
> ix3@pci0:131:0:1:       class=0x020000 card=0x00038086 chip=0x10fb8086
rev=0x01 hdr=0x00
Can you provide pciconf -lvc output for the ix[0-3] cards instead?  I
believe Jack Vogel will need this.  vmstat -i might also be helpful
(full output).

Thanks!

-- 
| Jeremy Chadwick                                   jdc@parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.              PGP: 4BD6C0CB |