freebsd stable - Mar 2012 - 9-STABLE + Infiniband - incorrect interface counters

Hi,

I'm playing with two FreeBSD 9-STABLE boxes connected via 10Gbps 
Infiniband (more details below) in Infiniband connected mode.

I see incorrect interface statistics (e.g. in netstat output), output 
counters are 2x more than expected.

EXAMPLE, ftp transfer of 1 GiB file:

ftp> put file /dev/null
local: file remote: /dev/null
229 Entering Extended Passive Mode (|||57978|)
150 Opening BINARY mode data connection for '/dev/null'.
100% |***********************************|   953 MiB  390.43 MiB/s    
00:00 ETA
226 Transfer complete.
1000000000 bytes sent in 00:02 (390.13 MiB/s)

Netstat on receiving side, counters are correct (for input):

lexa@home-gw:/home/lexa# netstat -I ib1 5
             input          (ib1)           output
    packets  errs idrops      bytes    packets  errs      bytes colls
          0     0     0          0          0     0          0     0
      13955     0     0  222688126       9027     0    1192796     0
      48921     0     0  780832960      32129     0    4240596     0
          0     0     0          0          0     0         80     0

Sum of bytes (input) is 1003521086, as expected.

Netstat on sending size, output is 2x more:

lexa@new-gw:/home/lexa# netstat -I ib0 5
             input          (ib0)           output
    packets  errs idrops      bytes    packets  errs      bytes colls
          1     0     0        100          0     0          0     0
      41162     0     0    2305210      62878     0 2008325984     0
          1     0     0        100          0     0          0     0

It looks like packet count is correct (13955+48921=62876, two packets 
missed somewhere), while byte count is exact 2x more.

==== More details on my setup ===
FreeBSD 9-STABLE, cvsuped today. One box is Core 2 Quad (Q9300), second 
one Core i7-920

1) Device MELLANOX MHEA28-XTC 10GBPS INFINIBAND HCA CARD (two port)
Boot message:
ib_mthca0: <ib_mthca> mem 0xfe900000-0xfe9fffff,0xfd000000-0xfd7fffff 
irq 16 at device 0.0 on pci1
ib_mthca: Mellanox InfiniBand HCA driver v1.0-ofed1.5.2 (August 4, 2010)

Two cards connected via cable, no Infiniband switch

2) Kernel config:
include GENERIC
options OFED
options SDP
device ipoib
options IPOIB_CM
device mthca


3) Regardles  of MTU settings (tried 16000, 32000, 48000), actual packed 
size in tcp flow is about 16000. Have not investigated it in details

4) There is no packet loss:
lexa@new-gw:/home/lexa# ping -s 32000 -c 10000 -f 10.1.1.1
PING 10.1.1.1 (10.1.1.1): 32000 data bytes
.
--- 10.1.1.1 ping statistics ---
10000 packets transmitted, 10000 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.157/0.225/1.758/0.156 ms


-- 
Alex Tutubalin
Web: http://blog.lexa.ru
mailto:lexa@lexa.ru

On Saturday, March 24, 2012 7:52:08 am Alex Tutubalin
wrote:
> Hi,
> 
> I'm playing with two FreeBSD 9-STABLE boxes connected via 10Gbps 
> Infiniband (more details below) in Infiniband connected mode.
> 
> I see incorrect interface statistics (e.g. in netstat output), output 
> counters are 2x more than expected.
> 
> EXAMPLE, ftp transfer of 1 GiB file:
> 
> ftp> put file /dev/null
> local: file remote: /dev/null
> 229 Entering Extended Passive Mode (|||57978|)
> 150 Opening BINARY mode data connection for '/dev/null'.
> 100% |***********************************|   953 MiB  390.43 MiB/s    
> 00:00 ETA
> 226 Transfer complete.
> 1000000000 bytes sent in 00:02 (390.13 MiB/s)
> 
> Netstat on receiving side, counters are correct (for input):
> 
> lexa@home-gw:/home/lexa# netstat -I ib1 5
>              input          (ib1)           output
>     packets  errs idrops      bytes    packets  errs      bytes colls
>           0     0     0          0          0     0          0     0
>       13955     0     0  222688126       9027     0    1192796     0
>       48921     0     0  780832960      32129     0    4240596     0
>           0     0     0          0          0     0         80     0
> 
> Sum of bytes (input) is 1003521086, as expected.
> 
> Netstat on sending size, output is 2x more:
> 
> lexa@new-gw:/home/lexa# netstat -I ib0 5
>              input          (ib0)           output
>     packets  errs idrops      bytes    packets  errs      bytes colls
>           1     0     0        100          0     0          0     0
>       41162     0     0    2305210      62878     0 2008325984     0
>           1     0     0        100          0     0          0     0
> 
> It looks like packet count is correct (13955+48921=62876, two packets 
> missed somewhere), while byte count is exact 2x more.
Yes, this is a bug.  if_obytes already gets incremented in IFQ_HANDOFF(), so 
the IB code doesn't need to do it again.  Try the patch at 
www.freebsd.org/~jhb/patches/ipoib_obytes.patch

I can't speak to the MTU issue though.

-- 
John Baldwin