xen discuss - Nov 2007 - xnbu_cksum_from_peer(), bug with NETTXF_data_validated?

In usr/src/uts/common/xen/io/xnbu.c, function xnbu_cksum_from_peer()
http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/uts/common/xen/io/xnbu.c#110
we have this:

        if ((flags & NETTXF_csum_blank) != 0) {
                /*
                 * The checksum is blank.  We must fill it in here.
                 */
                mp = xnb_process_cksum_flags(xnbp, mp, 0);

                /*
                 * Because we calculated the checksum ourselves we
                 * know that it must be good, so we assert this.
                 */
                flags &= NETTXF_data_validated;
        }

        if ((flags & NETTXF_data_validated) != 0) {
                /*
                 * The checksum is asserted valid.
                 *
         .....



"flags &= NETTXF_data_validated;" looks like a bug,
and doesn''t do what the comment says.

Shouldn''t we have "flags |= NETTXF_data_validated;" here ?
 
 
This message posted from opensolaris.org

On 14 Nov 2007, at 3:34pm, Jürgen Keil wrote:
> Shouldn''t we have "flags |= NETTXF_data_validated;" here
?
Yes. 6629832. Thanks!

dme.
-- 
David Edmondson, Solaris Engineering, http://dme.org

> On 14 Nov 2007, at 3:34pm, Jürgen Keil wrote:
> > Shouldn''t we have "flags |= NETTXF_data_validated;"
here ?
> 
> Yes. 6629832. Thanks!
Btw. an Ubuntu 7.10 x86_64 dom0, with Xen 3.1.0 hypervisor and 
"Intel Corporation 82801I (ICH9 Family) Gigabit Ethernet Controller"
running an OpenSolaris snv_75a pv domU is suffering from the udp/tcp 
checksum failures, mentioned under "Networking failures" on this
page: 

  
<URL:http://www.opensolaris.org/os/community/xen/docs/linux-dom0/#Networking_fai
lures>


I had to use "set xnf:xnf_cksum_offload = 0" in /etc/system
to get correct checksums.

And so far messing with "ethtool -K ... tx off" in the linux dom0
didn''t help...

On 14 Nov 2007, at 7:15pm, Juergen Keil wrote:
> Btw. an Ubuntu 7.10 x86_64 dom0, with Xen 3.1.0 hypervisor and
> "Intel Corporation 82801I (ICH9 Family) Gigabit Ethernet
Controller"
> running an OpenSolaris snv_75a pv domU is suffering from the udp/tcp
> checksum failures, mentioned under "Networking failures" on this
> page:
>
>
> <URL:http://www.opensolaris.org/os/community/xen/docs/linux-dom0/ 
> #Networking_fai
> lures>
>
>
> I had to use "set xnf:xnf_cksum_offload = 0" in /etc/system
> to get correct checksums.
>
> And so far messing with "ethtool -K ... tx off" in the linux dom0
> didn''t help...
I spent a couple of days looking at this last week, but I couldn''t  
see anything wrong.

The Linux dom0 is calculating a checksum and inserting it into the  
right place, but using neither software or hardware checksum in dom0  
results in a valid checksum. My best guess at the moment is that the  
length of the packet is wrong - the Linux stack assumes that the  
buffer it''s given is exactly the right size for the packet and  
calculates the checksum from the relevant offset to the end. If, say,  
we gave it a buffer that is a couple of bytes longer than the packet,  
it would cause problems.

Finding out more requires that I rebuild a Linux kernel for my test  
machine, which is a painful exercise for the un-familiar (i.e. me).  
(Sigh - spoilt by dtrace and kmdb...)

dme.
-- 
David Edmondson, Solaris Engineering, http://dme.org

David Edmondson wrote:
> On 14 Nov 2007, at 7:15pm, Juergen Keil wrote:
> > Btw. an Ubuntu 7.10 x86_64 dom0, with Xen 3.1.0 hypervisor and
> > "Intel Corporation 82801I (ICH9 Family) Gigabit Ethernet
Controller"
> > running an OpenSolaris snv_75a pv domU is suffering from the udp/tcp
> > checksum failures, mentioned under "Networking failures" on
this
> > page:
> >
> >
> > <URL:http://www.opensolaris.org/os/community/xen/docs/linux-dom0/ 
> > #Networking_fai
> > lures>
> >
> >
> > I had to use "set xnf:xnf_cksum_offload = 0" in /etc/system
> > to get correct checksums.
> >
> > And so far messing with "ethtool -K ... tx off" in the linux
dom0
> > didn''t help...
> 
> I spent a couple of days looking at this last week, but I couldn''t
> see anything wrong.
> 
> The Linux dom0 is calculating a checksum and inserting it into the  
> right place, but using neither software or hardware checksum in dom0  
> results in a valid checksum. My best guess at the moment is that the  
> length of the packet is wrong - the Linux stack assumes that the  
> buffer it''s given is exactly the right size for the packet and  
> calculates the checksum from the relevant offset to the end. If, say,  
> we gave it a buffer that is a couple of bytes longer than the packet,  
> it would cause problems.
No, I think the linux kernel is checksumming the correct part of the
buffer.


But a closer look at the bad packet shows that the udp checksum added by
the linux dom0 kernel is missing the 6 words from the udp/tcp "pseudo
header"
(source ip, dest ip, protocol and packet length).


E.g. this is an udp packet as seen by linux dom0 on the vif7.0 interface
(where xen domain 7 is a snv75a domU running with xnf_cksum_offload = 1):

# tcpdump -i vif7.0 -xx -vv -s 1600 -n ether host 0:4a:48:12:3e:da
tcpdump: WARNING: vif7.0: no IPv4 address assigned
tcpdump: listening on vif7.0, link-type EN10MB (Ethernet), capture size 1600
bytes
16:49:48.125908 IP (tos 0x0, ttl 255, id 28516, offset 0, flags [DF], proto UDP
(17), length 92) 172.20.1.190.33155 > 172.20.0.17.111: [no cksum] UDP, length
64
        0x0000:  0050 5635 0ddb 004a 4812 3eda 0800 4500
        0x0010:  005c 6f64 4000 ff11 b234 ac14 01be ac14
        0x0020:  0011 8183 006f 0048 0000 4746 738b 0000
        0x0030:  0000 0000 0002 0001 86a0 0000 0003 0000
        0x0040:  0003 0000 0000 0000 0000 0000 0000 0000
        0x0050:  0000 0001 86a4 0000 0002 0000 0003 7564
        0x0060:  7000 0000 0000 0000 0000


OpenSolaris is transmitting the udp packet with an udp checksum of
0x0000 (word at offset 0x0028, in the tcpdump above).


On the receiving box we see a bad udp checksum (0xd03a), added by the
linux dom0 network driver:

# /opt/sfw/sbin/tcpdump -xx -vv -s 1600 -n ether host 0:4a:48:12:3e:da

16:49:37.164112 IP (tos 0x0, ttl 255, id 28516, offset 0, flags [DF], length:
92) 172.20.1.190.33155 > 172.20.0.17.111: [bad udp cksum aea5!] UDP, length:
64
        0x0000:  0050 5635 0ddb 004a 4812 3eda 0800 4500  .PV5...JH.>...E.
        0x0010:  005c 6f64 4000 ff11 b234 ac14 01be ac14  .\od@....4......
        0x0020:  0011 8183 006f 0048 d03a 4746 738b 0000  .....o.H.:GFs...
        0x0030:  0000 0000 0002 0001 86a0 0000 0003 0000  ................
        0x0040:  0003 0000 0000 0000 0000 0000 0000 0000  ................
        0x0050:  0000 0001 86a4 0000 0002 0000 0003 7564  ..............ud
        0x0060:  7000 0000 0000 0000 0000                 p.........


I get the udp checksum of 0xd03a by computing the ip checksum for
the udp header and udp data bytes (starting from offset 0x0022 to the 
end of the packet, in the first tcpdump), and *omitting the udp pseudo
header* part:
> 
8183+006f+0048+0000+4746+738b+0000+0000+0000+0002+0001+86a0+0000+0003+0000+0003+0000+0000+0000+0000+0000+0000+0000+0000+0001+86a4+0000+0002+0000+0003+7564+7000+0000+0000+000
0+0000=X
                32fc2           
> (3+2fc2)^ffff=X
                d03a     
                
                


I''m not sure, but it seems that a linux domU is transmitting udp/tcp
packets with the checksum field filled with the internet checksum for
the pseudo header fields (instead of 0), in the hw checksum offload case.


linux dom0 computes the full checksum by just looking at the udp/tcp header
and data part, not caring any more about the pseudo header.  For the linux
dom0 xen network backend driver it seems to be important that the
protocol checksum field is prefilled with the sum for the pseudo header.

    ==========================================    
Here''s an udp packet from an opensuse 10.3 domU, as seen by a opensuse
10.3
dom0:

tcpdump: listening on vif5.0, link-type EN10MB (Ethernet), capture size 1600 byt
es
16:52:06.741190 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17),
length 56) 192.168.2.40.32769 > 192.168.2.21.53: [bad udp cksum eb33!] 9350+
A? foobar.com.
(28)
        0x0000:  00a0 d1d5 bcb4 0008 c7d2 8263 0800 4500
        0x0010:  0038 0000 4000 4011 b527 c0a8 0228 c0a8
        0x0020:  0215 8001 0035 0024 85c3 2486 0100 0001
        0x0030:  0000 0000 0000 0666 6f6f 6261 7203 636f
        0x0040:  6d00 0001 0001


Note the udp checksum 0x85c3 at offset 0x0028, which can be computed from
the source ip 0xc0a80228, dest ip 0xc0a80215, protocol 0x11 (udp) and
length 0x24:

c0a8+0228+c0a8+0215+11+24=X
185c2

1+85c2=X
85c3



Now the linux dom0 kernel computes the full checksum starting from
offset 0x0022:
>
8001+0035+0024+85c3+2486+0100+0001+0000+0000+0000+0666+6f6f+6261+7203+636f+6d00+0001+0001=X
                3464e           
> (3+464e)^ffff=X
                b9ae        
                

And that''s the correct udp checksum, as seen by the receiving system:

16:52:06.741436 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17),
length 56) 192.168.2.40.32769 > 192.168.2.21.53: [udp sum ok] 9350+ A?
foobar.com. (28)
        0x0000:  00a0 d1d5 bcb4 0008 c7d2 8263 0800 4500
        0x0010:  0038 0000 4000 4011 b527 c0a8 0228 c0a8
        0x0020:  0215 8001 0035 0024 b9ae 2486 0100 0001
        0x0030:  0000 0000 0000 0666 6f6f 6261 7203 636f
        0x0040:  6d00 0001 0001