In usr/src/uts/common/xen/io/xnbu.c, function xnbu_cksum_from_peer()
http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/uts/common/xen/io/xnbu.c#110
we have this:
if ((flags & NETTXF_csum_blank) != 0) {
/*
* The checksum is blank. We must fill it in here.
*/
mp = xnb_process_cksum_flags(xnbp, mp, 0);
/*
* Because we calculated the checksum ourselves we
* know that it must be good, so we assert this.
*/
flags &= NETTXF_data_validated;
}
if ((flags & NETTXF_data_validated) != 0) {
/*
* The checksum is asserted valid.
*
.....
"flags &= NETTXF_data_validated;" looks like a bug,
and doesn''t do what the comment says.
Shouldn''t we have "flags |= NETTXF_data_validated;" here ?
This message posted from opensolaris.org
David Edmondson
2007-Nov-14 18:50 UTC
Re: xnbu_cksum_from_peer(), bug with NETTXF_data_validated?
On 14 Nov 2007, at 3:34pm, Jürgen Keil wrote:> Shouldn''t we have "flags |= NETTXF_data_validated;" here ?Yes. 6629832. Thanks! dme. -- David Edmondson, Solaris Engineering, http://dme.org
Juergen Keil
2007-Nov-14 19:15 UTC
Re: xnbu_cksum_from_peer(), bug with NETTXF_data_validated?
> On 14 Nov 2007, at 3:34pm, Jürgen Keil wrote: > > Shouldn''t we have "flags |= NETTXF_data_validated;" here ? > > Yes. 6629832. Thanks!Btw. an Ubuntu 7.10 x86_64 dom0, with Xen 3.1.0 hypervisor and "Intel Corporation 82801I (ICH9 Family) Gigabit Ethernet Controller" running an OpenSolaris snv_75a pv domU is suffering from the udp/tcp checksum failures, mentioned under "Networking failures" on this page: <URL:http://www.opensolaris.org/os/community/xen/docs/linux-dom0/#Networking_fai lures> I had to use "set xnf:xnf_cksum_offload = 0" in /etc/system to get correct checksums. And so far messing with "ethtool -K ... tx off" in the linux dom0 didn''t help...
David Edmondson
2007-Nov-14 19:36 UTC
Re: xnbu_cksum_from_peer(), bug with NETTXF_data_validated?
On 14 Nov 2007, at 7:15pm, Juergen Keil wrote:> Btw. an Ubuntu 7.10 x86_64 dom0, with Xen 3.1.0 hypervisor and > "Intel Corporation 82801I (ICH9 Family) Gigabit Ethernet Controller" > running an OpenSolaris snv_75a pv domU is suffering from the udp/tcp > checksum failures, mentioned under "Networking failures" on this > page: > > > <URL:http://www.opensolaris.org/os/community/xen/docs/linux-dom0/ > #Networking_fai > lures> > > > I had to use "set xnf:xnf_cksum_offload = 0" in /etc/system > to get correct checksums. > > And so far messing with "ethtool -K ... tx off" in the linux dom0 > didn''t help...I spent a couple of days looking at this last week, but I couldn''t see anything wrong. The Linux dom0 is calculating a checksum and inserting it into the right place, but using neither software or hardware checksum in dom0 results in a valid checksum. My best guess at the moment is that the length of the packet is wrong - the Linux stack assumes that the buffer it''s given is exactly the right size for the packet and calculates the checksum from the relevant offset to the end. If, say, we gave it a buffer that is a couple of bytes longer than the packet, it would cause problems. Finding out more requires that I rebuild a Linux kernel for my test machine, which is a painful exercise for the un-familiar (i.e. me). (Sigh - spoilt by dtrace and kmdb...) dme. -- David Edmondson, Solaris Engineering, http://dme.org
Juergen Keil
2007-Nov-19 16:43 UTC
Re: xnbu_cksum_from_peer(), bug with NETTXF_data_validated?
David Edmondson wrote:> On 14 Nov 2007, at 7:15pm, Juergen Keil wrote: > > Btw. an Ubuntu 7.10 x86_64 dom0, with Xen 3.1.0 hypervisor and > > "Intel Corporation 82801I (ICH9 Family) Gigabit Ethernet Controller" > > running an OpenSolaris snv_75a pv domU is suffering from the udp/tcp > > checksum failures, mentioned under "Networking failures" on this > > page: > > > > > > <URL:http://www.opensolaris.org/os/community/xen/docs/linux-dom0/ > > #Networking_fai > > lures> > > > > > > I had to use "set xnf:xnf_cksum_offload = 0" in /etc/system > > to get correct checksums. > > > > And so far messing with "ethtool -K ... tx off" in the linux dom0 > > didn''t help... > > I spent a couple of days looking at this last week, but I couldn''t > see anything wrong. > > The Linux dom0 is calculating a checksum and inserting it into the > right place, but using neither software or hardware checksum in dom0 > results in a valid checksum. My best guess at the moment is that the > length of the packet is wrong - the Linux stack assumes that the > buffer it''s given is exactly the right size for the packet and > calculates the checksum from the relevant offset to the end. If, say, > we gave it a buffer that is a couple of bytes longer than the packet, > it would cause problems.No, I think the linux kernel is checksumming the correct part of the buffer. But a closer look at the bad packet shows that the udp checksum added by the linux dom0 kernel is missing the 6 words from the udp/tcp "pseudo header" (source ip, dest ip, protocol and packet length). E.g. this is an udp packet as seen by linux dom0 on the vif7.0 interface (where xen domain 7 is a snv75a domU running with xnf_cksum_offload = 1): # tcpdump -i vif7.0 -xx -vv -s 1600 -n ether host 0:4a:48:12:3e:da tcpdump: WARNING: vif7.0: no IPv4 address assigned tcpdump: listening on vif7.0, link-type EN10MB (Ethernet), capture size 1600 bytes 16:49:48.125908 IP (tos 0x0, ttl 255, id 28516, offset 0, flags [DF], proto UDP (17), length 92) 172.20.1.190.33155 > 172.20.0.17.111: [no cksum] UDP, length 64 0x0000: 0050 5635 0ddb 004a 4812 3eda 0800 4500 0x0010: 005c 6f64 4000 ff11 b234 ac14 01be ac14 0x0020: 0011 8183 006f 0048 0000 4746 738b 0000 0x0030: 0000 0000 0002 0001 86a0 0000 0003 0000 0x0040: 0003 0000 0000 0000 0000 0000 0000 0000 0x0050: 0000 0001 86a4 0000 0002 0000 0003 7564 0x0060: 7000 0000 0000 0000 0000 OpenSolaris is transmitting the udp packet with an udp checksum of 0x0000 (word at offset 0x0028, in the tcpdump above). On the receiving box we see a bad udp checksum (0xd03a), added by the linux dom0 network driver: # /opt/sfw/sbin/tcpdump -xx -vv -s 1600 -n ether host 0:4a:48:12:3e:da 16:49:37.164112 IP (tos 0x0, ttl 255, id 28516, offset 0, flags [DF], length: 92) 172.20.1.190.33155 > 172.20.0.17.111: [bad udp cksum aea5!] UDP, length: 64 0x0000: 0050 5635 0ddb 004a 4812 3eda 0800 4500 .PV5...JH.>...E. 0x0010: 005c 6f64 4000 ff11 b234 ac14 01be ac14 .\od@....4...... 0x0020: 0011 8183 006f 0048 d03a 4746 738b 0000 .....o.H.:GFs... 0x0030: 0000 0000 0002 0001 86a0 0000 0003 0000 ................ 0x0040: 0003 0000 0000 0000 0000 0000 0000 0000 ................ 0x0050: 0000 0001 86a4 0000 0002 0000 0003 7564 ..............ud 0x0060: 7000 0000 0000 0000 0000 p......... I get the udp checksum of 0xd03a by computing the ip checksum for the udp header and udp data bytes (starting from offset 0x0022 to the end of the packet, in the first tcpdump), and *omitting the udp pseudo header* part:>8183+006f+0048+0000+4746+738b+0000+0000+0000+0002+0001+86a0+0000+0003+0000+0003+0000+0000+0000+0000+0000+0000+0000+0000+0001+86a4+0000+0002+0000+0003+7564+7000+0000+0000+000 0+0000=X 32fc2> (3+2fc2)^ffff=Xd03a I''m not sure, but it seems that a linux domU is transmitting udp/tcp packets with the checksum field filled with the internet checksum for the pseudo header fields (instead of 0), in the hw checksum offload case. linux dom0 computes the full checksum by just looking at the udp/tcp header and data part, not caring any more about the pseudo header. For the linux dom0 xen network backend driver it seems to be important that the protocol checksum field is prefilled with the sum for the pseudo header. ========================================== Here''s an udp packet from an opensuse 10.3 domU, as seen by a opensuse 10.3 dom0: tcpdump: listening on vif5.0, link-type EN10MB (Ethernet), capture size 1600 byt es 16:52:06.741190 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 56) 192.168.2.40.32769 > 192.168.2.21.53: [bad udp cksum eb33!] 9350+ A? foobar.com. (28) 0x0000: 00a0 d1d5 bcb4 0008 c7d2 8263 0800 4500 0x0010: 0038 0000 4000 4011 b527 c0a8 0228 c0a8 0x0020: 0215 8001 0035 0024 85c3 2486 0100 0001 0x0030: 0000 0000 0000 0666 6f6f 6261 7203 636f 0x0040: 6d00 0001 0001 Note the udp checksum 0x85c3 at offset 0x0028, which can be computed from the source ip 0xc0a80228, dest ip 0xc0a80215, protocol 0x11 (udp) and length 0x24: c0a8+0228+c0a8+0215+11+24=X 185c2 1+85c2=X 85c3 Now the linux dom0 kernel computes the full checksum starting from offset 0x0022:> 8001+0035+0024+85c3+2486+0100+0001+0000+0000+0000+0666+6f6f+6261+7203+636f+6d00+0001+0001=X3464e> (3+464e)^ffff=Xb9ae And that''s the correct udp checksum, as seen by the receiving system: 16:52:06.741436 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 56) 192.168.2.40.32769 > 192.168.2.21.53: [udp sum ok] 9350+ A? foobar.com. (28) 0x0000: 00a0 d1d5 bcb4 0008 c7d2 8263 0800 4500 0x0010: 0038 0000 4000 4011 b527 c0a8 0228 c0a8 0x0020: 0215 8001 0035 0024 b9ae 2486 0100 0001 0x0030: 0000 0000 0000 0666 6f6f 6261 7203 636f 0x0040: 6d00 0001 0001