Xen devel - Mar 2011 - ip/udp checksum offload from minios guest

I''m just adding checksum offload support into a custom guest, and
wanted to clarify a few things.

In netfront, I can mark outgoing frames with:
- NETTXF_csum_blank
- NETTXF_data_validated

These refer to UDP or TCP checksums, and not the IP checksum, right? Linux seems
to never offload IP header checksumming, so it must be offloading the UDP/TCP
calculation and then adjusting the IPv4 checksum based on that calculation.

For outgoing UDP, my guest is setting the checksum to 0 (as it''s
optional in the protocol), and calculating the full IPv4 checksum in software,
and all works (slowly).  However, setting NETTXF_csum_blank in the outgoing
frame and leaving the IPv4 checksum at 0 doesn''t seem to result in any
adjustment by netback, and the packet gets dropped.

How am I supposed to entirely offload the IPv4 checksum calculation for UDP? 
Setting the flag unconditionally for non-TCP/UDP traffic (e.g. ARP/ICMP) results
in lots of dropped frames, so I am only setting it in the outgoing UDP frames.

-anil


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Tue, 2011-03-29 at 17:17 +0100, Anil Madhavapeddy
wrote:
> I''m just adding checksum offload support into a custom guest, and
wanted to clarify a few things.
> 
> In netfront, I can mark outgoing frames with:
> - NETTXF_csum_blank
> - NETTXF_data_validated
> 
> These refer to UDP or TCP checksums, and not the IP checksum, right?
> Linux seems to never offload IP header checksumming, so it must be
> offloading the UDP/TCP calculation and then adjusting the IPv4
> checksum based on that calculation.
> 
> For outgoing UDP, my guest is setting the checksum to 0 (as it''s
> optional in the protocol), and calculating the full IPv4 checksum in
> software, and all works (slowly).  However, setting NETTXF_csum_blank
> in the outgoing frame and leaving the IPv4 checksum at 0 doesn''t
seem
> to result in any adjustment by netback, and the packet gets dropped. 
I think you need to set the checksum to the psuedo-header checksum
rather than 0 since that is what csum_blank means (such a well named
flag!)...

It''s not clear why anything would drop a UDP frame with checksum==0
since, as you say, it is optional. My guess is that since the
NETTXF_csum_blank and data_validated turn into an skb->ip_summed =SKB_PARTIAL
on the backend side this causes the Linux network stack to
drop it because that statement conflicts with the checksum being 0.
> How am I supposed to entirely offload the IPv4 checksum calculation
> for UDP?  Setting the flag unconditionally for non-TCP/UDP traffic
> (e.g. ARP/ICMP) results in lots of dropped frames, so I am only
> setting it in the outgoing UDP frames.
Which is the right thing to do.

Ian.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 29 Mar 2011, at 12:37, Ian Campbell wrote:
> On Tue, 2011-03-29 at 17:17 +0100, Anil Madhavapeddy wrote:
>> I''m just adding checksum offload support into a custom guest,
and wanted to clarify a few things.
>> 
>> In netfront, I can mark outgoing frames with:
>> - NETTXF_csum_blank
>> - NETTXF_data_validated
>> 
>> These refer to UDP or TCP checksums, and not the IP checksum, right?
>> Linux seems to never offload IP header checksumming, so it must be
>> offloading the UDP/TCP calculation and then adjusting the IPv4
>> checksum based on that calculation.
>> 
>> For outgoing UDP, my guest is setting the checksum to 0 (as
it''s
>> optional in the protocol), and calculating the full IPv4 checksum in
>> software, and all works (slowly).  However, setting NETTXF_csum_blank
>> in the outgoing frame and leaving the IPv4 checksum at 0
doesn''t seem
>> to result in any adjustment by netback, and the packet gets dropped. 
> 
> I think you need to set the checksum to the psuedo-header checksum
> rather than 0 since that is what csum_blank means (such a well named
> flag!)...
> 
> It''s not clear why anything would drop a UDP frame with
checksum==0
> since, as you say, it is optional. My guess is that since the
> NETTXF_csum_blank and data_validated turn into an skb->ip_summed =>
SKB_PARTIAL on the backend side this causes the Linux network stack to
> drop it because that statement conflicts with the checksum being 0.
Right, but I''m setting the IPv4 checksum to 0 too, since I want to
offload the whole lot and never calculate a body checksum in the guest.  But if
the UDP checksum is set to 0, then the backend can''t just adjust it to
obtain the IPv4 checksum and has to iterate over the packet body at some stage.

I''ve tried setting the UDP checksum to 0, the pseudoheader, and the
full checksum, and the IPv4 checksum is never set by the backend in any of these
cases.  If I set the IPv4 checksum in software to the correct value, then
packets go through, but the UDP checksums are never altered.

Guess it''s time to start slapping printfs all over the dom0 kernel to
see what''s going on unless you know what I "should" be
setting both the IPv4/UDP checksums to with NETTXF_csum_blank...

Anil


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Tue, 2011-03-29 at 18:15 +0100, Anil Madhavapeddy
wrote:
> On 29 Mar 2011, at 12:37, Ian Campbell wrote:
> 
> > On Tue, 2011-03-29 at 17:17 +0100, Anil Madhavapeddy wrote:
> >> I''m just adding checksum offload support into a custom
guest, and wanted to clarify a few things.
> >> 
> >> In netfront, I can mark outgoing frames with:
> >> - NETTXF_csum_blank
> >> - NETTXF_data_validated
> >> 
> >> These refer to UDP or TCP checksums, and not the IP checksum,
right?
> >> Linux seems to never offload IP header checksumming, so it must be
> >> offloading the UDP/TCP calculation and then adjusting the IPv4
> >> checksum based on that calculation.
> >> 
> >> For outgoing UDP, my guest is setting the checksum to 0 (as
it''s
> >> optional in the protocol), and calculating the full IPv4 checksum
in
> >> software, and all works (slowly).  However, setting
NETTXF_csum_blank
> >> in the outgoing frame and leaving the IPv4 checksum at 0
doesn''t seem
> >> to result in any adjustment by netback, and the packet gets
dropped.
> > 
> > I think you need to set the checksum to the psuedo-header checksum
> > rather than 0 since that is what csum_blank means (such a well named
> > flag!)...
> > 
> > It''s not clear why anything would drop a UDP frame with
checksum==0
> > since, as you say, it is optional. My guess is that since the
> > NETTXF_csum_blank and data_validated turn into an skb->ip_summed
=> > SKB_PARTIAL on the backend side this causes the Linux network stack
to
> > drop it because that statement conflicts with the checksum being 0.
> 
> Right, but I''m setting the IPv4 checksum to 0 too, since I want to
> offload the whole lot and never calculate a body checksum in the
> guest.
Maybe I''m getting confused but the IPv4 checksum only covers the
headers, doesn''t it? I don''t think you can offload those, at
least not
using the current PV protocol. So you need to include a valid IP header
checksum but that doesn''t require summing over all the data, just the
headers.
> But if the UDP checksum is set to 0, then the backend can''t just
> adjust it to obtain the IPv4 checksum and has to iterate over the
> packet body at some stage.
I don''t think it needs to obtain the IPv4 checksum to do UDP checksum
if
the UDP checksum is initially set to the partial(/pseudo header)
checksum -- it''s already incorporated as necessary.
> I''ve tried setting the UDP checksum to 0, the pseudoheader, and
the
> full checksum, and the IPv4 checksum is never set by the backend in
> any of these cases.  If I set the IPv4 checksum in software to the
> correct value, then packets go through, but the UDP checksums are
> never altered.
> 
> Guess it''s time to start slapping printfs all over the dom0 kernel
to
> see what''s going on unless you know what I "should" be
setting both
> the IPv4/UDP checksums to with NETTXF_csum_blank...
The protocol is unhelpfully "defined" to be basically "what Linux
does".
i.e. whatever Linux generates in an SKB with skb->ip_summed =CHECKSUM_PARTIAL
is what you need to generate.

Ian.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Maybe there''s some confusion of terms here... There are 2 checksums:
the IPv4 header and UDP checksum.

The IPv4 header checksum must *always* be calculated by the frontend.

If NETTXF_csum_blank is set (implying NETTXF_data_validated must also be set)
then the UDP checksum must be set to cover the UDP pseudo-header since the SKB
will be marked CSUM_PARTIAL and thus any h/w driver it is presented to will
expect the pseudo-header checksum to be valid. If the SKB is presented to
another guest then CSUM_PARTIAL will translated into
NETRXF_csum_blank|NETRXF_data_validated and the frontend is at liberty to
present it up the stack as being checksum-valid without anything in the datapath
having had to walk over the payload to actually calculate the value of that
checksum.

  Paul
> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
> bounces@lists.xensource.com] On Behalf Of Anil Madhavapeddy
> Sent: 29 March 2011 18:16
> To: Ian Campbell
> Cc: xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] ip/udp checksum offload from minios guest
> 
> On 29 Mar 2011, at 12:37, Ian Campbell wrote:
> 
> > On Tue, 2011-03-29 at 17:17 +0100, Anil Madhavapeddy wrote:
> >> I''m just adding checksum offload support into a custom
guest, and
> wanted to clarify a few things.
> >>
> >> In netfront, I can mark outgoing frames with:
> >> - NETTXF_csum_blank
> >> - NETTXF_data_validated
> >>
> >> These refer to UDP or TCP checksums, and not the IP checksum,
> right?
> >> Linux seems to never offload IP header checksumming, so it must
> be
> >> offloading the UDP/TCP calculation and then adjusting the IPv4
> >> checksum based on that calculation.
> >>
> >> For outgoing UDP, my guest is setting the checksum to 0 (as
it''s
> >> optional in the protocol), and calculating the full IPv4 checksum
> in
> >> software, and all works (slowly).  However, setting
> NETTXF_csum_blank
> >> in the outgoing frame and leaving the IPv4 checksum at 0
doesn''t
> seem
> >> to result in any adjustment by netback, and the packet gets
> dropped.
> >
> > I think you need to set the checksum to the psuedo-header checksum
> > rather than 0 since that is what csum_blank means (such a well
> named
> > flag!)...
> >
> > It''s not clear why anything would drop a UDP frame with
> checksum==0
> > since, as you say, it is optional. My guess is that since the
> > NETTXF_csum_blank and data_validated turn into an skb->ip_summed
> => > SKB_PARTIAL on the backend side this causes the Linux network
> stack to
> > drop it because that statement conflicts with the checksum being
> 0.
> 
> Right, but I''m setting the IPv4 checksum to 0 too, since I want to
> offload the whole lot and never calculate a body checksum in the
> guest.  But if the UDP checksum is set to 0, then the backend
can''t
> just adjust it to obtain the IPv4 checksum and has to iterate over
> the packet body at some stage.
> 
> I''ve tried setting the UDP checksum to 0, the pseudoheader, and
the
> full checksum, and the IPv4 checksum is never set by the backend in
> any of these cases.  If I set the IPv4 checksum in software to the
> correct value, then packets go through, but the UDP checksums are
> never altered.
> 
> Guess it''s time to start slapping printfs all over the dom0 kernel
> to see what''s going on unless you know what I "should"
be setting
> both the IPv4/UDP checksums to with NETTXF_csum_blank...
> 
> Anil
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Thanks, this is all clear now. I was not setting NETTXF_data_validated along
with NETTXF_csum_blank in my transmit path, which was confusing the backend.

Ian: yes the IPv4 checksum is of course only over the header. I wrote the code
right, and then re-read it wrong while hacking on a flight :) The ICMP errors
confused me (since I was setting NETTXF_csum_blank for those too and the
checksum functions error out).

On 29 Mar 2011, at 15:19, Paul Durrant wrote:
> Maybe there''s some confusion of terms here... There are 2
checksums: the IPv4 header and UDP checksum.
> 
> The IPv4 header checksum must *always* be calculated by the frontend.
> 
> If NETTXF_csum_blank is set (implying NETTXF_data_validated must also be
set) then the UDP checksum must be set to cover the UDP pseudo-header since the
SKB will be marked CSUM_PARTIAL and thus any h/w driver it is presented to will
expect the pseudo-header checksum to be valid. If the SKB is presented to
another guest then CSUM_PARTIAL will translated into
NETRXF_csum_blank|NETRXF_data_validated and the frontend is at liberty to
present it up the stack as being checksum-valid without anything in the datapath
having had to walk over the payload to actually calculate the value of that
checksum.
> 
>  Paul
> 
>> -----Original Message-----
>> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
>> bounces@lists.xensource.com] On Behalf Of Anil Madhavapeddy
>> Sent: 29 March 2011 18:16
>> To: Ian Campbell
>> Cc: xen-devel@lists.xensource.com
>> Subject: Re: [Xen-devel] ip/udp checksum offload from minios guest
>> 
>> On 29 Mar 2011, at 12:37, Ian Campbell wrote:
>> 
>>> On Tue, 2011-03-29 at 17:17 +0100, Anil Madhavapeddy wrote:
>>>> I''m just adding checksum offload support into a custom
guest, and
>> wanted to clarify a few things.
>>>> 
>>>> In netfront, I can mark outgoing frames with:
>>>> - NETTXF_csum_blank
>>>> - NETTXF_data_validated
>>>> 
>>>> These refer to UDP or TCP checksums, and not the IP checksum,
>> right?
>>>> Linux seems to never offload IP header checksumming, so it must
>> be
>>>> offloading the UDP/TCP calculation and then adjusting the IPv4
>>>> checksum based on that calculation.
>>>> 
>>>> For outgoing UDP, my guest is setting the checksum to 0 (as
it''s
>>>> optional in the protocol), and calculating the full IPv4
checksum
>> in
>>>> software, and all works (slowly).  However, setting
>> NETTXF_csum_blank
>>>> in the outgoing frame and leaving the IPv4 checksum at 0
doesn''t
>> seem
>>>> to result in any adjustment by netback, and the packet gets
>> dropped.
>>> 
>>> I think you need to set the checksum to the psuedo-header checksum
>>> rather than 0 since that is what csum_blank means (such a well
>> named
>>> flag!)...
>>> 
>>> It''s not clear why anything would drop a UDP frame with
>> checksum==0
>>> since, as you say, it is optional. My guess is that since the
>>> NETTXF_csum_blank and data_validated turn into an skb->ip_summed
>> =>>> SKB_PARTIAL on the backend side this causes the Linux
network
>> stack to
>>> drop it because that statement conflicts with the checksum being
>> 0.
>> 
>> Right, but I''m setting the IPv4 checksum to 0 too, since I
want to
>> offload the whole lot and never calculate a body checksum in the
>> guest.  But if the UDP checksum is set to 0, then the backend
can''t
>> just adjust it to obtain the IPv4 checksum and has to iterate over
>> the packet body at some stage.
>> 
>> I''ve tried setting the UDP checksum to 0, the pseudoheader,
and the
>> full checksum, and the IPv4 checksum is never set by the backend in
>> any of these cases.  If I set the IPv4 checksum in software to the
>> correct value, then packets go through, but the UDP checksums are
>> never altered.
>> 
>> Guess it''s time to start slapping printfs all over the dom0
kernel
>> to see what''s going on unless you know what I
"should" be setting
>> both the IPv4/UDP checksums to with NETTXF_csum_blank...
>> 
>> Anil
>> 
>> 
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Tue, 2011-03-29 at 23:26 +0100, Anil Madhavapeddy
wrote:
> Thanks, this is all clear now. I was not setting NETTXF_data_validated
> along with NETTXF_csum_blank in my transmit path, which was confusing
> the backend.
Right, that would have ended up confusing things, I think. c_s && !d_v
still ends up a CHECKSUM_PARTIAL in relatively recent backends. I''m not
sure that has always been implemented correctly though.

In recent netback there is fixup for the case where a GSO packet sets
csum_blank but not data_validated which recalculates the partial
checksum (since Linux requires that all GSO SKBs are CHECKSUM_PARTIAL to
simplify the software GSO checksum stuff). That''s only in the GSO case
though since we can''t detect it otherwise.
> Ian: yes the IPv4 checksum is of course only over the header. I wrote
> the code right, and then re-read it wrong while hacking on a flight :)
Easily done!
> The ICMP errors confused me (since I was setting NETTXF_csum_blank for
> those too and the checksum functions error out).
Yeah, don''t do that ;-)

Ian.



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> 
> On Tue, 2011-03-29 at 23:26 +0100, Anil Madhavapeddy wrote:
> > Thanks, this is all clear now. I was not setting NETTXF_data_validated
> > along with NETTXF_csum_blank in my transmit path, which was confusing
> > the backend.
> 
> Right, that would have ended up confusing things, I think. c_s &&
!d_v
> still ends up a CHECKSUM_PARTIAL in relatively recent backends. I'm not
> sure that has always been implemented correctly though.
> 
> In recent netback there is fixup for the case where a GSO packet sets
> csum_blank but not data_validated which recalculates the partial
> checksum (since Linux requires that all GSO SKBs are CHECKSUM_PARTIAL to
> simplify the software GSO checksum stuff). That's only in the GSO case
> though since we can't detect it otherwise.
> 
> > Ian: yes the IPv4 checksum is of course only over the header. I wrote
> > the code right, and then re-read it wrong while hacking on a flight :)
> 
> Easily done!
> 
> > The ICMP errors confused me (since I was setting NETTXF_csum_blank for
> > those too and the checksum functions error out).
> 
> Yeah, don't do that ;-)
> 
Is IPv6 offload (gso and/or checksum) supported at this time?

James

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Wed, 2011-03-30 at 11:23 +0100, James Harper wrote:
> > 
> > On Tue, 2011-03-29 at 23:26 +0100, Anil Madhavapeddy wrote:
> > > Thanks, this is all clear now. I was not setting
NETTXF_data_validated
> > > along with NETTXF_csum_blank in my transmit path, which was
confusing
> > > the backend.
> > 
> > Right, that would have ended up confusing things, I think. c_s
&& !d_v
> > still ends up a CHECKSUM_PARTIAL in relatively recent backends.
I''m not
> > sure that has always been implemented correctly though.
> > 
> > In recent netback there is fixup for the case where a GSO packet sets
> > csum_blank but not data_validated which recalculates the partial
> > checksum (since Linux requires that all GSO SKBs are CHECKSUM_PARTIAL
to
> > simplify the software GSO checksum stuff). That''s only in the
GSO case
> > though since we can''t detect it otherwise.
> > 
> > > Ian: yes the IPv4 checksum is of course only over the header. I
wrote
> > > the code right, and then re-read it wrong while hacking on a
flight :)
> > 
> > Easily done!
> > 
> > > The ICMP errors confused me (since I was setting
NETTXF_csum_blank for
> > > those too and the checksum functions error out).
> > 
> > Yeah, don''t do that ;-)
> > 
> 
> Is IPv6 offload (gso and/or checksum) supported at this time?
For GSO only TCPv4 is supported.

For checksum only TCPv4 and UDPv4 are supported.

Extending netback to cover more cases would be relatively trivial, I
think. e.g. extending checksum_setup and/or netbk_set_skb_gso in
netback.c, plus doing the opposite same on the guest RX paths.

I suspect most of the complexity will probably come from the correct
negotiation of the featureset via xenstore ;-)

Ian.



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> > Is IPv6 offload (gso and/or checksum) supported at this time?
> 
> For GSO only TCPv4 is supported.
> 
> For checksum only TCPv4 and UDPv4 are supported.
> 
> Extending netback to cover more cases would be relatively trivial, I
> think. e.g. extending checksum_setup and/or netbk_set_skb_gso in
> netback.c, plus doing the opposite same on the guest RX paths.
> 
> I suspect most of the complexity will probably come from the correct
> negotiation of the featureset via xenstore ;-)
> 
Yes. Given the problems I've seen under linux with offload it would be nice
to have even more granularity than we do now. We'd definitely want to be
able to turn offload on and off for all combinations of tcp/udp/csum/gso
individually in case there is a problem somewhere.

The main problem I've seen is that Linux doesn't seem to understand that
my network card supports offload functions but not in combination with VLAN, so
only the untagged packets (eg vlan1) can be offloaded. That was back at 2.6.18
so hopefully the support is better now.

James

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Wed, 2011-03-30 at 11:37 +0100, James Harper wrote:
> > > Is IPv6 offload (gso and/or checksum) supported at this time?
> > 
> > For GSO only TCPv4 is supported.
> > 
> > For checksum only TCPv4 and UDPv4 are supported.
> > 
> > Extending netback to cover more cases would be relatively trivial, I
> > think. e.g. extending checksum_setup and/or netbk_set_skb_gso in
> > netback.c, plus doing the opposite same on the guest RX paths.
> > 
> > I suspect most of the complexity will probably come from the correct
> > negotiation of the featureset via xenstore ;-)
> > 
> 
> Yes. Given the problems I''ve seen under linux with offload it
would be
> nice to have even more granularity than we do now. We''d definitely
> want to be able to turn offload on and off for all combinations of
> tcp/udp/csum/gso individually in case there is a problem somewhere.
Absolutely. Paul Durrant made some improvements in this area not so long
ago. Also from the looks of things upstream have recently separated out
the concept of what the h/w supports from what is actually currently
configured to be enabled which should be useful going forward.
> The main problem I''ve seen is that Linux doesn''t seem to
understand
> that my network card supports offload functions but not in combination
> with VLAN, so only the untagged packets (eg vlan1) can be offloaded.
> That was back at 2.6.18 so hopefully the support is better now.
I should hope so ;-)

In Linux a VLAN device is a new net device which sits on top of the
physical device and has it''s own offload settings etc so at least there
is a route to fixing it there...

Ian



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel