On Tue, Dec 03, 2013 at 01:00:23PM +0000, Balraj Singh wrote:> Hi, > > I''m working on verifying TCP checksums on incoming packets in Mirage, but > I''ve run into a bit of a problem. > > If TCP checksum offload is turned on on a virtual interface (this is the > default), and if the TCP connection is local to the machine, it looks like > Xen does not calculate the checksum at all. This may be valid because Xen > may be providing a stronger guarantee, but it means that incoming packets > don''t have a valid checksum in the header. This then means that in Mirage > we can''t just have checksum verification turned on all the time. This > would have been the safe fall back option and detecting that checksum > offload is on, and then not duplicating the verification in Mirage would > have been an optimisation. But it looks like this is not an option. Now I > need to know for every incoming packet whether checksum verification should > be done or not. It should ideally be for every packet since chksum offload > can be turned off and on on the VIF and existing tcp connections should > continue. If not every packet, I need to get a notification or efficiently > detect right away that the setting is changed on the VIF.This is a question that seems to keep coming up even for Linux and Windows, as the combination of local<->local VMs vs local<->off-host and the checksum offload is quite confusing. CCing xen-devel: is the appropriate behaviour for a guest VM that wants to use checksum offloading in all situations documented anywhere? -- Anil Madhavapeddy http://anil.recoil.org
On Thu, 2013-12-05 at 11:29 +0000, Anil Madhavapeddy wrote:> On Tue, Dec 03, 2013 at 01:00:23PM +0000, Balraj Singh wrote: > > Hi, > > > > I''m working on verifying TCP checksums on incoming packets in Mirage, but > > I''ve run into a bit of a problem. > > > > If TCP checksum offload is turned on on a virtual interface (this is the > > default), and if the TCP connection is local to the machine, it looks like > > Xen does not calculate the checksum at all. This may be valid because Xen > > may be providing a stronger guarantee, but it means that incoming packets > > don''t have a valid checksum in the header. This then means that in Mirage > > we can''t just have checksum verification turned on all the time. This > > would have been the safe fall back option and detecting that checksum > > offload is on, and then not duplicating the verification in Mirage would > > have been an optimisation. But it looks like this is not an option. Now I > > need to know for every incoming packet whether checksum verification should > > be done or not. It should ideally be for every packet since chksum offload > > can be turned off and on on the VIF and existing tcp connections should > > continue. If not every packet, I need to get a notification or efficiently > > detect right away that the setting is changed on the VIF. > > This is a question that seems to keep coming up even for Linux and > Windows, as the combination of local<->local VMs vs local<->off-host and > the checksum offload is quite confusing. > > CCing xen-devel: is the appropriate behaviour for a guest VM that wants to > use checksum offloading in all situations documented anywhere?I don''t understand the question/concern. If you have enabled checksum offload then of course you don''t recalculate the checksum, that''s the whole point of offloading it. Ian.
On 5 Dec 2013, at 11:39, Ian Campbell wrote:> On Thu, 2013-12-05 at 11:29 +0000, Anil Madhavapeddy wrote: >> On Tue, Dec 03, 2013 at 01:00:23PM +0000, Balraj Singh wrote: >>> Hi, >>> >>> I''m working on verifying TCP checksums on incoming packets in Mirage, but >>> I''ve run into a bit of a problem. >>> >>> If TCP checksum offload is turned on on a virtual interface (this is the >>> default), and if the TCP connection is local to the machine, it looks like >>> Xen does not calculate the checksum at all. This may be valid because Xen >>> may be providing a stronger guarantee, but it means that incoming packets >>> don''t have a valid checksum in the header. This then means that in Mirage >>> we can''t just have checksum verification turned on all the time. This >>> would have been the safe fall back option and detecting that checksum >>> offload is on, and then not duplicating the verification in Mirage would >>> have been an optimisation. But it looks like this is not an option. Now I >>> need to know for every incoming packet whether checksum verification should >>> be done or not. It should ideally be for every packet since chksum offload >>> can be turned off and on on the VIF and existing tcp connections should >>> continue. If not every packet, I need to get a notification or efficiently >>> detect right away that the setting is changed on the VIF. >> >> This is a question that seems to keep coming up even for Linux and >> Windows, as the combination of local<->local VMs vs local<->off-host and >> the checksum offload is quite confusing. >> >> CCing xen-devel: is the appropriate behaviour for a guest VM that wants to >> use checksum offloading in all situations documented anywhere? > > I don''t understand the question/concern. If you have enabled checksum > offload then of course you don''t recalculate the checksum, that''s the > whole point of offloading it.i think balraj''s question arises because the status of checksum offload can change mid-tcp-flow. how does he know whether it''s on or off for a given packet? -- Cheers, R. This message and any attachment are intended solely for the addressee and may contain confidential information. If you have received this message in error, please send it back to me, and immediately delete it. Please do not use, copy or disclose the information contained in this message or in any attachment. Any views or opinions expressed by the author of this email do not necessarily reflect the views of the University of Nottingham. This message has been checked for viruses but the contents of an attachment may still contain software viruses which could damage your computer system, you are advised to perform your own checks. Email communications with the University of Nottingham may be monitored as permitted by UK legislation.
> -----Original Message----- > From: xen-devel-bounces@lists.xen.org [mailto:xen-devel- > bounces@lists.xen.org] On Behalf Of Ian Campbell > Sent: 05 December 2013 11:39 > To: Anil Madhavapeddy > Cc: xen-devel@lists.xenproject.org; Balraj Singh; Mirage List > Subject: Re: [Xen-devel] Question about TCP checksum offload in Xen > > On Thu, 2013-12-05 at 11:29 +0000, Anil Madhavapeddy wrote: > > On Tue, Dec 03, 2013 at 01:00:23PM +0000, Balraj Singh wrote: > > > Hi, > > > > > > I''m working on verifying TCP checksums on incoming packets in Mirage, > but > > > I''ve run into a bit of a problem. > > > > > > If TCP checksum offload is turned on on a virtual interface (this is the > > > default), and if the TCP connection is local to the machine, it looks like > > > Xen does not calculate the checksum at all. This may be valid because > Xen > > > may be providing a stronger guarantee, but it means that incoming > packets > > > don''t have a valid checksum in the header. This then means that in > Mirage > > > we can''t just have checksum verification turned on all the time. This > > > would have been the safe fall back option and detecting that checksum > > > offload is on, and then not duplicating the verification in Mirage would > > > have been an optimisation. But it looks like this is not an option. Now I > > > need to know for every incoming packet whether checksum verification > should > > > be done or not. It should ideally be for every packet since chksum > offload > > > can be turned off and on on the VIF and existing tcp connections should > > > continue. If not every packet, I need to get a notification or efficiently > > > detect right away that the setting is changed on the VIF. > > > > This is a question that seems to keep coming up even for Linux and > > Windows, as the combination of local<->local VMs vs local<->off-host and > > the checksum offload is quite confusing. > > > > CCing xen-devel: is the appropriate behaviour for a guest VM that wants to > > use checksum offloading in all situations documented anywhere? > > I don''t understand the question/concern. If you have enabled checksum > offload then of course you don''t recalculate the checksum, that''s the > whole point of offloading it. >If your frontend doesn''t advertise checksum offload at ring connect time then the backend should not advertise checksum offload to the network stack and hence all packets passing to the VM should have valid checksum. If your frontend *does* advertise offload then you need to look at the NETRXF flags in the head descriptor of any packet you handle to determine the presence/validity of any checksum. Paul
On Thu, 2013-12-05 at 11:45 +0000, Richard Mortier wrote:> On 5 Dec 2013, at 11:39, Ian Campbell wrote: > > > On Thu, 2013-12-05 at 11:29 +0000, Anil Madhavapeddy wrote: > >> On Tue, Dec 03, 2013 at 01:00:23PM +0000, Balraj Singh wrote: > >>> Hi, > >>> > >>> I''m working on verifying TCP checksums on incoming packets in Mirage, but > >>> I''ve run into a bit of a problem. > >>> > >>> If TCP checksum offload is turned on on a virtual interface (this is the > >>> default), and if the TCP connection is local to the machine, it looks like > >>> Xen does not calculate the checksum at all. This may be valid because Xen > >>> may be providing a stronger guarantee, but it means that incoming packets > >>> don''t have a valid checksum in the header. This then means that in Mirage > >>> we can''t just have checksum verification turned on all the time. This > >>> would have been the safe fall back option and detecting that checksum > >>> offload is on, and then not duplicating the verification in Mirage would > >>> have been an optimisation. But it looks like this is not an option. Now I > >>> need to know for every incoming packet whether checksum verification should > >>> be done or not. It should ideally be for every packet since chksum offload > >>> can be turned off and on on the VIF and existing tcp connections should > >>> continue. If not every packet, I need to get a notification or efficiently > >>> detect right away that the setting is changed on the VIF. > >> > >> This is a question that seems to keep coming up even for Linux and > >> Windows, as the combination of local<->local VMs vs local<->off-host and > >> the checksum offload is quite confusing. > >> > >> CCing xen-devel: is the appropriate behaviour for a guest VM that wants to > >> use checksum offloading in all situations documented anywhere? > > > > I don''t understand the question/concern. If you have enabled checksum > > offload then of course you don''t recalculate the checksum, that''s the > > whole point of offloading it. > > i think balraj''s question arises because the status of checksum > offload can change mid-tcp-flow. how does he know whether it''s on or > off for a given packet?It''s a property of the NIC configuration. In Linux for example a received skb gets a field set which indicates the offload state at the time it was received. Ian.
On 05/12/13 11:39, Ian Campbell wrote:> On Thu, 2013-12-05 at 11:29 +0000, Anil Madhavapeddy wrote: >> > On Tue, Dec 03, 2013 at 01:00:23PM +0000, Balraj Singh wrote: >>> > > Hi, >>> > > >>> > > I''m working on verifying TCP checksums on incoming packets in Mirage, but >>> > > I''ve run into a bit of a problem. >>> > > >>> > > If TCP checksum offload is turned on on a virtual interface (this is the >>> > > default), and if the TCP connection is local to the machine, it looks like >>> > > Xen does not calculate the checksum at all. This may be valid because Xen >>> > > may be providing a stronger guarantee, but it means that incoming packets >>> > > don''t have a valid checksum in the header. This then means that in Mirage >>> > > we can''t just have checksum verification turned on all the time. This >>> > > would have been the safe fall back option and detecting that checksum >>> > > offload is on, and then not duplicating the verification in Mirage would >>> > > have been an optimisation. But it looks like this is not an option. Now I >>> > > need to know for every incoming packet whether checksum verification should >>> > > be done or not. It should ideally be for every packet since chksum offload >>> > > can be turned off and on on the VIF and existing tcp connections should >>> > > continue. If not every packet, I need to get a notification or efficiently >>> > > detect right away that the setting is changed on the VIF. >> > >> > This is a question that seems to keep coming up even for Linux and >> > Windows, as the combination of local<->local VMs vs local<->off-host and >> > the checksum offload is quite confusing. >> > >> > CCing xen-devel: is the appropriate behaviour for a guest VM that wants to >> > use checksum offloading in all situations documented anywhere? > I don''t understand the question/concern. If you have enabled checksum > offload then of course you don''t recalculate the checksum, that''s the > whole point of offloading it.I get this a lot. There are a few different cases: * domain to domain traffic * domain to external traffic with egress from a NIC that does offload * domain to external through a non-offloading NIC With xen checksum offloading, domain to domain traffic appears to be received with a bad checksum. This is OK, there is no point in calculating a checksum if the packets are only going through memory. If your memory is going to randomly corrupt packets you have more bigger problems to worry about. However, this does upset at least Solaris: if you''re using a Solaris guest for NAT then the NAT module on Solaris gets all upset if the checksum is wrong and drops the packets. (This is Solaris''s NAT module being overly picky, it may need to recalculate or at least invalidate the existing checksum, but it doesn''t need to check it as well.) The second two cases are of interest from the domain perspective. A domain has no way of knowing how any given packet is going to leave the host (or even if it is) so it can''t know ahead of time whether to calculate any checksums: the skb''s are just marked with "checksum needed" as usual and either the egress NIC will do the job or dom0 will do it. There is absolutely nothing wrong in any of this (Solaris notwithstanding). The difficulty is getting people to realise that checksums are only calculated when a packet hits the cat-5. It doesn''t need documenting, it just needs a little thought. I got tired of hammering the point home :) jch
> -----Original Message----- > From: xen-devel-bounces@lists.xen.org [mailto:xen-devel- > bounces@lists.xen.org] On Behalf Of Richard Mortier > Sent: 05 December 2013 11:45 > To: Ian Campbell > Cc: xen-devel@lists.xenproject.org; Balraj Singh; Mirage List; Anil > Madhavapeddy > Subject: Re: [Xen-devel] Question about TCP checksum offload in Xen > > > On 5 Dec 2013, at 11:39, Ian Campbell wrote: > > > On Thu, 2013-12-05 at 11:29 +0000, Anil Madhavapeddy wrote: > >> On Tue, Dec 03, 2013 at 01:00:23PM +0000, Balraj Singh wrote: > >>> Hi, > >>> > >>> I''m working on verifying TCP checksums on incoming packets in Mirage, > but > >>> I''ve run into a bit of a problem. > >>> > >>> If TCP checksum offload is turned on on a virtual interface (this is the > >>> default), and if the TCP connection is local to the machine, it looks like > >>> Xen does not calculate the checksum at all. This may be valid because > Xen > >>> may be providing a stronger guarantee, but it means that incoming > packets > >>> don''t have a valid checksum in the header. This then means that in > Mirage > >>> we can''t just have checksum verification turned on all the time. This > >>> would have been the safe fall back option and detecting that checksum > >>> offload is on, and then not duplicating the verification in Mirage would > >>> have been an optimisation. But it looks like this is not an option. Now I > >>> need to know for every incoming packet whether checksum verification > should > >>> be done or not. It should ideally be for every packet since chksum > offload > >>> can be turned off and on on the VIF and existing tcp connections should > >>> continue. If not every packet, I need to get a notification or efficiently > >>> detect right away that the setting is changed on the VIF. > >> > >> This is a question that seems to keep coming up even for Linux and > >> Windows, as the combination of local<->local VMs vs local<->off-host and > >> the checksum offload is quite confusing. > >> > >> CCing xen-devel: is the appropriate behaviour for a guest VM that wants > to > >> use checksum offloading in all situations documented anywhere? > > > > I don''t understand the question/concern. If you have enabled checksum > > offload then of course you don''t recalculate the checksum, that''s the > > whole point of offloading it. > > i think balraj''s question arises because the status of checksum offload can > change mid-tcp-flow. how does he know whether it''s on or off for a given > packet? >Why do you assert that it can change mid-flow? The configuration is decided at ring connect time. Nothing should change after that. If, at that time, the frontend declares it can handle packets with no checksum then clearly it may get some packets that do have checksum as well as ones that don''t, depending on where they originated. Paul
On 5 Dec 2013, at 12:47, Paul Durrant wrote: ...>>>> On Tue, Dec 03, 2013 at 01:00:23PM +0000, Balraj Singh wrote: >>>>...>>>>> It should ideally be for every packet since chksum >> offload >>>>> can be turned off and on on the VIF and existing tcp connections should >>>>> continue. If not every packet, I need to get a notification or efficiently >>>>> detect right away that the setting is changed on the VIF....> earlier today, i wrote: > >> i think balraj''s question arises because the status of checksum offload can >> change mid-tcp-flow. how does he know whether it''s on or off for a given >> packet? > > Why do you assert that it can change mid-flow? The configuration is decided at ring connect time. Nothing should change after that. If, at that time, the frontend declares it can handle packets with no checksum then clearly it may get some packets that do have checksum as well as ones that don''t, depending on where they originated.i didn''t mean to :) i was just trying to point out to ian the reason for balraj''s question -- i''ll leave it to balraj from here... -- Cheers, R. This message and any attachment are intended solely for the addressee and may contain confidential information. If you have received this message in error, please send it back to me, and immediately delete it. Please do not use, copy or disclose the information contained in this message or in any attachment. Any views or opinions expressed by the author of this email do not necessarily reflect the views of the University of Nottingham. This message has been checked for viruses but the contents of an attachment may still contain software viruses which could damage your computer system, you are advised to perform your own checks. Email communications with the University of Nottingham may be monitored as permitted by UK legislation.
Thanks very much. This is fine and exactly what is observed (that the checksum is only calculated going to/from a real wire). The engineering choice makes sense. But this does mean that I need to know when the stack receives a pkt if the checksum should be verified for that packet. I can''t do a redundant verification because a bad chksum may be because the field was clobbered (not set) during earlier verification. So, what I wanted to know was where to look for this flag - a snippet of code or a ptr to a file :). Thanks, Balraj On Thu, Dec 5, 2013 at 12:37 PM, John Haxby <john.haxby@oracle.com> wrote:> On 05/12/13 11:39, Ian Campbell wrote: > > On Thu, 2013-12-05 at 11:29 +0000, Anil Madhavapeddy wrote: > >> > On Tue, Dec 03, 2013 at 01:00:23PM +0000, Balraj Singh wrote: > >>> > > Hi, > >>> > > > >>> > > I''m working on verifying TCP checksums on incoming packets in > Mirage, but > >>> > > I''ve run into a bit of a problem. > >>> > > > >>> > > If TCP checksum offload is turned on on a virtual interface (this > is the > >>> > > default), and if the TCP connection is local to the machine, it > looks like > >>> > > Xen does not calculate the checksum at all. This may be valid > because Xen > >>> > > may be providing a stronger guarantee, but it means that incoming > packets > >>> > > don''t have a valid checksum in the header. This then means that > in Mirage > >>> > > we can''t just have checksum verification turned on all the time. > This > >>> > > would have been the safe fall back option and detecting that > checksum > >>> > > offload is on, and then not duplicating the verification in Mirage > would > >>> > > have been an optimisation. But it looks like this is not an > option. Now I > >>> > > need to know for every incoming packet whether checksum > verification should > >>> > > be done or not. It should ideally be for every packet since > chksum offload > >>> > > can be turned off and on on the VIF and existing tcp connections > should > >>> > > continue. If not every packet, I need to get a notification or > efficiently > >>> > > detect right away that the setting is changed on the VIF. > >> > > >> > This is a question that seems to keep coming up even for Linux and > >> > Windows, as the combination of local<->local VMs vs local<->off-host > and > >> > the checksum offload is quite confusing. > >> > > >> > CCing xen-devel: is the appropriate behaviour for a guest VM that > wants to > >> > use checksum offloading in all situations documented anywhere? > > I don''t understand the question/concern. If you have enabled checksum > > offload then of course you don''t recalculate the checksum, that''s the > > whole point of offloading it. > > I get this a lot. > > There are a few different cases: > > * domain to domain traffic > * domain to external traffic with egress from a NIC that does offload > * domain to external through a non-offloading NIC > > With xen checksum offloading, domain to domain traffic appears to be > received with a bad checksum. This is OK, there is no point in > calculating a checksum if the packets are only going through memory. If > your memory is going to randomly corrupt packets you have more bigger > problems to worry about. However, this does upset at least Solaris: if > you''re using a Solaris guest for NAT then the NAT module on Solaris gets > all upset if the checksum is wrong and drops the packets. (This is > Solaris''s NAT module being overly picky, it may need to recalculate or > at least invalidate the existing checksum, but it doesn''t need to check > it as well.) > > The second two cases are of interest from the domain perspective. A > domain has no way of knowing how any given packet is going to leave the > host (or even if it is) so it can''t know ahead of time whether to > calculate any checksums: the skb''s are just marked with "checksum > needed" as usual and either the egress NIC will do the job or dom0 will > do it. > > There is absolutely nothing wrong in any of this (Solaris > notwithstanding). The difficulty is getting people to realise that > checksums are only calculated when a packet hits the cat-5. It doesn''t > need documenting, it just needs a little thought. I got tired of > hammering the point home :) > > jch >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
i thought a value of 0 in the tcp chcksum field indicated "no checksum" and could be used in the cases you identify and ought not to trigger problems in correct code? On Thu, Dec 5, 2013 at 12:37 PM, John Haxby <john.haxby@oracle.com> wrote:> On 05/12/13 11:39, Ian Campbell wrote: > > On Thu, 2013-12-05 at 11:29 +0000, Anil Madhavapeddy wrote: > >> > On Tue, Dec 03, 2013 at 01:00:23PM +0000, Balraj Singh wrote: > >>> > > Hi, > >>> > > > >>> > > I''m working on verifying TCP checksums on incoming packets in > Mirage, but > >>> > > I''ve run into a bit of a problem. > >>> > > > >>> > > If TCP checksum offload is turned on on a virtual interface (this > is the > >>> > > default), and if the TCP connection is local to the machine, it > looks like > >>> > > Xen does not calculate the checksum at all. This may be valid > because Xen > >>> > > may be providing a stronger guarantee, but it means that incoming > packets > >>> > > don''t have a valid checksum in the header. This then means that > in Mirage > >>> > > we can''t just have checksum verification turned on all the time. > This > >>> > > would have been the safe fall back option and detecting that > checksum > >>> > > offload is on, and then not duplicating the verification in Mirage > would > >>> > > have been an optimisation. But it looks like this is not an > option. Now I > >>> > > need to know for every incoming packet whether checksum > verification should > >>> > > be done or not. It should ideally be for every packet since > chksum offload > >>> > > can be turned off and on on the VIF and existing tcp connections > should > >>> > > continue. If not every packet, I need to get a notification or > efficiently > >>> > > detect right away that the setting is changed on the VIF. > >> > > >> > This is a question that seems to keep coming up even for Linux and > >> > Windows, as the combination of local<->local VMs vs local<->off-host > and > >> > the checksum offload is quite confusing. > >> > > >> > CCing xen-devel: is the appropriate behaviour for a guest VM that > wants to > >> > use checksum offloading in all situations documented anywhere? > > I don''t understand the question/concern. If you have enabled checksum > > offload then of course you don''t recalculate the checksum, that''s the > > whole point of offloading it. > > I get this a lot. > > There are a few different cases: > > * domain to domain traffic > * domain to external traffic with egress from a NIC that does offload > * domain to external through a non-offloading NIC > > With xen checksum offloading, domain to domain traffic appears to be > received with a bad checksum. This is OK, there is no point in > calculating a checksum if the packets are only going through memory. If > your memory is going to randomly corrupt packets you have more bigger > problems to worry about. However, this does upset at least Solaris: if > you''re using a Solaris guest for NAT then the NAT module on Solaris gets > all upset if the checksum is wrong and drops the packets. (This is > Solaris''s NAT module being overly picky, it may need to recalculate or > at least invalidate the existing checksum, but it doesn''t need to check > it as well.) > > The second two cases are of interest from the domain perspective. A > domain has no way of knowing how any given packet is going to leave the > host (or even if it is) so it can''t know ahead of time whether to > calculate any checksums: the skb''s are just marked with "checksum > needed" as usual and either the egress NIC will do the job or dom0 will > do it. > > There is absolutely nothing wrong in any of this (Solaris > notwithstanding). The difficulty is getting people to realise that > checksums are only calculated when a packet hits the cat-5. It doesn''t > need documenting, it just needs a little thought. I got tired of > hammering the point home :) > > jch > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Thanks for clarifying Mort. I did this test: I set up a long lived TCP connection from a linux guest to a mirage guest each thru a VIF to the s/w bridge. The various offload settings including checksum offload were the default which is ON. All packets received by Mirage had bad checksums. Now from dom0 I changed the offload settings on the VIFs using ethtool, so the checksum offload was OFF. Now all packets had good checksums. So to have no packets in the jeopardy of being accepted without their checksum verified, I would like to know on each received pkt if the checksum was already verified. Detecting immediately that the setting on the VIF has changed could also work, but it would involve throwing away some packets. Then again, I may have misunderstood something about how the stack should interact with the backend. We do declare (maybe just by staying mute, I think) that we can handle packets with no checksums. But does that mean that we never need to do a checksum verification on RX packets? If so that''s the best. Though then I will need to think about what was going on in my test. Thanks, Balraj On Thu, Dec 5, 2013 at 12:55 PM, Richard Mortier < Richard.Mortier@nottingham.ac.uk> wrote:> > On 5 Dec 2013, at 12:47, Paul Durrant wrote: > > ... > >>>> On Tue, Dec 03, 2013 at 01:00:23PM +0000, Balraj Singh wrote: > >>>> > ... > >>>>> It should ideally be for every packet since chksum > >> offload > >>>>> can be turned off and on on the VIF and existing tcp connections > should > >>>>> continue. If not every packet, I need to get a notification or > efficiently > >>>>> detect right away that the setting is changed on the VIF. > > ... > > > earlier today, i wrote: > > > >> i think balraj''s question arises because the status of checksum offload > can > >> change mid-tcp-flow. how does he know whether it''s on or off for a given > >> packet? > > > > Why do you assert that it can change mid-flow? The configuration is > decided at ring connect time. Nothing should change after that. If, at that > time, the frontend declares it can handle packets with no checksum then > clearly it may get some packets that do have checksum as well as ones that > don''t, depending on where they originated. > > i didn''t mean to :) i was just trying to point out to ian the reason for > balraj''s question -- i''ll leave it to balraj from here... > > -- > Cheers, > > R. > > > > > This message and any attachment are intended solely for the addressee and > may contain confidential information. If you have received this message in > error, please send it back to me, and immediately delete it. Please do > not use, copy or disclose the information contained in this message or in > any attachment. Any views or opinions expressed by the author of this > email do not necessarily reflect the views of the University of Nottingham. > > This message has been checked for viruses but the contents of an attachment > may still contain software viruses which could damage your computer > system, you are advised to perform your own checks. Email communications > with the University of Nottingham may be monitored as permitted by UK > legislation. > > > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
De-html... ------ From: balraj885@gmail.com [mailto:balraj885@gmail.com] On Behalf Of Balraj Singh Sent: 05 December 2013 13:33 To: Richard Mortier Cc: Paul Durrant; Ian Campbell; xen-devel@lists.xenproject.org; Mirage List; Anil Madhavapeddy Subject: Re: [Xen-devel] Question about TCP checksum offload in Xen Thanks for clarifying Mort. I did this test: I set up a long lived TCP connection from a linux guest to a mirage guest each thru a VIF to the s/w bridge. The various offload settings including checksum offload were the default which is ON. All packets received by Mirage had bad checksums. Now from dom0 I changed the offload settings on the VIFs using ethtool, so the checksum offload was OFF. Now all packets had good checksums. So to have no packets in the jeopardy of being accepted without their checksum verified, I would like to know on each received pkt if the checksum was already verified. Detecting immediately that the setting on the VIF has changed could also work, but it would involve throwing away some packets. Then again, I may have misunderstood something about how the stack should interact with the backend. We do declare (maybe just by staying mute, I think) that we can handle packets with no checksums. But does that mean that we never need to do a checksum verification on RX packets? If so that''s the best. Though then I will need to think about what was going on in my test. ----- And can we avoid top-posting in future, please. It makes it very hard to follow threads. Possibly you ran into the issue because (for some reason) the default is to expect a frontend to handle IPv4 TCP/UDP checksum offload. The frontend needs to set feature-no-csum-offload to avoid this expectation. I imagine that''s what you want to do. Also, the state of the checksum field when being passed a packet for offloading will not be zero. It should contain a valid pseudo-header checksum. Paul
That could work too to signal in the packet itself that the checksum is ok. But I don''t think the received packets had 0 in the chksum field. It was just some random value, though I can check again. Thanks, Balraj On Thu, Dec 5, 2013 at 1:15 PM, Jon Crowcroft <jon.crowcroft@cl.cam.ac.uk>wrote:> i thought a value of 0 in the tcp chcksum field indicated "no checksum" > and could be used in the cases you identify and ought not to trigger > problems in correct code? > > > On Thu, Dec 5, 2013 at 12:37 PM, John Haxby <john.haxby@oracle.com> wrote: > >> On 05/12/13 11:39, Ian Campbell wrote: >> > On Thu, 2013-12-05 at 11:29 +0000, Anil Madhavapeddy wrote: >> >> > On Tue, Dec 03, 2013 at 01:00:23PM +0000, Balraj Singh wrote: >> >>> > > Hi, >> >>> > > >> >>> > > I''m working on verifying TCP checksums on incoming packets in >> Mirage, but >> >>> > > I''ve run into a bit of a problem. >> >>> > > >> >>> > > If TCP checksum offload is turned on on a virtual interface (this >> is the >> >>> > > default), and if the TCP connection is local to the machine, it >> looks like >> >>> > > Xen does not calculate the checksum at all. This may be valid >> because Xen >> >>> > > may be providing a stronger guarantee, but it means that incoming >> packets >> >>> > > don''t have a valid checksum in the header. This then means that >> in Mirage >> >>> > > we can''t just have checksum verification turned on all the time. >> This >> >>> > > would have been the safe fall back option and detecting that >> checksum >> >>> > > offload is on, and then not duplicating the verification in >> Mirage would >> >>> > > have been an optimisation. But it looks like this is not an >> option. Now I >> >>> > > need to know for every incoming packet whether checksum >> verification should >> >>> > > be done or not. It should ideally be for every packet since >> chksum offload >> >>> > > can be turned off and on on the VIF and existing tcp connections >> should >> >>> > > continue. If not every packet, I need to get a notification or >> efficiently >> >>> > > detect right away that the setting is changed on the VIF. >> >> > >> >> > This is a question that seems to keep coming up even for Linux and >> >> > Windows, as the combination of local<->local VMs vs local<->off-host >> and >> >> > the checksum offload is quite confusing. >> >> > >> >> > CCing xen-devel: is the appropriate behaviour for a guest VM that >> wants to >> >> > use checksum offloading in all situations documented anywhere? >> > I don''t understand the question/concern. If you have enabled checksum >> > offload then of course you don''t recalculate the checksum, that''s the >> > whole point of offloading it. >> >> I get this a lot. >> >> There are a few different cases: >> >> * domain to domain traffic >> * domain to external traffic with egress from a NIC that does offload >> * domain to external through a non-offloading NIC >> >> With xen checksum offloading, domain to domain traffic appears to be >> received with a bad checksum. This is OK, there is no point in >> calculating a checksum if the packets are only going through memory. If >> your memory is going to randomly corrupt packets you have more bigger >> problems to worry about. However, this does upset at least Solaris: if >> you''re using a Solaris guest for NAT then the NAT module on Solaris gets >> all upset if the checksum is wrong and drops the packets. (This is >> Solaris''s NAT module being overly picky, it may need to recalculate or >> at least invalidate the existing checksum, but it doesn''t need to check >> it as well.) >> >> The second two cases are of interest from the domain perspective. A >> domain has no way of knowing how any given packet is going to leave the >> host (or even if it is) so it can''t know ahead of time whether to >> calculate any checksums: the skb''s are just marked with "checksum >> needed" as usual and either the egress NIC will do the job or dom0 will >> do it. >> >> There is absolutely nothing wrong in any of this (Solaris >> notwithstanding). The difficulty is getting people to realise that >> checksums are only calculated when a packet hits the cat-5. It doesn''t >> need documenting, it just needs a little thought. I got tired of >> hammering the point home :) >> >> jch >> >> >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On 05/12/13 12:56, Balraj Singh wrote:> Thanks very much. This is fine and exactly what is observed (that the > checksum is only calculated going to/from a real wire). The engineering > choice makes sense. But this does mean that I need to know when the > stack receives a pkt if the checksum should be verified for that packet. > I can''t do a redundant verification because a bad chksum may be because > the field was clobbered (not set) during earlier verification. So, what > I wanted to know was where to look for this flag - a snippet of code or > a ptr to a file :). >Look at the skb in the netfront: there''s a flag in there to say whether the checksum is valid. Interestingly, if the checksum is valid then you know that the data has come from outside the host (otherwise it would have been dropped). Why do you even care about the checksum? Packets with bad checksums are dropped by the NIC (with checksum offload) or by the NIC''s driver (without) -- the only packets you''ll see with a bad checksum are those that have come from another domain on the same host. jch> Thanks, > > Balraj > > > > On Thu, Dec 5, 2013 at 12:37 PM, John Haxby <john.haxby@oracle.com > <mailto:john.haxby@oracle.com>> wrote: > > On 05/12/13 11:39, Ian Campbell wrote: > > On Thu, 2013-12-05 at 11:29 +0000, Anil Madhavapeddy wrote: > >> > On Tue, Dec 03, 2013 at 01:00:23PM +0000, Balraj Singh wrote: > >>> > > Hi, > >>> > > > >>> > > I''m working on verifying TCP checksums on incoming packets > in Mirage, but > >>> > > I''ve run into a bit of a problem. > >>> > > > >>> > > If TCP checksum offload is turned on on a virtual interface > (this is the > >>> > > default), and if the TCP connection is local to the machine, > it looks like > >>> > > Xen does not calculate the checksum at all. This may be > valid because Xen > >>> > > may be providing a stronger guarantee, but it means that > incoming packets > >>> > > don''t have a valid checksum in the header. This then means > that in Mirage > >>> > > we can''t just have checksum verification turned on all the > time. This > >>> > > would have been the safe fall back option and detecting that > checksum > >>> > > offload is on, and then not duplicating the verification in > Mirage would > >>> > > have been an optimisation. But it looks like this is not an > option. Now I > >>> > > need to know for every incoming packet whether checksum > verification should > >>> > > be done or not. It should ideally be for every packet since > chksum offload > >>> > > can be turned off and on on the VIF and existing tcp > connections should > >>> > > continue. If not every packet, I need to get a notification > or efficiently > >>> > > detect right away that the setting is changed on the VIF. > >> > > >> > This is a question that seems to keep coming up even for Linux and > >> > Windows, as the combination of local<->local VMs vs > local<->off-host and > >> > the checksum offload is quite confusing. > >> > > >> > CCing xen-devel: is the appropriate behaviour for a guest VM > that wants to > >> > use checksum offloading in all situations documented anywhere? > > I don''t understand the question/concern. If you have enabled checksum > > offload then of course you don''t recalculate the checksum, that''s the > > whole point of offloading it. > > I get this a lot. > > There are a few different cases: > > * domain to domain traffic > * domain to external traffic with egress from a NIC that does offload > * domain to external through a non-offloading NIC > > With xen checksum offloading, domain to domain traffic appears to be > received with a bad checksum. This is OK, there is no point in > calculating a checksum if the packets are only going through memory. If > your memory is going to randomly corrupt packets you have more bigger > problems to worry about. However, this does upset at least Solaris: if > you''re using a Solaris guest for NAT then the NAT module on Solaris gets > all upset if the checksum is wrong and drops the packets. (This is > Solaris''s NAT module being overly picky, it may need to recalculate or > at least invalidate the existing checksum, but it doesn''t need to check > it as well.) > > The second two cases are of interest from the domain perspective. A > domain has no way of knowing how any given packet is going to leave the > host (or even if it is) so it can''t know ahead of time whether to > calculate any checksums: the skb''s are just marked with "checksum > needed" as usual and either the egress NIC will do the job or dom0 will > do it. > > There is absolutely nothing wrong in any of this (Solaris > notwithstanding). The difficulty is getting people to realise that > checksums are only calculated when a packet hits the cat-5. It doesn''t > need documenting, it just needs a little thought. I got tired of > hammering the point home :) > > jch > >
> With xen checksum offloading, domain to domain traffic appears to be > received with a bad checksum. This is OK, there is no point in > calculating a checksum if the packets are only going through memory. If > your memory is going to randomly corrupt packets you have more bigger > problems to worry about. However, this does upset at least Solaris: if > you''re using a Solaris guest for NAT then the NAT module on Solaris gets > all upset if the checksum is wrong and drops the packets. (This is > Solaris''s NAT module being overly picky, it may need to recalculate or > at least invalidate the existing checksum, but it doesn''t need to check > it as well.)Windows 2003 can''t handle receiving packets that have a bad checksum either. Despite the fact that the ''checksum good'' flag is set, Windows seems to go ahead and check it anyway. I wonder if it''s the firewall doing that in Windows too... I always just assumed it was NDIS itself. Windows also seems to differ from Linux in its opinion on whether the total packet length in a gso packet is included in the psuedoheader checksum. That took a bit of mucking around to find as some hardware seems to ignore the psuedoheader checksum and calculate the whole checksum again anyway, so you only see the problem on some cards. James
Possibly Parallel Threads
- [Community Review] Mirage Incubation Project Proposal
- Slow TCP performance between Windows Vista and Xen PV-on-HVM guest
- UDP checksums broken in Dom0 -> DomU vif transfer
- [PATCH V2 net-next 0/6] virtio-net: Add SCTP checksum offload support
- [PATCH V2 net-next 0/6] virtio-net: Add SCTP checksum offload support