Michael S. Tsirkin
2020-Feb-11 10:37 UTC
[PATCH] virtio: Work around frames incorrectly marked as gso
On Tue, Feb 11, 2020 at 07:42:37AM +0000, Anton Ivanov wrote:> On 11/02/2020 02:51, Jason Wang wrote: > > > > On 2020/2/11 ??12:55, Anton Ivanov wrote: > > > > > > > > > On 09/12/2019 10:48, anton.ivanov at cambridgegreys.com wrote: > > > > From: Anton Ivanov <anton.ivanov at cambridgegreys.com> > > > > > > > > Some of the frames marked as GSO which arrive at > > > > virtio_net_hdr_from_skb() have no GSO_TYPE, no > > > > fragments (data_len = 0) and length significantly shorter > > > > than the MTU (752 in my experiments). > > > > > > > > This is observed on raw sockets reading off vEth interfaces > > > > in all 4.x and 5.x kernels I tested. > > > > > > > > These frames are reported as invalid while they are in fact > > > > gso-less frames. > > > > > > > > This patch marks the vnet header as no-GSO for them instead > > > > of reporting it as invalid. > > > > > > > > Signed-off-by: Anton Ivanov <anton.ivanov at cambridgegreys.com> > > > > --- > > > > ? include/linux/virtio_net.h | 8 ++++++-- > > > > ? 1 file changed, 6 insertions(+), 2 deletions(-) > > > > > > > > diff --git a/include/linux/virtio_net.h b/include/linux/virtio_net.h > > > > index 0d1fe9297ac6..d90d5cff1b9a 100644 > > > > --- a/include/linux/virtio_net.h > > > > +++ b/include/linux/virtio_net.h > > > > @@ -112,8 +112,12 @@ static inline int > > > > virtio_net_hdr_from_skb(const struct sk_buff *skb, > > > > ????????????? hdr->gso_type = VIRTIO_NET_HDR_GSO_TCPV4; > > > > ????????? else if (sinfo->gso_type & SKB_GSO_TCPV6) > > > > ????????????? hdr->gso_type = VIRTIO_NET_HDR_GSO_TCPV6; > > > > -??????? else > > > > -??????????? return -EINVAL; > > > > +??????? else { > > > > +??????????? if (skb->data_len == 0) > > > > +??????????????? hdr->gso_type = VIRTIO_NET_HDR_GSO_NONE; > > > > +??????????? else > > > > +??????????????? return -EINVAL; > > > > +??????? } > > > > ????????? if (sinfo->gso_type & SKB_GSO_TCP_ECN) > > > > ????????????? hdr->gso_type |= VIRTIO_NET_HDR_GSO_ECN; > > > > ????? } else > > > > > > > > > > ping. > > > > > > > Do you mean gso_size is set but gso_type is not? Looks like a bug > > elsewhere. > > > > Thanks > > > > > Yes. > > I could not trace it where it is coming from. > > I see it when doing recvmmsg on raw sockets in the UML vector network > drivers. >I think we need to find the culprit and fix it there, lots of other things can break otherwise. Just printing out skb->dev->name should do the trick, no?> -- > Anton R. Ivanov > Cambridgegreys Limited. Registered in England. Company Number 10273661 > https://www.cambridgegreys.com/
Anton Ivanov
2020-Feb-12 10:03 UTC
[PATCH] virtio: Work around frames incorrectly marked as gso
On 11/02/2020 10:37, Michael S. Tsirkin wrote:> On Tue, Feb 11, 2020 at 07:42:37AM +0000, Anton Ivanov wrote: >> On 11/02/2020 02:51, Jason Wang wrote: >>> >>> On 2020/2/11 ??12:55, Anton Ivanov wrote: >>>> >>>> >>>> On 09/12/2019 10:48, anton.ivanov at cambridgegreys.com wrote: >>>>> From: Anton Ivanov <anton.ivanov at cambridgegreys.com> >>>>> >>>>> Some of the frames marked as GSO which arrive at >>>>> virtio_net_hdr_from_skb() have no GSO_TYPE, no >>>>> fragments (data_len = 0) and length significantly shorter >>>>> than the MTU (752 in my experiments). >>>>> >>>>> This is observed on raw sockets reading off vEth interfaces >>>>> in all 4.x and 5.x kernels I tested. >>>>> >>>>> These frames are reported as invalid while they are in fact >>>>> gso-less frames. >>>>> >>>>> This patch marks the vnet header as no-GSO for them instead >>>>> of reporting it as invalid. >>>>> >>>>> Signed-off-by: Anton Ivanov <anton.ivanov at cambridgegreys.com> >>>>> --- >>>>> ? include/linux/virtio_net.h | 8 ++++++-- >>>>> ? 1 file changed, 6 insertions(+), 2 deletions(-) >>>>> >>>>> diff --git a/include/linux/virtio_net.h b/include/linux/virtio_net.h >>>>> index 0d1fe9297ac6..d90d5cff1b9a 100644 >>>>> --- a/include/linux/virtio_net.h >>>>> +++ b/include/linux/virtio_net.h >>>>> @@ -112,8 +112,12 @@ static inline int >>>>> virtio_net_hdr_from_skb(const struct sk_buff *skb, >>>>> ????????????? hdr->gso_type = VIRTIO_NET_HDR_GSO_TCPV4; >>>>> ????????? else if (sinfo->gso_type & SKB_GSO_TCPV6) >>>>> ????????????? hdr->gso_type = VIRTIO_NET_HDR_GSO_TCPV6; >>>>> -??????? else >>>>> -??????????? return -EINVAL; >>>>> +??????? else { >>>>> +??????????? if (skb->data_len == 0) >>>>> +??????????????? hdr->gso_type = VIRTIO_NET_HDR_GSO_NONE; >>>>> +??????????? else >>>>> +??????????????? return -EINVAL; >>>>> +??????? } >>>>> ????????? if (sinfo->gso_type & SKB_GSO_TCP_ECN) >>>>> ????????????? hdr->gso_type |= VIRTIO_NET_HDR_GSO_ECN; >>>>> ????? } else >>>>> >>>> >>>> ping. >>>> >>> >>> Do you mean gso_size is set but gso_type is not? Looks like a bug >>> elsewhere. >>> >>> Thanks >>> >>> >> Yes. >> >> I could not trace it where it is coming from. >> >> I see it when doing recvmmsg on raw sockets in the UML vector network >> drivers. >> > > I think we need to find the culprit and fix it there, lots of other things > can break otherwise. > Just printing out skb->dev->name should do the trick, no?I will rebuild my rig and retest (it's been a while since I worked on this bug). In theory, it should be veth - the test is over a vEth pair and all frames are locally originated by iperf. In practice - I will retest and post the results sometimes later today. Brgds, >> > >> -- >> Anton R. Ivanov >> Cambridgegreys Limited. Registered in England. Company Number 10273661 >> https://www.cambridgegreys.com/ > > > _______________________________________________ > linux-um mailing list > linux-um at lists.infradead.org > http://lists.infradead.org/mailman/listinfo/linux-um >-- Anton R. Ivanov Cambridgegreys Limited. Registered in England. Company Number 10273661 https://www.cambridgegreys.com/
Michael S. Tsirkin
2020-Feb-12 10:19 UTC
[PATCH] virtio: Work around frames incorrectly marked as gso
On Wed, Feb 12, 2020 at 10:03:31AM +0000, Anton Ivanov wrote:> > > On 11/02/2020 10:37, Michael S. Tsirkin wrote: > > On Tue, Feb 11, 2020 at 07:42:37AM +0000, Anton Ivanov wrote: > > > On 11/02/2020 02:51, Jason Wang wrote: > > > > > > > > On 2020/2/11 ??12:55, Anton Ivanov wrote: > > > > > > > > > > > > > > > On 09/12/2019 10:48, anton.ivanov at cambridgegreys.com wrote: > > > > > > From: Anton Ivanov <anton.ivanov at cambridgegreys.com> > > > > > > > > > > > > Some of the frames marked as GSO which arrive at > > > > > > virtio_net_hdr_from_skb() have no GSO_TYPE, no > > > > > > fragments (data_len = 0) and length significantly shorter > > > > > > than the MTU (752 in my experiments). > > > > > > > > > > > > This is observed on raw sockets reading off vEth interfaces > > > > > > in all 4.x and 5.x kernels I tested. > > > > > > > > > > > > These frames are reported as invalid while they are in fact > > > > > > gso-less frames. > > > > > > > > > > > > This patch marks the vnet header as no-GSO for them instead > > > > > > of reporting it as invalid. > > > > > > > > > > > > Signed-off-by: Anton Ivanov <anton.ivanov at cambridgegreys.com> > > > > > > --- > > > > > > ? include/linux/virtio_net.h | 8 ++++++-- > > > > > > ? 1 file changed, 6 insertions(+), 2 deletions(-) > > > > > > > > > > > > diff --git a/include/linux/virtio_net.h b/include/linux/virtio_net.h > > > > > > index 0d1fe9297ac6..d90d5cff1b9a 100644 > > > > > > --- a/include/linux/virtio_net.h > > > > > > +++ b/include/linux/virtio_net.h > > > > > > @@ -112,8 +112,12 @@ static inline int > > > > > > virtio_net_hdr_from_skb(const struct sk_buff *skb, > > > > > > ????????????? hdr->gso_type = VIRTIO_NET_HDR_GSO_TCPV4; > > > > > > ????????? else if (sinfo->gso_type & SKB_GSO_TCPV6) > > > > > > ????????????? hdr->gso_type = VIRTIO_NET_HDR_GSO_TCPV6; > > > > > > -??????? else > > > > > > -??????????? return -EINVAL; > > > > > > +??????? else { > > > > > > +??????????? if (skb->data_len == 0) > > > > > > +??????????????? hdr->gso_type = VIRTIO_NET_HDR_GSO_NONE; > > > > > > +??????????? else > > > > > > +??????????????? return -EINVAL; > > > > > > +??????? } > > > > > > ????????? if (sinfo->gso_type & SKB_GSO_TCP_ECN) > > > > > > ????????????? hdr->gso_type |= VIRTIO_NET_HDR_GSO_ECN; > > > > > > ????? } else > > > > > > > > > > > > > > > > ping. > > > > > > > > > > > > > Do you mean gso_size is set but gso_type is not? Looks like a bug > > > > elsewhere. > > > > > > > > Thanks > > > > > > > > > > > Yes. > > > > > > I could not trace it where it is coming from. > > > > > > I see it when doing recvmmsg on raw sockets in the UML vector network > > > drivers. > > > > > > > I think we need to find the culprit and fix it there, lots of other things > > can break otherwise. > > Just printing out skb->dev->name should do the trick, no? > > I will rebuild my rig and retest (it's been a while since I worked on this bug). > > In theory, it should be veth - the test is over a vEth pair and all frames are locally originated by iperf. > > In practice - I will retest and post the results sometimes later today. > > Brgds,ok if it's veth then you need to add a similar printk patch to veth and re-run to see where does it come from originally.> > > > > > > > > -- > > > Anton R. Ivanov > > > Cambridgegreys Limited. Registered in England. Company Number 10273661 > > > https://www.cambridgegreys.com/ > > > > > > _______________________________________________ > > linux-um mailing list > > linux-um at lists.infradead.org > > http://lists.infradead.org/mailman/listinfo/linux-um > > > > -- > Anton R. Ivanov > Cambridgegreys Limited. Registered in England. Company Number 10273661 > https://www.cambridgegreys.com/
Anton Ivanov
2020-Feb-12 17:38 UTC
[PATCH] virtio: Work around frames incorrectly marked as gso
On 11/02/2020 10:37, Michael S. Tsirkin wrote:> On Tue, Feb 11, 2020 at 07:42:37AM +0000, Anton Ivanov wrote: >> On 11/02/2020 02:51, Jason Wang wrote: >>> >>> On 2020/2/11 ??12:55, Anton Ivanov wrote: >>>> >>>> >>>> On 09/12/2019 10:48, anton.ivanov at cambridgegreys.com wrote: >>>>> From: Anton Ivanov <anton.ivanov at cambridgegreys.com> >>>>> >>>>> Some of the frames marked as GSO which arrive at >>>>> virtio_net_hdr_from_skb() have no GSO_TYPE, no >>>>> fragments (data_len = 0) and length significantly shorter >>>>> than the MTU (752 in my experiments). >>>>> >>>>> This is observed on raw sockets reading off vEth interfaces >>>>> in all 4.x and 5.x kernels I tested. >>>>> >>>>> These frames are reported as invalid while they are in fact >>>>> gso-less frames. >>>>> >>>>> This patch marks the vnet header as no-GSO for them instead >>>>> of reporting it as invalid. >>>>> >>>>> Signed-off-by: Anton Ivanov <anton.ivanov at cambridgegreys.com> >>>>> --- >>>>> ? include/linux/virtio_net.h | 8 ++++++-- >>>>> ? 1 file changed, 6 insertions(+), 2 deletions(-) >>>>> >>>>> diff --git a/include/linux/virtio_net.h b/include/linux/virtio_net.h >>>>> index 0d1fe9297ac6..d90d5cff1b9a 100644 >>>>> --- a/include/linux/virtio_net.h >>>>> +++ b/include/linux/virtio_net.h >>>>> @@ -112,8 +112,12 @@ static inline int >>>>> virtio_net_hdr_from_skb(const struct sk_buff *skb, >>>>> ????????????? hdr->gso_type = VIRTIO_NET_HDR_GSO_TCPV4; >>>>> ????????? else if (sinfo->gso_type & SKB_GSO_TCPV6) >>>>> ????????????? hdr->gso_type = VIRTIO_NET_HDR_GSO_TCPV6; >>>>> -??????? else >>>>> -??????????? return -EINVAL; >>>>> +??????? else { >>>>> +??????????? if (skb->data_len == 0) >>>>> +??????????????? hdr->gso_type = VIRTIO_NET_HDR_GSO_NONE; >>>>> +??????????? else >>>>> +??????????????? return -EINVAL; >>>>> +??????? } >>>>> ????????? if (sinfo->gso_type & SKB_GSO_TCP_ECN) >>>>> ????????????? hdr->gso_type |= VIRTIO_NET_HDR_GSO_ECN; >>>>> ????? } else >>>>> >>>> >>>> ping. >>>> >>> >>> Do you mean gso_size is set but gso_type is not? Looks like a bug >>> elsewhere. >>> >>> Thanks >>> >>> >> Yes. >> >> I could not trace it where it is coming from. >> >> I see it when doing recvmmsg on raw sockets in the UML vector network >> drivers. >> > > I think we need to find the culprit and fix it there, lots of other things > can break otherwise. > Just printing out skb->dev->name should do the trick, no?The printk in virtio_net_hdr_from_skb says NULL. That is probably normal for a locally originated frame. I cannot reproduce this with network traffic by the way - it happens only if the traffic is locally originated on the host. A,> > >> -- >> Anton R. Ivanov >> Cambridgegreys Limited. Registered in England. Company Number 10273661 >> https://www.cambridgegreys.com/ > > > _______________________________________________ > linux-um mailing list > linux-um at lists.infradead.org > http://lists.infradead.org/mailman/listinfo/linux-um >-- Anton R. Ivanov Cambridgegreys Limited. Registered in England. Company Number 10273661 https://www.cambridgegreys.com/
Jason Wang
2020-Feb-13 03:31 UTC
[PATCH] virtio: Work around frames incorrectly marked as gso
On 2020/2/13 ??1:38, Anton Ivanov wrote:> > > On 11/02/2020 10:37, Michael S. Tsirkin wrote: >> On Tue, Feb 11, 2020 at 07:42:37AM +0000, Anton Ivanov wrote: >>> On 11/02/2020 02:51, Jason Wang wrote: >>>> >>>> On 2020/2/11 ??12:55, Anton Ivanov wrote: >>>>> >>>>> >>>>> On 09/12/2019 10:48, anton.ivanov at cambridgegreys.com wrote: >>>>>> From: Anton Ivanov <anton.ivanov at cambridgegreys.com> >>>>>> >>>>>> Some of the frames marked as GSO which arrive at >>>>>> virtio_net_hdr_from_skb() have no GSO_TYPE, no >>>>>> fragments (data_len = 0) and length significantly shorter >>>>>> than the MTU (752 in my experiments). >>>>>> >>>>>> This is observed on raw sockets reading off vEth interfaces >>>>>> in all 4.x and 5.x kernels I tested. >>>>>> >>>>>> These frames are reported as invalid while they are in fact >>>>>> gso-less frames. >>>>>> >>>>>> This patch marks the vnet header as no-GSO for them instead >>>>>> of reporting it as invalid. >>>>>> >>>>>> Signed-off-by: Anton Ivanov <anton.ivanov at cambridgegreys.com> >>>>>> --- >>>>>> ?? include/linux/virtio_net.h | 8 ++++++-- >>>>>> ?? 1 file changed, 6 insertions(+), 2 deletions(-) >>>>>> >>>>>> diff --git a/include/linux/virtio_net.h b/include/linux/virtio_net.h >>>>>> index 0d1fe9297ac6..d90d5cff1b9a 100644 >>>>>> --- a/include/linux/virtio_net.h >>>>>> +++ b/include/linux/virtio_net.h >>>>>> @@ -112,8 +112,12 @@ static inline int >>>>>> virtio_net_hdr_from_skb(const struct sk_buff *skb, >>>>>> ?????????????? hdr->gso_type = VIRTIO_NET_HDR_GSO_TCPV4; >>>>>> ?????????? else if (sinfo->gso_type & SKB_GSO_TCPV6) >>>>>> ?????????????? hdr->gso_type = VIRTIO_NET_HDR_GSO_TCPV6; >>>>>> -??????? else >>>>>> -??????????? return -EINVAL; >>>>>> +??????? else { >>>>>> +??????????? if (skb->data_len == 0) >>>>>> +??????????????? hdr->gso_type = VIRTIO_NET_HDR_GSO_NONE; >>>>>> +??????????? else >>>>>> +??????????????? return -EINVAL; >>>>>> +??????? } >>>>>> ?????????? if (sinfo->gso_type & SKB_GSO_TCP_ECN) >>>>>> ?????????????? hdr->gso_type |= VIRTIO_NET_HDR_GSO_ECN; >>>>>> ?????? } else >>>>>> >>>>> >>>>> ping. >>>>> >>>> >>>> Do you mean gso_size is set but gso_type is not? Looks like a bug >>>> elsewhere. >>>> >>>> Thanks >>>> >>>> >>> Yes. >>> >>> I could not trace it where it is coming from. >>> >>> I see it when doing recvmmsg on raw sockets in the UML vector network >>> drivers. >>> >> >> I think we need to find the culprit and fix it there, lots of other >> things >> can break otherwise. >> Just printing out skb->dev->name should do the trick, no? > > The printk in virtio_net_hdr_from_skb says NULL. > > That is probably normal for a locally originated frame. > > I cannot reproduce this with network traffic by the way - it happens > only if the traffic is locally originated on the host. > > A,Or maybe you can try add dump_stack() there. Thanks
Michael S. Tsirkin
2020-Feb-13 10:00 UTC
[PATCH] virtio: Work around frames incorrectly marked as gso
On Wed, Feb 12, 2020 at 05:38:09PM +0000, Anton Ivanov wrote:> > > On 11/02/2020 10:37, Michael S. Tsirkin wrote: > > On Tue, Feb 11, 2020 at 07:42:37AM +0000, Anton Ivanov wrote: > > > On 11/02/2020 02:51, Jason Wang wrote: > > > > > > > > On 2020/2/11 ??12:55, Anton Ivanov wrote: > > > > > > > > > > > > > > > On 09/12/2019 10:48, anton.ivanov at cambridgegreys.com wrote: > > > > > > From: Anton Ivanov <anton.ivanov at cambridgegreys.com> > > > > > > > > > > > > Some of the frames marked as GSO which arrive at > > > > > > virtio_net_hdr_from_skb() have no GSO_TYPE, no > > > > > > fragments (data_len = 0) and length significantly shorter > > > > > > than the MTU (752 in my experiments). > > > > > > > > > > > > This is observed on raw sockets reading off vEth interfaces > > > > > > in all 4.x and 5.x kernels I tested. > > > > > > > > > > > > These frames are reported as invalid while they are in fact > > > > > > gso-less frames. > > > > > > > > > > > > This patch marks the vnet header as no-GSO for them instead > > > > > > of reporting it as invalid. > > > > > > > > > > > > Signed-off-by: Anton Ivanov <anton.ivanov at cambridgegreys.com> > > > > > > --- > > > > > > ? include/linux/virtio_net.h | 8 ++++++-- > > > > > > ? 1 file changed, 6 insertions(+), 2 deletions(-) > > > > > > > > > > > > diff --git a/include/linux/virtio_net.h b/include/linux/virtio_net.h > > > > > > index 0d1fe9297ac6..d90d5cff1b9a 100644 > > > > > > --- a/include/linux/virtio_net.h > > > > > > +++ b/include/linux/virtio_net.h > > > > > > @@ -112,8 +112,12 @@ static inline int > > > > > > virtio_net_hdr_from_skb(const struct sk_buff *skb, > > > > > > ????????????? hdr->gso_type = VIRTIO_NET_HDR_GSO_TCPV4; > > > > > > ????????? else if (sinfo->gso_type & SKB_GSO_TCPV6) > > > > > > ????????????? hdr->gso_type = VIRTIO_NET_HDR_GSO_TCPV6; > > > > > > -??????? else > > > > > > -??????????? return -EINVAL; > > > > > > +??????? else { > > > > > > +??????????? if (skb->data_len == 0) > > > > > > +??????????????? hdr->gso_type = VIRTIO_NET_HDR_GSO_NONE; > > > > > > +??????????? else > > > > > > +??????????????? return -EINVAL; > > > > > > +??????? } > > > > > > ????????? if (sinfo->gso_type & SKB_GSO_TCP_ECN) > > > > > > ????????????? hdr->gso_type |= VIRTIO_NET_HDR_GSO_ECN; > > > > > > ????? } else > > > > > > > > > > > > > > > > ping. > > > > > > > > > > > > > Do you mean gso_size is set but gso_type is not? Looks like a bug > > > > elsewhere. > > > > > > > > Thanks > > > > > > > > > > > Yes. > > > > > > I could not trace it where it is coming from. > > > > > > I see it when doing recvmmsg on raw sockets in the UML vector network > > > drivers. > > > > > > > I think we need to find the culprit and fix it there, lots of other things > > can break otherwise. > > Just printing out skb->dev->name should do the trick, no? > > The printk in virtio_net_hdr_from_skb says NULL. > > That is probably normal for a locally originated frame. > > I cannot reproduce this with network traffic by the way - it happens only if the traffic is locally originated on the host. > > A,OK so is it code in __tcp_transmit_skb that sets gso_size to non-null when gso_type is 0?> > > > > > > -- > > > Anton R. Ivanov > > > Cambridgegreys Limited. Registered in England. Company Number 10273661 > > > https://www.cambridgegreys.com/ > > > > > > _______________________________________________ > > linux-um mailing list > > linux-um at lists.infradead.org > > http://lists.infradead.org/mailman/listinfo/linux-um > > > > -- > Anton R. Ivanov > Cambridgegreys Limited. Registered in England. Company Number 10273661 > https://www.cambridgegreys.com/
Possibly Parallel Threads
- [PATCH] virtio: Work around frames incorrectly marked as gso
- [PATCH] virtio: Work around frames incorrectly marked as gso
- [PATCH] virtio: Work around frames incorrectly marked as gso
- [PATCH] virtio: Work around frames incorrectly marked as gso
- [PATCH] virtio: Work around frames incorrectly marked as gso