Linux Virtualization - Jun 2013 - [PATCH 0/2] fix kernel crash with macvtap on top of LRO

On Wed, Jun 12, 2013 at 03:56:36PM +0100, Ben Hutchings
wrote:
> On Mon, 2013-06-10 at 10:07 +0300, Michael S. Tsirkin wrote:
> > On Thu, Feb 07, 2013 at 01:14:20PM -0500, David Miller wrote:
> > > From: Ben Hutchings <bhutchings at solarflare.com>
> > > Date: Thu, 7 Feb 2013 16:20:46 +0000
> > > 
> > > > If the consensus is still that we must preserve packets
exactly (aside
> > > > from the usual modifications by IP routers) then LRO should
be disabled
> > > > on all devices for which forwarding is enabled.
> > > 
> > > I believe this is still undoubtedly the consensus.
> > 
> > With virtio we are getting packets from a linux host,
> > so we could thinkably preserve packets exactly
> > even with LRO. I am guessing other hardware could be
> > doing this as well.
> > 
> > I am not sure what information would need to be preserved -
> > could someone help clarify please?
> 
> Some LRO implementations may not preserve:
> 
> - Packet boundaries
>   - TSO/GSO produces packets all the same size, except possibly for the
>     last one.  GRO therefore flushes a flow after merging a packet with
>     a different segment size.
> - IPv4 TTL, IPv6 hop-limit, TCP timestamp
>   - TSO/GSO will put the same values in all packets.  GRO flushes a flow
>     if they change.
> - IPv4 fragment ID
>   - TSO/GSO produces consecutive fragment IDs.  GRO flushes a flow
>     if it sees a non-consecutive fragment ID.
> - MAC header, IPv4 TOS, IPv6 traffic class
>   - Should be the same for all packets in a flow.  GRO actually checks
>     and flushes a flow if they change.
> 
> Ben.
Thanks!

Okay so since LRO in virtio is actually running on top
of GRO/TSO in linux, looks like the only things worth preserving
that we don't preserve at the moment are the packet
boundaries, for which it's enough to report
the first segment size to guest - we have the total length,
from that we can calculate the last segment size.
> -- 
> Ben Hutchings, Staff Engineer, Solarflare
> Not speaking for my employer; that's the marketing department's
job.
> They asked us to note that Solarflare product names are trademarked.

"Michael S. Tsirkin" <mst at redhat.com>
writes:
> On Wed, Jun 12, 2013 at 03:56:36PM +0100, Ben Hutchings wrote:
>> On Mon, 2013-06-10 at 10:07 +0300, Michael S. Tsirkin wrote:
>> > On Thu, Feb 07, 2013 at 01:14:20PM -0500, David Miller wrote:
>> > > From: Ben Hutchings <bhutchings at solarflare.com>
>> > > Date: Thu, 7 Feb 2013 16:20:46 +0000
>> > > 
>> > > > If the consensus is still that we must preserve packets
exactly (aside
>> > > > from the usual modifications by IP routers) then LRO
should be disabled
>> > > > on all devices for which forwarding is enabled.
>> > > 
>> > > I believe this is still undoubtedly the consensus.
>> > 
>> > With virtio we are getting packets from a linux host,
>> > so we could thinkably preserve packets exactly
>> > even with LRO. I am guessing other hardware could be
>> > doing this as well.
>> > 
>> > I am not sure what information would need to be preserved -
>> > could someone help clarify please?
>> 
>> Some LRO implementations may not preserve:
>> 
>> - Packet boundaries
>>   - TSO/GSO produces packets all the same size, except possibly for the
>>     last one.  GRO therefore flushes a flow after merging a packet with
>>     a different segment size.
>> - IPv4 TTL, IPv6 hop-limit, TCP timestamp
>>   - TSO/GSO will put the same values in all packets.  GRO flushes a
flow
>>     if they change.
>> - IPv4 fragment ID
>>   - TSO/GSO produces consecutive fragment IDs.  GRO flushes a flow
>>     if it sees a non-consecutive fragment ID.
>> - MAC header, IPv4 TOS, IPv6 traffic class
>>   - Should be the same for all packets in a flow.  GRO actually checks
>>     and flushes a flow if they change.
>> 
>> Ben.
>
> Thanks!
>
> Okay so since LRO in virtio is actually running on top
> of GRO/TSO in linux, looks like the only things worth preserving
> that we don't preserve at the moment are the packet
> boundaries, for which it's enough to report
> the first segment size to guest - we have the total length,
> from that we can calculate the last segment size.
Isn't this reflected in virtio_net_hdr->gso_size?

But the bigger point here is that we shouldn't be feeding guests LRO
packets which don't meet the stricter GRO requirements, as we don't know
what the guest is doing with them.  It might be forwarding them itself.

I thought LRO was deprecated and GRO was the new hotness, but I haven't
been following.  Do we still care about LRO?

Cheers,
Rusty.

On Mon, 2013-06-17 at 11:05 +0930, Rusty Russell wrote:
> "Michael S. Tsirkin" <mst at redhat.com> writes:
> > On Wed, Jun 12, 2013 at 03:56:36PM +0100, Ben Hutchings wrote:
> >> On Mon, 2013-06-10 at 10:07 +0300, Michael S. Tsirkin wrote:
> >> > On Thu, Feb 07, 2013 at 01:14:20PM -0500, David Miller wrote:
> >> > > From: Ben Hutchings <bhutchings at solarflare.com>
> >> > > Date: Thu, 7 Feb 2013 16:20:46 +0000
> >> > > 
> >> > > > If the consensus is still that we must preserve
packets exactly (aside
> >> > > > from the usual modifications by IP routers) then
LRO should be disabled
> >> > > > on all devices for which forwarding is enabled.
> >> > > 
> >> > > I believe this is still undoubtedly the consensus.
> >> > 
> >> > With virtio we are getting packets from a linux host,
> >> > so we could thinkably preserve packets exactly
> >> > even with LRO. I am guessing other hardware could be
> >> > doing this as well.
> >> > 
> >> > I am not sure what information would need to be preserved -
> >> > could someone help clarify please?
> >> 
> >> Some LRO implementations may not preserve:
> >> 
> >> - Packet boundaries
> >>   - TSO/GSO produces packets all the same size, except possibly
for the
> >>     last one.  GRO therefore flushes a flow after merging a packet
with
> >>     a different segment size.
> >> - IPv4 TTL, IPv6 hop-limit, TCP timestamp
> >>   - TSO/GSO will put the same values in all packets.  GRO flushes
a flow
> >>     if they change.
> >> - IPv4 fragment ID
> >>   - TSO/GSO produces consecutive fragment IDs.  GRO flushes a flow
> >>     if it sees a non-consecutive fragment ID.
> >> - MAC header, IPv4 TOS, IPv6 traffic class
> >>   - Should be the same for all packets in a flow.  GRO actually
checks
> >>     and flushes a flow if they change.
> >> 
> >> Ben.
> >
> > Thanks!
> >
> > Okay so since LRO in virtio is actually running on top
> > of GRO/TSO in linux, looks like the only things worth preserving
> > that we don't preserve at the moment are the packet
> > boundaries, for which it's enough to report
> > the first segment size to guest - we have the total length,
> > from that we can calculate the last segment size.
> 
> Isn't this reflected in virtio_net_hdr->gso_size?
> 
> But the bigger point here is that we shouldn't be feeding guests LRO
> packets which don't meet the stricter GRO requirements, as we don't
know
> what the guest is doing with them.  It might be forwarding them itself.
> 
> I thought LRO was deprecated and GRO was the new hotness, but I haven't
> been following.  Do we still care about LRO?
The old software LRO implementation, inet_lro, is deprecated in favour
of GRO and is now only used by one or two drivers.  Hardware/firmware
implementations of LRO are still in use and not deprecated, but we try
to disable them on devices for which forwarding is enabled because of
this information loss.

The problem I was talking about is this: you can put macvlan on top of a
device that has LRO enabled, and then if the macvtap/macvlan device is
used for forwarding the output packets might not look the same as those
originally received.  So LRO should be disabled on the underlying device
whenever forwarding is enabled on the macvtap/macvlan device; however we
can't necessarily tell when that happens as the forwarding might be done
inside a VM.  Maybe this is just too obscure a use case to worry much
about getting it right automatically.

Ben.

-- 
Ben Hutchings, Staff Engineer, Solarflare
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.