Linux Virtualization - Apr 2023 - [PATCH net] virtio-net: reject small vring sizes

On Sun, Apr 23, 2023 at 12:28:49PM +0000, Alvaro Karsz
wrote:
> 
> > > > The rest of stuff can probably just be moved to after
find_vqs without
> > > > much pain.
> > > >
> > > Actually, I think that with a little bit of pain :)
> > > If we use small vrings and a GRO feature bit is set, Linux will
need to allocate 64KB of continuous memory for every receive descriptor..
> > 
> > Oh right. Hmm. Well this is same as big packets though, isn't it?
> > 
> 
> Well, when VIRTIO_NET_F_MRG_RXBUF is not negotiated and one of the GRO
features is, the receive buffers are page size buffers chained together to form
a 64K buffer.
> In this case, do all the chained descriptors actually point to a single
block of continuous memory, or is it possible for the descriptors to point to
pages spread all over?
> 
> > 
> > > Instead of failing probe if GRO/CVQ are set, can we just reset
the device if we discover small vrings and start over?
> > > Can we remember that this device uses small vrings, and then just
avoid negotiating the features that we cannot support?
> > 
> > 
> > We technically can of course. I am just not sure supporting CVQ with
just 1 s/g entry will
> > ever be viable.
> 
> Even if we won't support 1 s/g entry, do we want to fail probe in such
cases?
> We could just disable the CVQ feature (with reset, as suggested before).
> I'm not saying that we should, just raising the option.
> 
So, let's add some funky flags in virtio device to block out
features, have core compare these before and after,
detect change, reset and retry?


-- 
MST

> So, let's add some funky flags in virtio device to block out
> features, have core compare these before and after,
> detect change, reset and retry?
In the virtnet case, we'll decide which features to block based on the ring
size.
2 < ring < MAX_FRAGS + 2  -> BLOCK GRO + MRG_RXBUF
ring < 2  -> BLOCK GRO + MRG_RXBUF + CTRL_VQ

So we'll need a new virtio callback instead of flags.

Furthermore, other virtio drivers may decide which features to block based on
parameters different than ring size (I don't have a good example at the
moment).
So maybe we should leave it to the driver to handle (during probe), and offer a
virtio core function to re-negotiate the features?

In the solution I'm working on, I expose a new virtio core function that
resets the device and renegotiates the received features.
+ A new virtio_config_ops callback peek_vqs_len to peek at the VQ lengths before
calling find_vqs. (The callback must be called after the features negotiation)

So, the flow is something like:

* Super early in virtnet probe, we peek at the VQ lengths and decide if we are 
   using small vrings, if so, we reset and renegotiate the features.
* We continue normally and create the VQs.
* We check if the created rings are small.
   If they are and some blocked features were negotiated anyway (may occur if 
   the re-negotiation fails, or if the transport has no implementation for 
   peek_vqs_len), we fail probe.
   If the ring is small and the features are ok, we mark the virtnet device as 
   vring_small and fixup some variables.
 

peek_vqs_len is needed because we must know the VQ length before calling
init_vqs.

During virtnet_find_vqs we check the following:
vi->has_cvq
vi->big_packets
vi->mergeable_rx_bufs

But these will change if the ring is small..

(Of course, another solution will be to re-negotiate features after init_vqs,
but this will make a big mess, tons of things to clean and reconfigure)


The 2 < ring < MAX_FRAGS + 2 part is ready, I have tested a few cases and
it is working.

I'm considering splitting the effort into 2 series.
A 2 < ring < MAX_FRAGS + 2  series, and a follow up series with the ring
< 2 case.

I'm also thinking about sending the first series as an RFC soon, so it will
be more broadly tested.

What do you think?