> So, let's add some funky flags in virtio device to block out > features, have core compare these before and after, > detect change, reset and retry?In the virtnet case, we'll decide which features to block based on the ring size. 2 < ring < MAX_FRAGS + 2 -> BLOCK GRO + MRG_RXBUF ring < 2 -> BLOCK GRO + MRG_RXBUF + CTRL_VQ So we'll need a new virtio callback instead of flags. Furthermore, other virtio drivers may decide which features to block based on parameters different than ring size (I don't have a good example at the moment). So maybe we should leave it to the driver to handle (during probe), and offer a virtio core function to re-negotiate the features? In the solution I'm working on, I expose a new virtio core function that resets the device and renegotiates the received features. + A new virtio_config_ops callback peek_vqs_len to peek at the VQ lengths before calling find_vqs. (The callback must be called after the features negotiation) So, the flow is something like: * Super early in virtnet probe, we peek at the VQ lengths and decide if we are using small vrings, if so, we reset and renegotiate the features. * We continue normally and create the VQs. * We check if the created rings are small. If they are and some blocked features were negotiated anyway (may occur if the re-negotiation fails, or if the transport has no implementation for peek_vqs_len), we fail probe. If the ring is small and the features are ok, we mark the virtnet device as vring_small and fixup some variables. peek_vqs_len is needed because we must know the VQ length before calling init_vqs. During virtnet_find_vqs we check the following: vi->has_cvq vi->big_packets vi->mergeable_rx_bufs But these will change if the ring is small.. (Of course, another solution will be to re-negotiate features after init_vqs, but this will make a big mess, tons of things to clean and reconfigure) The 2 < ring < MAX_FRAGS + 2 part is ready, I have tested a few cases and it is working. I'm considering splitting the effort into 2 series. A 2 < ring < MAX_FRAGS + 2 series, and a follow up series with the ring < 2 case. I'm also thinking about sending the first series as an RFC soon, so it will be more broadly tested. What do you think?
> > So, let's add some funky flags in virtio device to block out > > features, have core compare these before and after, > > detect change, reset and retry? > > In the virtnet case, we'll decide which features to block based on the ring size. > 2 < ring < MAX_FRAGS + 2 -> BLOCK GRO + MRG_RXBUF > ring < 2 -> BLOCK GRO + MRG_RXBUF + CTRL_VQ > > So we'll need a new virtio callback instead of flags. > > Furthermore, other virtio drivers may decide which features to block based on parameters different than ring size (I don't have a good example at the moment). > So maybe we should leave it to the driver to handle (during probe), > and offer a virtio core function to re-negotiate the features? > > In the solution I'm working on, I expose a new virtio core function that resets the device and renegotiates the received features. > + A new virtio_config_ops callback peek_vqs_len to peek at the VQ lengths before calling find_vqs. (The callback must be called after the features negotiation) > > So, the flow is something like: > > * Super early in virtnet probe, we peek at the VQ lengths and decide if we are > using small vrings, if so, we reset and renegotiate the features. > * We continue normally and create the VQs. > * We check if the created rings are small. > If they are and some blocked features were negotiated anyway (may occur if > the re-negotiation fails, or if the transport has no implementation for > peek_vqs_len), we fail probe.Small fix: if the re-negotiation fails, we fail probe immediately. The only way to negotiate blocked features with a small vring is if the transport has no implementation for peek_vqs_len.> If the ring is small and the features are ok, we mark the virtnet device as > vring_small and fixup some variables. > > > peek_vqs_len is needed because we must know the VQ length before calling init_vqs. > > During virtnet_find_vqs we check the following: > vi->has_cvq > vi->big_packets > vi->mergeable_rx_bufs > > But these will change if the ring is small.. > > (Of course, another solution will be to re-negotiate features after init_vqs, but this will make a big mess, tons of things to clean and reconfigure) > > > The 2 < ring < MAX_FRAGS + 2 part is ready, I have tested a few cases and it is working. > > I'm considering splitting the effort into 2 series. > A 2 < ring < MAX_FRAGS + 2 series, and a follow up series with the ring < 2 case. > > I'm also thinking about sending the first series as an RFC soon, so it will be more broadly tested. > > What do you think? >
On Tue, Apr 25, 2023 at 09:41:35AM +0000, Alvaro Karsz wrote:> > So, let's add some funky flags in virtio device to block out > > features, have core compare these before and after, > > detect change, reset and retry? > > In the virtnet case, we'll decide which features to block based on the ring size. > 2 < ring < MAX_FRAGS + 2 -> BLOCK GRO + MRG_RXBUF > ring < 2 -> BLOCK GRO + MRG_RXBUF + CTRL_VQwhy MRG_RXBUF? what does it matter?> So we'll need a new virtio callback instead of flags. > Furthermore, other virtio drivers may decide which features to block based on parameters different than ring size (I don't have a good example at the moment). > So maybe we should leave it to the driver to handle (during probe), and offer a virtio core function to re-negotiate the features? > > In the solution I'm working on, I expose a new virtio core function that resets the device and renegotiates the received features. > + A new virtio_config_ops callback peek_vqs_len to peek at the VQ lengths before calling find_vqs. (The callback must be called after the features negotiation) > > So, the flow is something like: > > * Super early in virtnet probe, we peek at the VQ lengths and decide if we are > using small vrings, if so, we reset and renegotiate the features.Using which APIs? What does peek_vqs_len do and why does it matter that it is super early?> * We continue normally and create the VQs. > * We check if the created rings are small. > If they are and some blocked features were negotiated anyway (may occur if > the re-negotiation fails, or if the transport has no implementation for > peek_vqs_len), we fail probe. > If the ring is small and the features are ok, we mark the virtnet device as > vring_small and fixup some variables. > > > peek_vqs_len is needed because we must know the VQ length before calling init_vqs. > > During virtnet_find_vqs we check the following: > vi->has_cvq > vi->big_packets > vi->mergeable_rx_bufs > > But these will change if the ring is small.. > > (Of course, another solution will be to re-negotiate features after init_vqs, but this will make a big mess, tons of things to clean and reconfigure) > > > The 2 < ring < MAX_FRAGS + 2 part is ready, I have tested a few cases and it is working. > > I'm considering splitting the effort into 2 series. > A 2 < ring < MAX_FRAGS + 2 series, and a follow up series with the ring < 2 case. > > I'm also thinking about sending the first series as an RFC soon, so it will be more broadly tested. > > What do you think?Lots of work spilling over to transports. And I especially don't like that it slows down boot on good path. I have the following idea: - add a blocked features value in virtio_device - before calling probe, core saves blocked features - if probe fails, checks blocked features. if any were added, reset, negotiate all features except blocked ones and do the validate/probe dance again This will mean mostly no changes to drivers: just check condition, block feature and fail probe. -- MST