On Wed, Jun 15, 2022 at 8:10 AM Parav Pandit <parav at nvidia.com>
wrote:>
>
>
> > From: Jason Wang <jasowang at redhat.com>
> > Sent: Wednesday, June 1, 2022 11:54 PM
> >
> > On Thu, Jun 2, 2022 at 10:59 AM Parav Pandit <parav at
nvidia.com> wrote:
> > >
> > >
> > > > From: Jason Wang <jasowang at redhat.com>
> > > > Sent: Wednesday, June 1, 2022 10:00 PM
> > > >
> > > > On Thu, Jun 2, 2022 at 2:58 AM Parav Pandit <parav at
nvidia.com> wrote:
> > > > >
> > > > >
> > > > > > From: Jason Wang <jasowang at redhat.com>
> > > > > > Sent: Tuesday, May 31, 2022 10:42 PM
> > > > > >
> > > > > > Well, the ability to query the virtqueue state was
proposed as
> > > > > > another feature (Eugenio, please correct me). This
should be
> > > > > > sufficient for making virtio-net to be live
migrated.
> > > > > >
> > > > > The device is stopped, it won't answer to this
special vq config done
> > here.
> > > >
> > > > This depends on the definition of the stop. Any query to the
device
> > > > state should be allowed otherwise it's meaningless for
us.
> > > >
> > > > > Programming all of these using cfg registers
doesn't scale for
> > > > > on-chip
> > > > memory and for the speed.
> > > >
> > > > Well, they are orthogonal and what I want to say is, we
should first
> > > > define the semantics of stop and state of the virtqueue.
> > > >
> > > > Such a facility could be accessed by either transport
specific
> > > > method or admin virtqueue, it totally depends on the
hardware
> > architecture of the vendor.
> > > >
> > > I find it hard to believe that a vendor can implement a CVQ but
not AQ and
> > chose to expose tens of hundreds of registers.
> > > But maybe, it fits some specific hw.
> >
> > You can have a look at the ifcvf dpdk driver as an example.
> >
> Ifcvf is an example of using registers.
> It is not an answer why AQ is hard for it. :)
Well, it's an example of how vDPA is implemented. I think we agree
that for vDPA, vendors have the flexibility to implement their
perferrable datapath.
> virtio spec has definition of queue now and implementing yet another queue
shouldn't be a problem.
>
> So far no one seem to have problem with the additional queue.
> So I take it as AQ is ok.
>
> > But another thing that is unrelated to hardware architecture is the
nesting
> > support. Having admin virtqueue in a nesting environment looks like an
> > overkill. Presenting a register in L1 and map it to L0's admin
should be good
> > enough.
> So may be a optimized interface can be added that fits nested env.
> At this point in time real users that we heard are interested in non-nested
use cases. Let's enable them first.
That's fine. For nests, it's actually really easy, just adding an
interface within the existing transport should be sufficient.
>
>
> >
> > >
> > > I like to learn the advantages of such method other than
simplicity.
> > >
> > > We can clearly that we are shifting away from such PCI registers
with SIOV,
> > IMS and other scalable solutions.
> > > virtio drifting in reverse direction by introducing more
registers as
> > transport.
> > > I expect it to an optional transport like AQ.
> >
> > Actually, I had a proposal of using admin virtqueue as a transport,
it's
> > designed to be SIOV/IMS capable. And it's not hard to extend it
with the
> > state/stop support etc.
> >
> > >
> > > > >
> > > > > Next would be to program hundreds of statistics of the
64 VQs
> > > > > through a
> > > > giant PCI config space register in some busy polling scheme.
> > > >
> > > > We don't need giant config space, and this method has
been
> > > > implemented by some vDPA vendors.
> > > >
> > > There are tens of 64-bit counters per VQs. These needs to
programmed on
> > destination side.
> > > Programming these via registers requires exposing them on the
registers.
> > > In one of the proposals, I see them being queried via CVQ from
the device.
> >
> > I didn't see a proposal like this. And I don't think querying
general virtio state
> > like idx with a device specific CVQ is a good design.
> >
> My example was not for the idx. But for VQ statistics that is queried via
CVQ.
>
> > >
> > > Programming them via cfg registers requires large cfg space or
synchronous
> > programming until receiving ACK from it.
> > > This means one entry at a time...
> > >
> > > Programming them via CVQ needs replicate and align cmd values etc
on all
> > device types. All duplicate and hard to maintain.
> > >
> > >
> > > > >
> > > > > I can clearly see how all these are inefficient for
faster LM.
> > > > > We need an efficient AQ to proceed with at minimum.
> > > >
> > > > I'm fine with admin virtqueue, but the stop and state
are orthogonal to
> > that.
> > > > And using admin virtqueue for stop/state will be more
natural if we
> > > > use admin virtqueue as a transport.
> > > Ok.
> > > We should have defined it bit earlier that all vendors can use.
:(
> >
> > I agree.
>
> I remember few months back, you acked in the weekly meeting that TC has
approved the AQ direction.
> And we are still in this circle of debating the AQ.
I think not. Just to make sure we are on the same page, the proposal
here is for vDPA, and hope it can provide forward compatibility to
virtio. So in the context of vDPA, admin virtqueue is not a must.
Thanks