Linux Virtualization - Jun 2013 - [PATCH RFC] virtio-pci: new config layout: using memory BAR

On Mon, Jun 03, 2013 at 09:56:15AM +0930, Rusty Russell
wrote:
> "Michael S. Tsirkin" <mst at redhat.com> writes:
> > On Thu, May 30, 2013 at 08:53:45AM -0500, Anthony Liguori wrote:
> >> Rusty Russell <rusty at rustcorp.com.au> writes:
> >> 
> >> > Anthony Liguori <aliguori at us.ibm.com> writes:
> >> >> Forcing a guest driver change is a really big
> >> >> deal and I see no reason to do that unless there's a
compelling reason
> >> >> to.
> >> >>
> >> >> So we're stuck with the 1.0 config layout for a very
long time.
> >> >
> >> > We definitely must not force a guest change.  The explicit
aim of the
> >> > standard is that "legacy" and 1.0 be backward
compatible.  One
> >> > deliverable is a document detailing how this is done
(effectively a
> >> > summary of changes between what we have and 1.0).
> >> 
> >> If 2.0 is fully backwards compatible, great.  It seems like such a
> >> difference that that would be impossible but I need to investigate
> >> further.
> >> 
> >> Regards,
> >> 
> >> Anthony Liguori
> >
> > If you look at my patches you'll see how it works.
> > Basically old guests use BAR0 new ones don't, so
> > it's easy: BAR0 access means legacy guest.
> > Only started testing but things seem to work
> > fine with old guests so far.
> >
> > I think we need a spec, not just driver code.
> >
> > Rusty what's the plan? Want me to write it?
> 
> We need both, of course, but the spec work will happen in the OASIS WG.
> A draft is good, but let's not commit anything to upstream QEMU until
we
> get the spec finalized.  And that is proposed to be late this year.
Well that would be quite sad really.

This means we can't make virtio a spec compliant pci express device,
and we can't add any more feature bits, so no
flexible buffer optimizations for virtio net.

There are probably more projects that will be blocked.

So how about we keep extending legacy layout for a bit longer:
- add a way to access device with MMIO
- use feature bit 31 to signal 64 bit features
  (and shift device config accordingly)

No endian-ness rework, no per queue enable etc.


Then when we start discussions we will have a working
express and working 64 bit feature support,
and it will be that much easier to make it pretty it.

> Since I'm going to have to reformat the spec and adapt it into OASIS
> style anyway, perhaps you should prepare a description as a standalone
> text document.  Easier to email and work with...
> 
> Now, the idea is that if you want to support 0.9 and 1.0 (or whatever we
> call them; I used the term "legacy" for existing implementations
in the
> OASIS WG proposal), you add capabilities and don't point them into (the
> start of?) BAR0.  Old drivers use BAR0 as now.
> 
> One trick to note: while drivers shouldn't use both old and new style
on
> the same device, you need to allow it for kexec, particularly reset via
> BAR0.
> 
> Cheers,
> Rusty.

"Michael S. Tsirkin" <mst at redhat.com>
writes:
> On Mon, Jun 03, 2013 at 09:56:15AM +0930, Rusty Russell wrote:
>> "Michael S. Tsirkin" <mst at redhat.com> writes:
>> > On Thu, May 30, 2013 at 08:53:45AM -0500, Anthony Liguori wrote:
>> >> Rusty Russell <rusty at rustcorp.com.au> writes:
>> >> 
>> >> > Anthony Liguori <aliguori at us.ibm.com> writes:
>> >> >> Forcing a guest driver change is a really big
>> >> >> deal and I see no reason to do that unless
there's a compelling reason
>> >> >> to.
>> >> >>
>> >> >> So we're stuck with the 1.0 config layout for a
very long time.
>> >> >
>> >> > We definitely must not force a guest change.  The
explicit aim of the
>> >> > standard is that "legacy" and 1.0 be backward
compatible.  One
>> >> > deliverable is a document detailing how this is done
(effectively a
>> >> > summary of changes between what we have and 1.0).
>> >> 
>> >> If 2.0 is fully backwards compatible, great.  It seems like
such a
>> >> difference that that would be impossible but I need to
investigate
>> >> further.
>> >> 
>> >> Regards,
>> >> 
>> >> Anthony Liguori
>> >
>> > If you look at my patches you'll see how it works.
>> > Basically old guests use BAR0 new ones don't, so
>> > it's easy: BAR0 access means legacy guest.
>> > Only started testing but things seem to work
>> > fine with old guests so far.
>> >
>> > I think we need a spec, not just driver code.
>> >
>> > Rusty what's the plan? Want me to write it?
>> 
>> We need both, of course, but the spec work will happen in the OASIS WG.
>> A draft is good, but let's not commit anything to upstream QEMU
until we
>> get the spec finalized.  And that is proposed to be late this year.
>
> Well that would be quite sad really.
> 
> This means we can't make virtio a spec compliant pci express device,
> and we can't add any more feature bits, so no
> flexible buffer optimizations for virtio net.
>
> There are probably more projects that will be blocked.
>
> So how about we keep extending legacy layout for a bit longer:
> - add a way to access device with MMIO
> - use feature bit 31 to signal 64 bit features
>   (and shift device config accordingly)
By my count, net still has 7 feature bits left, so I don't think the
feature bits are likely to be a limitation in the next 6 months?

MMIO is a bigger problem.  Linux guests are happy with it: does it break
the Windows drivers?

Thanks,
Rusty.

On Tue, Jun 04, 2013 at 03:01:50PM +0930, Rusty Russell
wrote:
> "Michael S. Tsirkin" <mst at redhat.com> writes:
> > On Mon, Jun 03, 2013 at 09:56:15AM +0930, Rusty Russell wrote:
> >> "Michael S. Tsirkin" <mst at redhat.com> writes:
> >> > On Thu, May 30, 2013 at 08:53:45AM -0500, Anthony Liguori
wrote:
> >> >> Rusty Russell <rusty at rustcorp.com.au> writes:
> >> >> 
> >> >> > Anthony Liguori <aliguori at us.ibm.com>
writes:
> >> >> >> Forcing a guest driver change is a really big
> >> >> >> deal and I see no reason to do that unless
there's a compelling reason
> >> >> >> to.
> >> >> >>
> >> >> >> So we're stuck with the 1.0 config layout
for a very long time.
> >> >> >
> >> >> > We definitely must not force a guest change.  The
explicit aim of the
> >> >> > standard is that "legacy" and 1.0 be
backward compatible.  One
> >> >> > deliverable is a document detailing how this is done
(effectively a
> >> >> > summary of changes between what we have and 1.0).
> >> >> 
> >> >> If 2.0 is fully backwards compatible, great.  It seems
like such a
> >> >> difference that that would be impossible but I need to
investigate
> >> >> further.
> >> >> 
> >> >> Regards,
> >> >> 
> >> >> Anthony Liguori
> >> >
> >> > If you look at my patches you'll see how it works.
> >> > Basically old guests use BAR0 new ones don't, so
> >> > it's easy: BAR0 access means legacy guest.
> >> > Only started testing but things seem to work
> >> > fine with old guests so far.
> >> >
> >> > I think we need a spec, not just driver code.
> >> >
> >> > Rusty what's the plan? Want me to write it?
> >> 
> >> We need both, of course, but the spec work will happen in the
OASIS WG.
> >> A draft is good, but let's not commit anything to upstream
QEMU until we
> >> get the spec finalized.  And that is proposed to be late this
year.
> >
> > Well that would be quite sad really.
> > 
> > This means we can't make virtio a spec compliant pci express
device,
> > and we can't add any more feature bits, so no
> > flexible buffer optimizations for virtio net.
> >
> > There are probably more projects that will be blocked.
> >
> > So how about we keep extending legacy layout for a bit longer:
> > - add a way to access device with MMIO
> > - use feature bit 31 to signal 64 bit features
> >   (and shift device config accordingly)
> 
> By my count, net still has 7 feature bits left, so I don't think the
> feature bits are likely to be a limitation in the next 6 months?
Yes but you wanted a generic transport feature bit
for flexible SG layout.
Are you happy with VIRTIO_NET_F_ANY_HEADER_SG then?
> MMIO is a bigger problem.  Linux guests are happy with it: does it break
> the Windows drivers?
> 
> Thanks,
> Rusty.
You mean make BAR0 an MMIO BAR?
Yes, it would break current windows guests.
Further, as long as we use same address to notify all queues,
we would also need to decode the instruction on x86 and that's
measureably slower than PIO.
We could go back to discussing hypercall use for notifications,
but that has its own set of issues...

-- 
MST

On Tue, Jun 04, 2013 at 03:01:50PM +0930, Rusty Russell
wrote:
> "Michael S. Tsirkin" <mst at redhat.com> writes:
> > On Mon, Jun 03, 2013 at 09:56:15AM +0930, Rusty Russell wrote:
> >> "Michael S. Tsirkin" <mst at redhat.com> writes:
> >> > On Thu, May 30, 2013 at 08:53:45AM -0500, Anthony Liguori
wrote:
> >> >> Rusty Russell <rusty at rustcorp.com.au> writes:
> >> >> 
> >> >> > Anthony Liguori <aliguori at us.ibm.com>
writes:
> >> >> >> Forcing a guest driver change is a really big
> >> >> >> deal and I see no reason to do that unless
there's a compelling reason
> >> >> >> to.
> >> >> >>
> >> >> >> So we're stuck with the 1.0 config layout
for a very long time.
> >> >> >
> >> >> > We definitely must not force a guest change.  The
explicit aim of the
> >> >> > standard is that "legacy" and 1.0 be
backward compatible.  One
> >> >> > deliverable is a document detailing how this is done
(effectively a
> >> >> > summary of changes between what we have and 1.0).
> >> >> 
> >> >> If 2.0 is fully backwards compatible, great.  It seems
like such a
> >> >> difference that that would be impossible but I need to
investigate
> >> >> further.
> >> >> 
> >> >> Regards,
> >> >> 
> >> >> Anthony Liguori
> >> >
> >> > If you look at my patches you'll see how it works.
> >> > Basically old guests use BAR0 new ones don't, so
> >> > it's easy: BAR0 access means legacy guest.
> >> > Only started testing but things seem to work
> >> > fine with old guests so far.
> >> >
> >> > I think we need a spec, not just driver code.
> >> >
> >> > Rusty what's the plan? Want me to write it?
> >> 
> >> We need both, of course, but the spec work will happen in the
OASIS WG.
> >> A draft is good, but let's not commit anything to upstream
QEMU until we
> >> get the spec finalized.  And that is proposed to be late this
year.
> >
> > Well that would be quite sad really.
> > 
> > This means we can't make virtio a spec compliant pci express
device,
> > and we can't add any more feature bits, so no
> > flexible buffer optimizations for virtio net.
> >
> > There are probably more projects that will be blocked.
> >
> > So how about we keep extending legacy layout for a bit longer:
> > - add a way to access device with MMIO
> > - use feature bit 31 to signal 64 bit features
> >   (and shift device config accordingly)
> 
> By my count, net still has 7 feature bits left, so I don't think the
> feature bits are likely to be a limitation in the next 6 months?
Actually I count 5 net specific ones:

3, 4 and then 25, 26, 27

Unless you count 31 even though it's a generic transport bit?

That's still 6 ...
> MMIO is a bigger problem.  Linux guests are happy with it: does it break
> the Windows drivers?
> 
> Thanks,
> Rusty.