Virtualization - Jan 2015 - [PATCH v3 00/16] virtio-pci: towards virtio 1.0 guest support

On Thu, Jan 15, 2015 at 10:18:18PM +0100, Gerd Hoffmann
wrote:
>   Hi,
> 
> > QEMU side is still undergoing polishing, but is already testable.
> 
> Looked art the qemu side for now, commenting here due to qemu patches
> not being posted to the list yet (guess that is at least partly already
> on your todo list):
Absolutely.
>  * Both legacy and modern should be switchable.  Modern needs to be
>    switchable _anyway_ for backward compatibility with old qemu
>    versions.  But legacy should be too, for testing purposes and for
>    new device types which don't need legacy in the first place.
I agree.
>  * I'd like to see some more flexibility in the pci bar layout.  Stuff
>    I have in mind:
>     - New devices which don't need a legacy bar can use bar 0 for
>       modern.
>     - One MMIO bar is enough, we can place both virtio regions and
>       msi-x regions there.  I'd suggest to add msi-x sub-regions to
>       the modern bar.
Why exactly? It seems simpler to separate things, extra BARs
have no cost.
>  * What is the reason for making the modern bar 8M in size?  Looks a bit
>    excessive, given that only 64k or so of that are actually used ...
I use a page per VQ for architectures that can locate the offset of the
accessed page that triggered EPT violation faster than the offset within
page.  I think this is the case for SVM.
> 
> virtio-scsi seems to be broken, at least my usual fedora guest didn't
> boot up from virtio-scsi disk when using a guest kernel with this patch
> series applied.
> 
> cheers,
>   Gerd
I'll re-test. Do other devices work for you? Thanks!

Hi,
> >  * I'd like to see some more flexibility in the pci bar layout. 
Stuff
> >    I have in mind:
> >     - New devices which don't need a legacy bar can use bar 0 for
> >       modern.
> >     - One MMIO bar is enough, we can place both virtio regions and
> >       msi-x regions there.  I'd suggest to add msi-x sub-regions
to
> >       the modern bar.
> 
> Why exactly? It seems simpler to separate things, extra BARs
> have no cost.
Well, there are only six BARs.  legacy bar, modern mmio bar, msi-x bar,
modern io bar (for fast isr).  That already four out of six ...

We have a mmio bar, which we partition into subregions for virtio-1.0
anyway.  Also placing msi-x there is a single msix_init() call.  xhci is
doing that too:

    00000000febf0000-00000000febf3fff (prio 1, RW): xhci
      00000000febf0000-00000000febf003f (prio 0, RW): capabilities
      00000000febf0040-00000000febf043f (prio 0, RW): operational
      00000000febf0440-00000000febf044f (prio 0, RW): usb3 port #1
      00000000febf0450-00000000febf045f (prio 0, RW): usb3 port #2
      00000000febf0460-00000000febf046f (prio 0, RW): usb3 port #3
      00000000febf0470-00000000febf047f (prio 0, RW): usb3 port #4
      00000000febf0480-00000000febf048f (prio 0, RW): usb2 port #1
      00000000febf0490-00000000febf049f (prio 0, RW): usb2 port #2
      00000000febf04a0-00000000febf04af (prio 0, RW): usb2 port #3
      00000000febf04b0-00000000febf04bf (prio 0, RW): usb2 port #4
      00000000febf1000-00000000febf121f (prio 0, RW): runtime
      00000000febf2000-00000000febf281f (prio 0, RW): doorbell
      00000000febf3000-00000000febf30ff (prio 0, RW): msix-table
      00000000febf3800-00000000febf3807 (prio 0, RW): msix-pba

> >  * What is the reason for making the modern bar 8M in size?  Looks a
bit
> >    excessive, given that only 64k or so of that are actually used ...
> 
> I use a page per VQ for architectures that can locate the offset of the
> accessed page that triggered EPT violation faster than the offset within
> page.  I think this is the case for SVM.
8M still looks excessive, given that we typically have a small number of
queues per device.  Do you allocate address space for the maximum
possible number of queues unconditionally?
> > virtio-scsi seems to be broken, at least my usual fedora guest
didn't
> > boot up from virtio-scsi disk when using a guest kernel with this
patch
> > series applied.
> 
> I'll re-test. Do other devices work for you? Thanks!
Didn't came very far yet in my testing due to the guest not booting.  I
plan to try other storage for the image, but didn't found the time yet.

I've tried to boot a F21 live iso with virtio-net (legacy guest driver
obviously), which seems to work ok in light testing.

BTW: is there a tool (or pciutils patch) which can decode the virtio
capabilities?

cheers,
  Gerd

On Fri, Jan 16, 2015 at 09:32:26AM +0100, Gerd Hoffmann
wrote:
>   Hi,
> 
> > >  * I'd like to see some more flexibility in the pci bar
layout.  Stuff
> > >    I have in mind:
> > >     - New devices which don't need a legacy bar can use bar 0
for
> > >       modern.
> > >     - One MMIO bar is enough, we can place both virtio regions
and
> > >       msi-x regions there.  I'd suggest to add msi-x
sub-regions to
> > >       the modern bar.
> > 
> > Why exactly? It seems simpler to separate things, extra BARs
> > have no cost.
> 
> Well, there are only six BARs.  legacy bar, modern mmio bar, msi-x bar,
> modern io bar (for fast isr).  That already four out of six ...
So?  If we need to add another bar, we'll change things.
> We have a mmio bar, which we partition into subregions for virtio-1.0
> anyway.  Also placing msi-x there is a single msix_init() call.  xhci is
> doing that too:
> 
>     00000000febf0000-00000000febf3fff (prio 1, RW): xhci
>       00000000febf0000-00000000febf003f (prio 0, RW): capabilities
>       00000000febf0040-00000000febf043f (prio 0, RW): operational
>       00000000febf0440-00000000febf044f (prio 0, RW): usb3 port #1
>       00000000febf0450-00000000febf045f (prio 0, RW): usb3 port #2
>       00000000febf0460-00000000febf046f (prio 0, RW): usb3 port #3
>       00000000febf0470-00000000febf047f (prio 0, RW): usb3 port #4
>       00000000febf0480-00000000febf048f (prio 0, RW): usb2 port #1
>       00000000febf0490-00000000febf049f (prio 0, RW): usb2 port #2
>       00000000febf04a0-00000000febf04af (prio 0, RW): usb2 port #3
>       00000000febf04b0-00000000febf04bf (prio 0, RW): usb2 port #4
>       00000000febf1000-00000000febf121f (prio 0, RW): runtime
>       00000000febf2000-00000000febf281f (prio 0, RW): doorbell
>       00000000febf3000-00000000febf30ff (prio 0, RW): msix-table
>       00000000febf3800-00000000febf3807 (prio 0, RW): msix-pba
> 
However that one is non prefetheable, and I prefer putting
virtio caps in a prefetcheable bar.
> > >  * What is the reason for making the modern bar 8M in size? 
Looks a bit
> > >    excessive, given that only 64k or so of that are actually used
...
> > 
> > I use a page per VQ for architectures that can locate the offset of
the
> > accessed page that triggered EPT violation faster than the offset
within
> > page.  I think this is the case for SVM.
> 
> 8M still looks excessive, given that we typically have a small number of
> queues per device.  Do you allocate address space for the maximum
> possible number of queues unconditionally?
Yes, simpler this way. I'll check if I can find the real # of VQs.
> > > virtio-scsi seems to be broken, at least my usual fedora guest
didn't
> > > boot up from virtio-scsi disk when using a guest kernel with this
patch
> > > series applied.
> > 
> > I'll re-test. Do other devices work for you? Thanks!
> 
> Didn't came very far yet in my testing due to the guest not booting.
Try virtio-blk - that worked for me.
>  I
> plan to try other storage for the image, but didn't found the time yet.
> 
> I've tried to boot a F21 live iso with virtio-net (legacy guest driver
> obviously), which seems to work ok in light testing.
> 
> BTW: is there a tool (or pciutils patch) which can decode the virtio
> capabilities?
> 
> cheers,
>   Gerd
>

Hi,
> BTW: is there a tool (or pciutils patch) which can decode the virtio
> capabilities?
Searched for a patch today, and all google found me was this mail asking
for one :-o

So I went ahead and coded one up.  Attached.

While hacking it up I've noticed spec doesn't match reality.  The
"Virtio Structure PCI Capabilities" section here ...

http://docs.oasis-open.org/virtio/virtio/v1.0/cs01/virtio-v1.0-cs01.html#x1-690004

... doesn't match what qemu is doing.  Huh?


Also note:

[root at fedora ~]# ~kraxel/projects/pciutils/lspci -vvsa
00:0a.0 Communication controller: Red Hat, Inc Virtio console
[ ... ]
	Capabilities: [64] VirtIO: Notify
		BAR=2 offset=00003000 size=00400000
		multiplier=00010000
[ ... ]

Why multiplier is 64k instead of 4k?  Just a tyops?


cheers,
  Gerd

-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-add-virtio-vendor-caps.patch
Type: text/x-patch
Size: 3354 bytes
Desc: not available
URL:
<http://lists.linuxfoundation.org/pipermail/virtualization/attachments/20150119/e8571cc0/attachment.bin>