Add optional MSI-X support: use a vector per virtqueue with fallback to a common vector and finally to regular interrupt. Teach all drivers to use it. I added 2 new virtio operations: request_vqs/free_vqs because MSI needs to know the total number of vectors upfront. Signed-off-by: Michael S. Tsirkin <mst at redhat.com> --- Here's a draft set of patches for MSI-X support in the guest. It still needs to be tested properly, and performance impact measured, but I thought I'd share it here in the hope of getting some very early feedback/flames. Michael S. Tsirkin (8): virtio: add request_vqs/free_vqs virtio operations virtio_blk: add request_vqs/free_vqs calls virtio-rng: add request_vqs/free_vqs calls virtio_console: add request_vqs/free_vqs calls virtio_net: add request_vqs/free_vqs calls virtio_balloon: add request_vqs/free_vqs calls virtio_pci: split up vp_interrupt virtio_pci: optional MSI-X support drivers/block/virtio_blk.c | 9 ++- drivers/char/hw_random/virtio-rng.c | 10 ++- drivers/char/virtio_console.c | 8 ++- drivers/net/virtio_net.c | 16 +++- drivers/virtio/virtio_balloon.c | 9 ++- drivers/virtio/virtio_pci.c | 192 ++++++++++++++++++++++++++++++----- include/linux/virtio_config.h | 35 +++++++ 7 files changed, 247 insertions(+), 32 deletions(-)
Christian Borntraeger
2009-Apr-27 14:00 UTC
[PATCH RFC 0/8] virtio: add guest MSI-X support
Am Monday 27 April 2009 14:31:36 schrieb Michael S. Tsirkin:> Add optional MSI-X support: use a vector per virtqueue with > fallback to a common vector and finally to regular interrupt. > Teach all drivers to use it. > > I added 2 new virtio operations: request_vqs/free_vqs because MSI > needs to know the total number of vectors upfront. > > Signed-off-by: Michael S. Tsirkin <mst at redhat.com>I dont know, if that is feasible for MSI, but the transport(virtio_pci) should already know the number of virtqueues, which should match the number of vectors, no? In fact, the transport has to have a way of getting the number of virtqeues because find_vq returns ENOENT on invalid index numbers. Christian
On Mon, Apr 27, 2009 at 04:00:30PM +0200, Christian Borntraeger wrote:> Am Monday 27 April 2009 14:31:36 schrieb Michael S. Tsirkin: > > Add optional MSI-X support: use a vector per virtqueue with > > fallback to a common vector and finally to regular interrupt. > > Teach all drivers to use it. > > > > I added 2 new virtio operations: request_vqs/free_vqs because MSI > > needs to know the total number of vectors upfront. > > > > Signed-off-by: Michael S. Tsirkin <mst at redhat.com> > > I dont know, if that is feasible for MSI, but the transport(virtio_pci) should > already know the number of virtqueues, which should match the number of > vectors, no?I think no, the transport can find out the max number of vectors the device supports (pci_msix_table_size), but not how many virtqueues are needed by the driver. As number of virtqueues <= number of vectors, we could pre-allocate all vectors that host supports, but this seems a bit drastic as an MSI-X device could support up to 2K vectors.> In fact, the transport has to have a way of getting the number of virtqeues > because find_vq returns ENOENT on invalid index numbers. > > ChristianSo again, I think this is an upper bound supported by host. Right? -- MST
Michael S. Tsirkin wrote:> Add optional MSI-X support: use a vector per virtqueue with > fallback to a common vector and finally to regular interrupt. > Teach all drivers to use it. > > I added 2 new virtio operations: request_vqs/free_vqs because MSI > needs to know the total number of vectors upfront. > > Signed-off-by: Michael S. Tsirkin <mst at redhat.com> >Do you have userspace patches for testing? Regards, Anthony Liguori
Christian Borntraeger wrote:> Am Monday 27 April 2009 14:31:36 schrieb Michael S. Tsirkin: > >> Add optional MSI-X support: use a vector per virtqueue with >> fallback to a common vector and finally to regular interrupt. >> Teach all drivers to use it. >> >> I added 2 new virtio operations: request_vqs/free_vqs because MSI >> needs to know the total number of vectors upfront. >> >> Signed-off-by: Michael S. Tsirkin <mst at redhat.com> >> > > I dont know, if that is feasible for MSI, but the transport(virtio_pci) should > already know the number of virtqueues, which should match the number of > vectors, no? > > In fact, the transport has to have a way of getting the number of virtqeues > because find_vq returns ENOENT on invalid index numbers. >One thing I thought was to decouple the virtqueue count from the MSI entry count, and let the guest choose how many MSI entries it wants to use. This is useful if it wants several queues to share an interrupt, perhaps it wants both rx and tx rings on one cpu to share a single vector. So the device could expose a large, constant number of MSI entries, and in addition expose a table mapping ring number to msi entry number. The guest could point all rings to one entry, or have each ring use a private entry, or anything in between. That saves us the new API (at the expense of a lot more code, but with added flexibility). -- error compiling committee.c: too many arguments to function
Christian Borntraeger
2009-Apr-27 15:37 UTC
[PATCH RFC 0/8] virtio: add guest MSI-X support
Am Monday 27 April 2009 16:32:56 schrieb Michael S. Tsirkin:> > I dont know, if that is feasible for MSI, but the transport(virtio_pci) should > > already know the number of virtqueues, which should match the number of > > vectors, no? > > I think no, the transport can find out the max number of vectors the > device supports (pci_msix_table_size), but not how many virtqueues are > needed by the driver.I was not talking about the number of vectors, but the number of real exisiting virtqueues. I just read virtio_pci again and yes,it would be a bit hard to get the number of available virtqueues, because we need to loop over all possible vqs: (something like) for (i=0; i < SOMEMAX; i++) { iowrite16(i, vp_dev->ioaddr + VIRTIO_PCI_QUEUE_SEL); if (!ioread16(vp_dev->ioaddr + VIRTIO_PCI_QUEUE_NUM)) break; } return i; which is obviously not very nice. On the other hand, at the moment we have up to 3 virtqueues per device (network), and we have a current maximum of 16 virtqeues in qemu (VIRTIO_PCI_QUEUE_MAX in virtio.c), which would reduce the number.> As number of virtqueues <= number of vectors, > we could pre-allocate all vectors that host supports, but this seems > a bit drastic as an MSI-X device could support up to 2K vectors. > > > In fact, the transport has to have a way of getting the number of virtqeues > > because find_vq returns ENOENT on invalid index numbers. > > > > Christian > > So again, I think this is an upper bound supported by host. Right?Not the upper bound, but the real available virtqueues. (With current qemu 3 for virtio-net, 2 for virtio-console etc.) Since I use a different transport (drivers/s390/kvm/kvm_virtio.c), my motiviation is to keep the virtio interface as generic as possible. I dont really like the new interface, but I cannot give you silver bullet technical reasons - its more a gut feeling. The interface would work with lguest and s390. Anyway. Avis suggestion to decouple MSI count and virtqueue count looks like a promising approach. Christian
On Mon, Apr 27, 2009 at 09:39:06AM -0500, Anthony Liguori wrote:> Michael S. Tsirkin wrote: >> Add optional MSI-X support: use a vector per virtqueue with >> fallback to a common vector and finally to regular interrupt. >> Teach all drivers to use it. >> >> I added 2 new virtio operations: request_vqs/free_vqs because MSI >> needs to know the total number of vectors upfront. >> >> Signed-off-by: Michael S. Tsirkin <mst at redhat.com> >> > > Do you have userspace patches for testing? > > Regards, > > Anthony LiguoriNot ready yet, sorry. -- MST
Possibly Parallel Threads
- [PATCH RFC 0/8] virtio: add guest MSI-X support
- [PATCH 1/3] virtio: find_vqs/del_vqs virtio operations
- [PATCH 1/3] virtio: find_vqs/del_vqs virtio operations
- [PATCHv5 1/3] virtio: find_vqs/del_vqs virtio operations
- [PATCHv5 1/3] virtio: find_vqs/del_vqs virtio operations