Linux Virtualization - Sep 2014 - [PATCH v4 0/4] virtio: Clean up scatterlists and use the DMA API

On Wed, Sep 03, 2014 at 04:12:01PM +0930, Rusty Russell
wrote:
> Andy Lutomirski <luto at amacapital.net> writes:
> > There really are virtio devices that are pieces of silicon and not
> > figments of a hypervisor's imagination [1].
> 
> Hi Andy,
> 
>         As you're discovering, there's a reason no one has done the
DMA
> API before.
> 
> So the problem is that ppc64's IOMMU is a platform thing, not a bus
> thing.  They really do carve out an exception for virtio devices,
> because performance (LOTS of performance).  It remains to be seen if
> other platforms have the same performance issues, but in absence of
> other evidence, the answer is yes.
> 
> It's a hack.  But having specific virtual-only devices are an even
> bigger hack.
> 
> Physical virtio devices have been talked about, but don't actually
exist
> in Real Life.  And someone a virtio PCI card is going to have serious
> performance issues: mainly because they'll want the rings in the
card's
> MMIO region, not allocated by the driver.
Why? What's wrong with rings in memory?
>  Being broken on PPC is really
> the least of their problems.
> 
> So, what do we do?  It'd be nice if Linux virtio Just Worked under Xen,
> though Xen's IOMMU is outside the virtio spec.  Since virtio_pci can be
> a module, obvious hacks like having xen_arch_setup initialize a dma_ops
pointer
> exposed by virtio_pci.c is out.
Well virtio could probe for xen, it's not a lot of code.
> I think the best approach is to have a new feature bit (25 is free),
> VIRTIO_F_USE_BUS_MAPPING which indicates that a device really wants to
> use the mapping for the bus it is on.  A real device would set this,
> or it won't work behind an IOMMU.  A Xen device would also set this.
> 
> Thoughts?
> Rusty.
OK and it should then be active even if guest does not ack
the feature (so in fact, it would have to be a mandatory feature).
That can work, but I still find this a bit inelegant: this is
a property of the platform, not of the device.

> PS.  I cc'd OASIS virtio-dev: it's subscriber only for IP reasons
(to
>      subscribe you have to promise we can use your suggestion in the
>      standard).  Feel free to remove in any replies, but it's part of
>      the world we live in...

"Michael S. Tsirkin" <mst at redhat.com>
writes:
> On Wed, Sep 03, 2014 at 04:12:01PM +0930, Rusty Russell wrote:
>> Andy Lutomirski <luto at amacapital.net> writes:
>> > There really are virtio devices that are pieces of silicon and not
>> > figments of a hypervisor's imagination [1].
>> 
>> Hi Andy,
>> 
>>         As you're discovering, there's a reason no one has done
the DMA
>> API before.
>> 
>> So the problem is that ppc64's IOMMU is a platform thing, not a bus
>> thing.  They really do carve out an exception for virtio devices,
>> because performance (LOTS of performance).  It remains to be seen if
>> other platforms have the same performance issues, but in absence of
>> other evidence, the answer is yes.
>> 
>> It's a hack.  But having specific virtual-only devices are an even
>> bigger hack.
>> 
>> Physical virtio devices have been talked about, but don't actually
exist
>> in Real Life.  And someone a virtio PCI card is going to have serious
>> performance issues: mainly because they'll want the rings in the
card's
>> MMIO region, not allocated by the driver.
>
> Why? What's wrong with rings in memory?
AFAICT, the card would have to access guest memory to read it, using
multiple DMA cycles.  That's going to be slow.
>>  Being broken on PPC is really
>> the least of their problems.
>> 
>> So, what do we do?  It'd be nice if Linux virtio Just Worked under
Xen,
>> though Xen's IOMMU is outside the virtio spec.  Since virtio_pci
can be
>> a module, obvious hacks like having xen_arch_setup initialize a dma_ops
pointer
>> exposed by virtio_pci.c is out.
>
> Well virtio could probe for xen, it's not a lot of code.
We could, but I think this is going to be a more general problem in
future.  x86 is heading down the IOMMU path, and they're likely to
suffer similarly.
>> I think the best approach is to have a new feature bit (25 is free),
>> VIRTIO_F_USE_BUS_MAPPING which indicates that a device really wants to
>> use the mapping for the bus it is on.  A real device would set this,
>> or it won't work behind an IOMMU.  A Xen device would also set
this.
>> 
>> Thoughts?
>> Rusty.
>
> OK and it should then be active even if guest does not ack
> the feature (so in fact, it would have to be a mandatory feature).
> That can work, but I still find this a bit inelegant: this is
> a property of the platform, not of the device.
True.  If a device needs it though, we're no worse of having a device
which doesn't work if the driver understand the feature than we were
before.

Cheers,
Rusty.

On Thu, Sep 4, 2014 at 7:32 PM, Rusty Russell <rusty at rustcorp.com.au>
wrote:
> "Michael S. Tsirkin" <mst at redhat.com> writes:
>> On Wed, Sep 03, 2014 at 04:12:01PM +0930, Rusty Russell wrote:
>>> Andy Lutomirski <luto at amacapital.net> writes:
>>> > There really are virtio devices that are pieces of silicon and
not
>>> > figments of a hypervisor's imagination [1].
>>>
>>> Hi Andy,
>>>
>>>         As you're discovering, there's a reason no one has
done the DMA
>>> API before.
>>>
>>> So the problem is that ppc64's IOMMU is a platform thing, not a
bus
>>> thing.  They really do carve out an exception for virtio devices,
>>> because performance (LOTS of performance).  It remains to be seen
if
>>> other platforms have the same performance issues, but in absence of
>>> other evidence, the answer is yes.
>>>
>>> It's a hack.  But having specific virtual-only devices are an
even
>>> bigger hack.
>>>
>>> Physical virtio devices have been talked about, but don't
actually exist
>>> in Real Life.  And someone a virtio PCI card is going to have
serious
>>> performance issues: mainly because they'll want the rings in
the card's
>>> MMIO region, not allocated by the driver.
>>
>> Why? What's wrong with rings in memory?
>
> AFAICT, the card would have to access guest memory to read it, using
> multiple DMA cycles.  That's going to be slow.
I don't personally know all the considerations, but AFAICT NVMe puts
its rings in memory, and NMVe is very much focused on performance.

There might be an argument for trying to avoid using indirect rings on
real hardware to reduce the number of DMA round-trips needed for a
comment.

--Andy