Linux Virtualization - Sep 2014 - [PATCH v5 2/3] virtio_pci: Use the DMA API for virtqueues when possible

On Wed, Sep 17, 2014 at 7:16 AM, Michael S. Tsirkin <mst at redhat.com>
wrote:
> On Wed, Sep 17, 2014 at 08:02:31AM -0400, Benjamin Herrenschmidt wrote:
>> On Tue, 2014-09-16 at 22:22 -0700, Andy Lutomirski wrote:
>> > On non-PPC systems, virtio_pci should use the DMA API.  This fixes
>> > virtio_pci on Xen.  On PPC, using the DMA API would break things,
so
>> > we need to preserve the old behavior.
>> >
>> > The big comment in this patch explains the considerations in more
>> > detail.
>>
>> I still disagree with using CONFIG_PPC as a trigger here.
>>
>> Fundamentally, the qemu implementation today bypasses IOMMUs on all
>> platforms as far as I can tell.
>>
>> If that changes, we'll have a backward compatibility problem.
>>
>> The virtio device should advertise whether it's using that bypass
>> mode of operation and virtio_pci should react accordingly.
>
> Well if there's a way to detect that - that's outside the
> device, then we probably shouldn't use device-specific
> interfaces for this capability.
>
>
>> There is a demand for being able to operate on top of an IOMMU on
>> powerpc as well for some embedded stuff using PCI as a transport so
your
>> patch precludes that.
>>
>> Cheers,
>> Ben.
>
> As far as I can see, nothing changes on PPC so this patch
> does not preclude anything that was working?
Rusty and Michael, what's the status of this?

I think that (aside from the trivial DMI/DMA typo) the only real issue
here is that the situation on PPC is ugly.  We're failing to enable
physical virtio hardware on PPC with these patches, but that never
worked anyway.  I don't think that there are any regressions other
than ugliness.

My preference would be to apply the patches as is (or with "DMA"
spelled correctly), and then to:

 - Make sure that all virtio-mmio systems have working DMA ops so that
virtio-mmio can the DMA API

 - Fix the DMA API on s390 (probably easy) and on PPC (not necessarily so easy)

 - Remove the non-DMA-API code, which would be a very small change on
top of these patches.

--Andy

On Mon, 2014-09-29 at 11:55 -0700, Andy Lutomirski wrote:
> Rusty and Michael, what's the status of this?
The status is that I still think we need *a* way to actually inform the
guest whether the virtio implementation will or will not bypass the
IOMMU. I don't know Xen enough to figure out how to do that and we could
maybe just make it something qemu puts in the device-tree on powerpc
only.

However I dislike making it global or per-bus, we could have a
combination of qemu and HW virtio on the same guest, so I really think
this needs to be a capability of the virtio device.

I don't completely understand what games Xen is playing here, but from
what I can tell, it's pretty clear that today's qemu implementation
always bypasses any iommu and so should always be exported as such on
all platforms, at least all kvm and pure qemu ones.
> I think that (aside from the trivial DMI/DMA typo) the only real issue
> here is that the situation on PPC is ugly.  We're failing to enable
> physical virtio hardware on PPC with these patches, but that never
> worked anyway.  I don't think that there are any regressions other
> than ugliness.
> 
> My preference would be to apply the patches as is (or with "DMA"
> spelled correctly), and then to:
> 
>  - Make sure that all virtio-mmio systems have working DMA ops so that
> virtio-mmio can the DMA API
> 
>  - Fix the DMA API on s390 (probably easy) and on PPC (not necessarily so
easy)
> 
>  - Remove the non-DMA-API code, which would be a very small change on
> top of these patches.
> 
> --Andy

On Mon, Sep 29, 2014 at 1:49 PM, Benjamin Herrenschmidt
<benh at kernel.crashing.org> wrote:
> On Mon, 2014-09-29 at 11:55 -0700, Andy Lutomirski wrote:
>
>> Rusty and Michael, what's the status of this?
>
> The status is that I still think we need *a* way to actually inform the
> guest whether the virtio implementation will or will not bypass the
> IOMMU. I don't know Xen enough to figure out how to do that and we
could
> maybe just make it something qemu puts in the device-tree on powerpc
> only.
>
> However I dislike making it global or per-bus, we could have a
> combination of qemu and HW virtio on the same guest, so I really think
> this needs to be a capability of the virtio device.
Or a capability of the PCI slot, somehow.  I don't understand PCI
topology very well.
>
> I don't completely understand what games Xen is playing here, but from
> what I can tell, it's pretty clear that today's qemu implementation
> always bypasses any iommu and so should always be exported as such on
> all platforms, at least all kvm and pure qemu ones.
Except that I think that PPC is the only platform on which QEMU's code
actually bypasses any IOMMU.  Unless we've all missed something, there
is no QEMU release that will put a virtio device behind an IOMMU on
any platform other than PPC.

If the eventual solution is to say that virtio 1.0 PCI devices always
respect an IOMMU unless they set a magic flag saying "I'm not real
hardware and I bypass the IOMMU", then I don't really object to that,
except that it'll be a mess if the guest is running Xen.  But even Xen
would (I think) be okay if it actually worked by having a new DMA API
operation that says "this device is magically identity mapped" and
then just teaching Xen to implement that.

But I'm not an OASIS member, so I can't really do this.  I agree that
this issue needs to be addressed somehow, but I don't think it needs
to block these patches.

--Andy
>
>> I think that (aside from the trivial DMI/DMA typo) the only real issue
>> here is that the situation on PPC is ugly.  We're failing to enable
>> physical virtio hardware on PPC with these patches, but that never
>> worked anyway.  I don't think that there are any regressions other
>> than ugliness.
>>
>> My preference would be to apply the patches as is (or with
"DMA"
>> spelled correctly), and then to:
>>
>>  - Make sure that all virtio-mmio systems have working DMA ops so that
>> virtio-mmio can the DMA API
>>
>>  - Fix the DMA API on s390 (probably easy) and on PPC (not necessarily
so easy)
>>
>>  - Remove the non-DMA-API code, which would be a very small change on
>> top of these patches.
>>
>> --Andy
>
>


-- 
Andy Lutomirski
AMA Capital Management, LLC

Am 29.09.2014 20:55, schrieb Andy Lutomirski:
> On Wed, Sep 17, 2014 at 7:16 AM, Michael S. Tsirkin <mst at
redhat.com> wrote:
>> On Wed, Sep 17, 2014 at 08:02:31AM -0400, Benjamin Herrenschmidt wrote:
>>> On Tue, 2014-09-16 at 22:22 -0700, Andy Lutomirski wrote:
>>>> On non-PPC systems, virtio_pci should use the DMA API.  This
fixes
>>>> virtio_pci on Xen.  On PPC, using the DMA API would break
things, so
>>>> we need to preserve the old behavior.
>>>>
>>>> The big comment in this patch explains the considerations in
more
>>>> detail.
>>>
>>> I still disagree with using CONFIG_PPC as a trigger here.
>>>
>>> Fundamentally, the qemu implementation today bypasses IOMMUs on all
>>> platforms as far as I can tell.
>>>
>>> If that changes, we'll have a backward compatibility problem.
>>>
>>> The virtio device should advertise whether it's using that
bypass
>>> mode of operation and virtio_pci should react accordingly.
>>
>> Well if there's a way to detect that - that's outside the
>> device, then we probably shouldn't use device-specific
>> interfaces for this capability.
>>
>>
>>> There is a demand for being able to operate on top of an IOMMU on
>>> powerpc as well for some embedded stuff using PCI as a transport so
your
>>> patch precludes that.
>>>
>>> Cheers,
>>> Ben.
>>
>> As far as I can see, nothing changes on PPC so this patch
>> does not preclude anything that was working?
> 
> Rusty and Michael, what's the status of this?
> 
> I think that (aside from the trivial DMI/DMA typo) the only real issue
> here is that the situation on PPC is ugly.  We're failing to enable
> physical virtio hardware on PPC with these patches, but that never
> worked anyway.  I don't think that there are any regressions other
> than ugliness.
> 
> My preference would be to apply the patches as is (or with "DMA"
> spelled correctly), and then to:
> 
>  - Make sure that all virtio-mmio systems have working DMA ops so that
> virtio-mmio can the DMA API
> 
>  - Fix the DMA API on s390 (probably easy) and on PPC (not necessarily so
easy)
Just as a comment: On s390 we always considered the memory access as access to
real memory (not device memory) for virtio accesses. I prefer to not touch the
DMA API on s390 as it is quite s390-PCI specific but it is somewhat strange for
CCW devices.

We would basically have two kinds of DMA mappings on s390 then, one iommu like
for the PCI devices and one that maps DMA memory 1:1 to real memory. We would
also need some buy-in from the s390 arch maintainers then.

> 
>  - Remove the non-DMA-API code, which would be a very small change on
> top of these patches.
> 
> --Andy
> --
> To unsubscribe from this list: send the line "unsubscribe
linux-s390" in
> the body of a message to majordomo at vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

On Mon, 2014-10-06 at 11:59 +0200, Christian Borntraeger
wrote:
> Just as a comment: On s390 we always considered the memory access as
> access to real memory (not device memory) for virtio accesses. I
> prefer to not touch the DMA API on s390 as it is quite s390-PCI
> specific but it is somewhat strange for CCW devices.
> 
> We would basically have two kinds of DMA mappings on s390 then, one
> iommu like for the PCI devices and one that maps DMA memory 1:1 to
> real memory. We would also need some buy-in from the s390 arch
> maintainers then.
It's pretty standard to have the dma API go via dma-ops hanging off
struct device, you can then easily make them to the right thing for the
type of device... Provided we have a way to identify "bypass
everyting"
virtio vs. virtio going through normal PCI translation.

Cheers,
Ben.