Linux Virtualization - Sep 2014 - [PATCH v5 2/3] virtio_pci: Use the DMA API for virtqueues when possible

On Tue, Sep 30, 2014 at 8:38 AM, Michael S. Tsirkin <mst at redhat.com>
wrote:
> I thought hard about this, I think we are better off waiting till the
> next release: there's a chance QEMU will have IOMMU support for KVM x86
> then, and this will make it easier to judge which way does the wind
> blow.
>
> It seems that we lose nothing substantial keeping the status quo a bit
longer,
> but if we make an incompatible change in guests now we might
> create nasty compatibility headaches going forward.
>
I would argue for the opposite approach.  Having a QEMU release that
supports an IOMMU on x86 and exposes a commonly used PCI device that
bypasses that IOMMU without any explicit notification to the guest
(and specification!) that this is happening is IMO insane.  Once that
happens, we'll have to address the nasty case on both x86 and PPC.
This will suck.

If we accept the guest change and make sure that there is never a QEMU
release that has a visible IOMMU cheat on any arch other than PPC,
then at least the damage will be contained.

x86 will be worse than PPC, too: the special case needed to support
QEMU 2.2 with IOMMU and virtio enabled with a Xen guest will be fairly
large and disgusting and will only exist to support something that IMO
should never have existed in the first place.

PPC at least avoids *that* problem by virtue of not having Xen
paravirt.  (And please don't add Xen paravirt to PPC -- x86 is trying
to kill it off, but this is a 5-10 year project.)

[..., reordered]
>>
>> Except that I think that PPC is the only platform on which QEMU's
code
>> actually bypasses any IOMMU.  Unless we've all missed something,
there
>> is no QEMU release that will put a virtio device behind an IOMMU on
>> any platform other than PPC.
>
> I think that is true but it seems that this will be true for x86 for
> QEMU 2.2 unless we make some changes there.
> Which we might not have the time for since 2.2 is feature frozen
> from tomorrow.
> Maybe we should disable the IOMMU in 2.2, this is worth considering.
>
Please do.

Also, try booting this 2.2 QEMU candidate with nested virtualization
on.  Then bind vfio to a virtio-pci device and watch the guest get
corrupted.  QEMU will blame Linux for incorrectly programming the
hardware, and Linux will blame QEMU for its blatant violation of the
ACPI spec.  Given that this is presumably most of the point of adding
IOMMU support, it seems like a terrible idea to let code like that
into the wild.

If this happens, Linux may also end up needing a quirk to prevent vfio
from binding to QEMU 2.2's virtio-pci devices.

--Andy

On Tue, Sep 30, 2014 at 8:48 AM, Andy Lutomirski <luto at amacapital.net>
wrote:
> On Tue, Sep 30, 2014 at 8:38 AM, Michael S. Tsirkin <mst at
redhat.com> wrote:
>> Maybe we should disable the IOMMU in 2.2, this is worth considering.
>>
>
> Please do.
>
> Also, try booting this 2.2 QEMU candidate with nested virtualization
> on.  Then bind vfio to a virtio-pci device and watch the guest get
> corrupted.  QEMU will blame Linux for incorrectly programming the
> hardware, and Linux will blame QEMU for its blatant violation of the
> ACPI spec.  Given that this is presumably most of the point of adding
> IOMMU support, it seems like a terrible idea to let code like that
> into the wild.
>
> If this happens, Linux may also end up needing a quirk to prevent vfio
> from binding to QEMU 2.2's virtio-pci devices.
I just confirmed that a guest with my patches blows up if I run it
like this against QEMU master from today:

PATH=.:$PATH virtme-run --kimg
~/apps/linux-devel/arch/x86/boot/bzImage -a intel_iommu=on --qemu-opts
-machine q35,iommu=on

IOW, QEMU master is indeed presenting an IOMMU that blows up the guest
if the guest tries to use it as specified.

--Andy

> x86 will be worse than PPC, too: the special case needed to support
> QEMU 2.2 with IOMMU and virtio enabled with a Xen guest will be fairly
> large and disgusting and will only exist to support something that IMO
> should never have existed in the first place.
<scratches his head> I don't follow.
> 
> PPC at least avoids *that* problem by virtue of not having Xen
> paravirt.  (And please don't add Xen paravirt to PPC -- x86 is trying
> to kill it off, but this is a 5-10 year project.)
Correction:
 - The Xen project is trying to kill some of the paravirts off.
 - KVM uses paravirts as well (and then added some)
> 
> [..., reordered]
> 
> >>
> >> Except that I think that PPC is the only platform on which
QEMU's code
> >> actually bypasses any IOMMU.  Unless we've all missed
something, there
> >> is no QEMU release that will put a virtio device behind an IOMMU
on
> >> any platform other than PPC.
> >
> > I think that is true but it seems that this will be true for x86 for
> > QEMU 2.2 unless we make some changes there.
> > Which we might not have the time for since 2.2 is feature frozen
> > from tomorrow.
> > Maybe we should disable the IOMMU in 2.2, this is worth considering.
> >
> 
> Please do.
> 
> Also, try booting this 2.2 QEMU candidate with nested virtualization
> on.  Then bind vfio to a virtio-pci device and watch the guest get
> corrupted.  QEMU will blame Linux for incorrectly programming the
Hehe.
> hardware, and Linux will blame QEMU for its blatant violation of the
> ACPI spec.  Given that this is presumably most of the point of adding
> IOMMU support, it seems like a terrible idea to let code like that
> into the wild.
> 
> If this happens, Linux may also end up needing a quirk to prevent vfio
> from binding to QEMU 2.2's virtio-pci devices.
> 
> --Andy

On Tue, Sep 30, 2014 at 10:53 AM, Konrad Rzeszutek Wilk
<konrad.wilk at oracle.com> wrote:
>> x86 will be worse than PPC, too: the special case needed to support
>> QEMU 2.2 with IOMMU and virtio enabled with a Xen guest will be fairly
>> large and disgusting and will only exist to support something that IMO
>> should never have existed in the first place.
>
> <scratches his head> I don't follow.
If you boot a Xen PV dom0 on QEMU master with -machine q35,iommu=on
and you add a virtio device, dom0 will end up with a PCI device that
does DMA to "machine" addresses.  These addresses are not compatible
with the DMA API (which works with bus addresses), nor are they the
same as physical addresses.

So virtio in current kernels won't work for the same reason they never
work on Xen.  But virtio-pci with my patches won't work either,
because they (or the Xen hypervisor) will try to program the IOMMU
with a non-identity mapping, causing everything to explode.

Hacking up the virtio-pci driver to explicitly ask Xen for machine
addresses might work, but, at the very least, it will be a giant
security hole if anyone binds a virtio device to a domain other than
dom0 (which, again, is kind of the point of having an IOMMU).
>>
>> PPC at least avoids *that* problem by virtue of not having Xen
>> paravirt.  (And please don't add Xen paravirt to PPC -- x86 is
trying
>> to kill it off, but this is a 5-10 year project.)
>
> Correction:
>  - The Xen project is trying to kill some of the paravirts off.
>  - KVM uses paravirts as well (and then added some)
By "paravirt" I meant PV, where there's the weird physical/machine
address discrepancy that's visible to the guest.  This is not to say
that Xen PVH wouldn't also be screwed running on QEMU master.

--Andy

On Tue, Sep 30, 2014 at 08:48:45AM -0700, Andy Lutomirski
wrote:
> On Tue, Sep 30, 2014 at 8:38 AM, Michael S. Tsirkin <mst at
redhat.com> wrote:
> > I thought hard about this, I think we are better off waiting till the
> > next release: there's a chance QEMU will have IOMMU support for
KVM x86
> > then, and this will make it easier to judge which way does the wind
> > blow.
> >
> > It seems that we lose nothing substantial keeping the status quo a bit
longer,
> > but if we make an incompatible change in guests now we might
> > create nasty compatibility headaches going forward.
> >
> 
> I would argue for the opposite approach.  Having a QEMU release that
> supports an IOMMU on x86 and exposes a commonly used PCI device that
> bypasses that IOMMU without any explicit notification to the guest
> (and specification!) that this is happening is IMO insane.  Once that
> happens, we'll have to address the nasty case on both x86 and PPC.
> This will suck.
>
> If we accept the guest change and make sure that there is never a QEMU
> release that has a visible IOMMU cheat on any arch other than PPC,
> then at least the damage will be contained.
Wrt QEMU this sounds reasonable. Wrt guest, deferring guest changes
a bit, until we have a better idea about how the host side behaves
sounds better to me than saying "this is how guests will behave,
let the host adapt to that".
> x86 will be worse than PPC, too: the special case needed to support
> QEMU 2.2 with IOMMU and virtio enabled with a Xen guest will be fairly
> large and disgusting and will only exist to support something that IMO
> should never have existed in the first place.
> 
> PPC at least avoids *that* problem by virtue of not having Xen
> paravirt.  (And please don't add Xen paravirt to PPC -- x86 is trying
> to kill it off, but this is a 5-10 year project.)
> 
> [..., reordered]
> 
> >>
> >> Except that I think that PPC is the only platform on which
QEMU's code
> >> actually bypasses any IOMMU.  Unless we've all missed
something, there
> >> is no QEMU release that will put a virtio device behind an IOMMU
on
> >> any platform other than PPC.
> >
> > I think that is true but it seems that this will be true for x86 for
> > QEMU 2.2 unless we make some changes there.
> > Which we might not have the time for since 2.2 is feature frozen
> > from tomorrow.
> > Maybe we should disable the IOMMU in 2.2, this is worth considering.
> >
> 
> Please do.
Or at least disable it just if there are virtio devices.
> Also, try booting this 2.2 QEMU candidate with nested virtualization
> on.  Then bind vfio to a virtio-pci device and watch the guest get
> corrupted.  QEMU will blame Linux for incorrectly programming the
> hardware, and Linux will blame QEMU for its blatant violation of the
> ACPI spec.  Given that this is presumably most of the point of adding
> IOMMU support, it seems like a terrible idea to let code like that
> into the wild.
> 
> If this happens, Linux may also end up needing a quirk to prevent vfio
> from binding to QEMU 2.2's virtio-pci devices.
> 
> --Andy
This specific item wouldn't worry me too much.

-- 
MST