Linux Virtualization - Jan 2017 - [PATCH v2 2/2] vring: Force use of DMA API for ARM-based systems

On Mon, Jan 16, 2017 at 10:40:28AM +0000, Will Deacon
wrote:
> On Fri, Jan 13, 2017 at 08:23:35PM +0200, Michael S. Tsirkin wrote:
> > On Fri, Jan 13, 2017 at 05:21:54PM +0000, Will Deacon wrote:
> > > On Fri, Jan 13, 2017 at 06:46:32PM +0200, Michael S. Tsirkin
wrote:
> > > > On Fri, Jan 13, 2017 at 09:25:22AM +0000, Will Deacon wrote:
> > > > > On Fri, Jan 13, 2017 at 12:12:56AM +0200, Michael S.
Tsirkin wrote:
> > > > > > I'd rather people didn't use SMMU with
legacy devices.
> > > > > 
> > > > > I'm afraid we've been doing that for two years
and the model already
> > > > > exists in a mature state, being actively used for
development and
> > > > > validation by ARM and our partners. One of the big
things its used for
> > > > > is to develop SMMU and GIC (our interrupt controller)
code with PCI, so
> > > > > dropping the SMMU from the picture isn't an option.
> > > > 
> > > > Oh so this fixes a regression?  This is something I
didn't realize.
> > > 
> > > Yes, thanks. The regression came about because we implemented
SMMU-backed
> > > DMA ops and only then was it apparent that the virtio stuff was
bypassing
> > > even with translation enabled (because it wasn't using the
DMA API).
> > 
> > Could you point out a commit ID?
> 
> There has been a fair amount of work in this area recently, but you're
> probably after something like 876945dbf649 ("arm64: Hook up IOMMU
dma_ops")
> as the culprit, which is the point at which we started to swizzle DMA
> ops for devices upstream of an SMMU automatically.
> 
> > > > A "Fixes:" tag can't hurt here.  I then wonder
> > > > might DMA ops ever use a DMA address which isn't a
physical address
> > > > from QEMU point of view? If that happens, this hack breaks
> > > > because in legacy mode QEMU still uses the GPA.
> > > 
> > > If QEMU doesn't advertise an SMMU, then it will work fine
with the GPA,
> > > because we won't swizzle the DMA ops for the master device.
If QEMU does
> > > advertise an SMMU, then we'll allocate DMA addresses to fit
within the
> > > the intersection of the SMMU aperture and device's DMA mask.
> > 
> > 
> > Right but doesn't just poking from qemu into phys addresses work
> > anymore? It used to ...
> 
> Provided that there's no SMMU, then it will continue to work. and my
> understanding (from talking to Peter Maydell) is that qemu doesn't
model
> an SMMU for ARM-based machines.
> 
> Will
So how come people report failures due to presence of SMMU?
Using some other hypervisor?

-- 
MST

On Mon, Jan 16, 2017 at 04:18:03PM +0200, Michael S. Tsirkin
wrote:
> On Mon, Jan 16, 2017 at 10:40:28AM +0000, Will Deacon wrote:
> > On Fri, Jan 13, 2017 at 08:23:35PM +0200, Michael S. Tsirkin wrote:
> > > On Fri, Jan 13, 2017 at 05:21:54PM +0000, Will Deacon wrote:
> > > > On Fri, Jan 13, 2017 at 06:46:32PM +0200, Michael S. Tsirkin
wrote:
> > > > > On Fri, Jan 13, 2017 at 09:25:22AM +0000, Will Deacon
wrote:
> > > > > > On Fri, Jan 13, 2017 at 12:12:56AM +0200, Michael
S. Tsirkin wrote:
> > > > > > > I'd rather people didn't use SMMU
with legacy devices.
> > > > > > 
> > > > > > I'm afraid we've been doing that for two
years and the model already
> > > > > > exists in a mature state, being actively used for
development and
> > > > > > validation by ARM and our partners. One of the big
things its used for
> > > > > > is to develop SMMU and GIC (our interrupt
controller) code with PCI, so
> > > > > > dropping the SMMU from the picture isn't an
option.
> > > > > 
> > > > > Oh so this fixes a regression?  This is something I
didn't realize.
> > > > 
> > > > Yes, thanks. The regression came about because we
implemented SMMU-backed
> > > > DMA ops and only then was it apparent that the virtio stuff
was bypassing
> > > > even with translation enabled (because it wasn't using
the DMA API).
> > > 
> > > Could you point out a commit ID?
> > 
> > There has been a fair amount of work in this area recently, but
you're
> > probably after something like 876945dbf649 ("arm64: Hook up IOMMU
dma_ops")
> > as the culprit, which is the point at which we started to swizzle DMA
> > ops for devices upstream of an SMMU automatically.
> > 
> > > > > A "Fixes:" tag can't hurt here.  I then
wonder
> > > > > might DMA ops ever use a DMA address which isn't a
physical address
> > > > > from QEMU point of view? If that happens, this hack
breaks
> > > > > because in legacy mode QEMU still uses the GPA.
> > > > 
> > > > If QEMU doesn't advertise an SMMU, then it will work
fine with the GPA,
> > > > because we won't swizzle the DMA ops for the master
device. If QEMU does
> > > > advertise an SMMU, then we'll allocate DMA addresses to
fit within the
> > > > the intersection of the SMMU aperture and device's DMA
mask.
> > > 
> > > 
> > > Right but doesn't just poking from qemu into phys addresses
work
> > > anymore? It used to ...
> > 
> > Provided that there's no SMMU, then it will continue to work. and
my
> > understanding (from talking to Peter Maydell) is that qemu doesn't
model
> > an SMMU for ARM-based machines.
> > 
> 
> So how come people report failures due to presence of SMMU?
> Using some other hypervisor?
The failures are reported on the ARM fastmodel (a complete system
emulation that runs on an x86 box), where an SMMU *is* present
downstream of the virtio-pci masters. There's no qemu involved there.

Will

On Mon, Jan 16, 2017 at 02:21:03PM +0000, Will Deacon
wrote:
> On Mon, Jan 16, 2017 at 04:18:03PM +0200, Michael S. Tsirkin wrote:
> > On Mon, Jan 16, 2017 at 10:40:28AM +0000, Will Deacon wrote:
> > > On Fri, Jan 13, 2017 at 08:23:35PM +0200, Michael S. Tsirkin
wrote:
> > > > On Fri, Jan 13, 2017 at 05:21:54PM +0000, Will Deacon wrote:
> > > > > On Fri, Jan 13, 2017 at 06:46:32PM +0200, Michael S.
Tsirkin wrote:
> > > > > > On Fri, Jan 13, 2017 at 09:25:22AM +0000, Will
Deacon wrote:
> > > > > > > On Fri, Jan 13, 2017 at 12:12:56AM +0200,
Michael S. Tsirkin wrote:
> > > > > > > > I'd rather people didn't use
SMMU with legacy devices.
> > > > > > > 
> > > > > > > I'm afraid we've been doing that for
two years and the model already
> > > > > > > exists in a mature state, being actively used
for development and
> > > > > > > validation by ARM and our partners. One of
the big things its used for
> > > > > > > is to develop SMMU and GIC (our interrupt
controller) code with PCI, so
> > > > > > > dropping the SMMU from the picture isn't
an option.
> > > > > > 
> > > > > > Oh so this fixes a regression?  This is something
I didn't realize.
> > > > > 
> > > > > Yes, thanks. The regression came about because we
implemented SMMU-backed
> > > > > DMA ops and only then was it apparent that the virtio
stuff was bypassing
> > > > > even with translation enabled (because it wasn't
using the DMA API).
> > > > 
> > > > Could you point out a commit ID?
> > > 
> > > There has been a fair amount of work in this area recently, but
you're
> > > probably after something like 876945dbf649 ("arm64: Hook up
IOMMU dma_ops")
> > > as the culprit, which is the point at which we started to swizzle
DMA
> > > ops for devices upstream of an SMMU automatically.
> > > 
> > > > > > A "Fixes:" tag can't hurt here.  I
then wonder
> > > > > > might DMA ops ever use a DMA address which
isn't a physical address
> > > > > > from QEMU point of view? If that happens, this
hack breaks
> > > > > > because in legacy mode QEMU still uses the GPA.
> > > > > 
> > > > > If QEMU doesn't advertise an SMMU, then it will
work fine with the GPA,
> > > > > because we won't swizzle the DMA ops for the master
device. If QEMU does
> > > > > advertise an SMMU, then we'll allocate DMA
addresses to fit within the
> > > > > the intersection of the SMMU aperture and device's
DMA mask.
> > > > 
> > > > 
> > > > Right but doesn't just poking from qemu into phys
addresses work
> > > > anymore? It used to ...
> > > 
> > > Provided that there's no SMMU, then it will continue to work.
and my
> > > understanding (from talking to Peter Maydell) is that qemu
doesn't model
> > > an SMMU for ARM-based machines.
> > > 
> > 
> > So how come people report failures due to presence of SMMU?
> > Using some other hypervisor?
> 
> The failures are reported on the ARM fastmodel (a complete system
> emulation that runs on an x86 box), where an SMMU *is* present
> downstream of the virtio-pci masters. There's no qemu involved there.
> 
> Will
I see. And this hypervisor actually coded up looking up
translations in the SMMU unconditionally for legacy devices,
and this worked as long as guest didn't touch the SMMU?

-- 
MST