Linux Virtualization - Sep 2015 - rfc: vhost user enhancements for vm2vm communication

On 2015-09-03 10:08, Michael S. Tsirkin wrote:
> On Tue, Sep 01, 2015 at 06:28:28PM +0200, Jan Kiszka wrote:
>> On 2015-09-01 18:02, Michael S. Tsirkin wrote:
>>> On Tue, Sep 01, 2015 at 05:34:37PM +0200, Jan Kiszka wrote:
>>>> On 2015-09-01 16:34, Michael S. Tsirkin wrote:
>>>>> On Tue, Sep 01, 2015 at 04:09:44PM +0200, Jan Kiszka wrote:
>>>>>> On 2015-09-01 11:24, Michael S. Tsirkin wrote:
>>>>>>> On Tue, Sep 01, 2015 at 11:11:52AM +0200, Jan
Kiszka wrote:
>>>>>>>> On 2015-09-01 10:01, Michael S. Tsirkin wrote:
>>>>>>>>> On Tue, Sep 01, 2015 at 09:35:21AM +0200,
Jan Kiszka wrote:
>>>>>>>>>> Leaving all the implementation and
interface details aside, this
>>>>>>>>>> discussion is first of all about two
fundamentally different approaches:
>>>>>>>>>> static shared memory windows vs.
dynamically remapped shared windows (a
>>>>>>>>>> third one would be copying in the
hypervisor, but I suppose we all agree
>>>>>>>>>> that the whole exercise is about
avoiding that). Which way do we want or
>>>>>>>>>> have to go?
>>>>>>>>>>
>>>>>>>>>> Jan
>>>>>>>>>
>>>>>>>>> Dynamic is a superset of static: you can
always make it static if you
>>>>>>>>> wish. Static has the advantage of
simplicity, but that's lost once you
>>>>>>>>> realize you need to invent interfaces to
make it work.  Since we can use
>>>>>>>>> existing IOMMU interfaces for the dynamic
one, what's the disadvantage?
>>>>>>>>
>>>>>>>> Complexity. Having to emulate even more of an
IOMMU in the hypervisor
>>>>>>>> (we already have to do a bit for VT-d IR in
Jailhouse) and doing this
>>>>>>>> per platform (AMD IOMMU, ARM SMMU, ...) is out
of scope for us. In that
>>>>>>>> sense, generic grant tables would be more
appealing.
>>>>>>>
>>>>>>> That's not how we do things for KVM, PV
features need to be
>>>>>>> modular and interchangeable with emulation.
>>>>>>
>>>>>> I know, and we may have to make some compromise for
Jailhouse if that
>>>>>> brings us valuable standardization and broad guest
support. But we will
>>>>>> surely not support an arbitrary amount of IOMMU models
for that reason.
>>>>>>
>>>>>>>
>>>>>>> If you just want something that's
cross-platform and easy to
>>>>>>> implement, just build a PV IOMMU. Maybe use virtio
for this.
>>>>>>
>>>>>> That is likely required to keep the complexity
manageable and to allow
>>>>>> static preconfiguration.
>>>>>
>>>>> Real IOMMU allow static configuration just fine. This is
exactly
>>>>> what VFIO uses.
>>>>
>>>> Please specify more precisely which feature in which IOMMU you
are
>>>> referring to. Also, given that you refer to VFIO, I suspect we
have
>>>> different thing in mind. I'm talking about an IOMMU device
model, like
>>>> the one we have in QEMU now for VT-d. That one is not at all
>>>> preconfigured by the host for VFIO.
>>>
>>> I really just mean that VFIO creates a mostly static IOMMU
configuration.
>>>
>>> It's configured by the guest, not the host.
>>
>> OK, that resolves my confusion.
>>
>>>
>>> I don't see host control over configuration as being
particularly important.
>>
>> We do, see below.
>>
>>>
>>>
>>>>>
>>>>>> Well, we could declare our virtio-shmem device to be an
IOMMU device
>>>>>> that controls access of a remote VM to RAM of the one
that owns the
>>>>>> device. In the static case, this access may at most be
enabled/disabled
>>>>>> but not moved around. The static regions would have to
be discoverable
>>>>>> for the VM (register read-back), and the guest's
firmware will likely
>>>>>> have to declare those ranges reserved to the guest OS.
>>>>>> In the dynamic case, the guest would be able to create
an alternative
>>>>>> mapping.
>>>>>
>>>>>
>>>>> I don't think we want a special device just to support
the
>>>>> static case. It might be a bit less code to write, but
>>>>> eventually it should be up to the guest.
>>>>> Fundamentally, it's policy that host has no business
>>>>> dictating.
>>>>
>>>> "A bit less" is to be validated, and I doubt its just
"a bit". But if
>>>> KVM and its guests will also support some PV-IOMMU that we can
reuse for
>>>> our scenarios, than that is fine. KVM would not have to mandate
support
>>>> for it while we would, that's all.
>>>
>>> Someone will have to do this work.
>>>
>>>>>
>>>>>> We would probably have to define a generic page table
structure
>>>>>> for that. Or do you rather have some MPU-like control
structure in mind,
>>>>>> more similar to the memory region descriptions vhost is
already using?
>>>>>
>>>>> I don't care much. Page tables use less memory if a lot
of memory needs
>>>>> to be covered. OTOH if you want to use virtio (e.g. to
allow command
>>>>> batching) that likely means commands to manipulate the
IOMMU, and
>>>>> maintaining it all on the host. You decide.
>>>>
>>>> I don't care very much about the dynamic case as we
won't support it
>>>> anyway. However, if the configuration concept used for it is
applicable
>>>> to static mode as well, then we could reuse it. But
preconfiguration
>>>> will required register-based region description, I suspect.
>>>
>>> I don't know what you mean by preconfiguration exactly.
>>>
>>> Do you want the host to configure the IOMMU? Why not let the
>>> guest do this?
>>
>> We simply freeze GPA-to-HPA mappings during runtime. Avoids having to
>> validate and synchronize guest-triggered changes.
> 
> Fine, but this assumes guest does very specific things, right?
> E.g. should guest reconfigure device's BAR, you would have
> to change GPA to HPA mappings?
> 
Yes, that's why we only support size exploration, not reallocation.
> 
>>>>>
>>>>>> Also not yet clear to me are how the vhost-pci device
and the
>>>>>> translations it will have to do should look like for
VM2.
>>>>>
>>>>> I think we can use vhost-pci BAR + VM1 bus address as the
>>>>> VM2 physical address. In other words, all memory exposed to
>>>>> virtio-pci by VM1 through it's IOMMU is mapped into BAR
of
>>>>> vhost-pci.
>>>>>
>>>>> Bus addresses can be validated to make sure they fit
>>>>> in the BAR.
>>>>
>>>> Sounds simple but may become challenging for VMs that have many
of such
>>>> devices (in order to connect to many possibly large VMs).
>>>
>>> You don't need to be able to map all guest memory if you know
>>> guest won't try to allow device access to all of it.
>>> It's a question of how good is the bus address allocator.
>>
>> But those BARs need to allocate a guest-physical address range as large
>> as the other guest's RAM is, possibly even larger if that RAM is
not
>> contiguous, and you can't put other resources into potential holes
>> because VM2 does not know where those holes will be.
> 
> No - only the RAM that you want addressable by VM2.
That's in the hand of VM1, not VM2 or the hypervisor, in case of
reconfigurable mapping. It's indeed a non-issue in our static case.
> 
> IOW if you wish, you actually can create a shared memory device,
> make it accessible to the IOMMU and place some or all
> data there.
> 
Actually, that could also be something more sophisticated, including
virtio-net, IF that device will be able to express its DMA window
restrictions (a bit like 32-bit PCI devices being restricted to <4G
addresses or ISA devices <1M).

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux

On Thu, Sep 03, 2015 at 10:21:28AM +0200, Jan Kiszka
wrote:
> On 2015-09-03 10:08, Michael S. Tsirkin wrote:
> > On Tue, Sep 01, 2015 at 06:28:28PM +0200, Jan Kiszka wrote:
> >> On 2015-09-01 18:02, Michael S. Tsirkin wrote:
> >>> On Tue, Sep 01, 2015 at 05:34:37PM +0200, Jan Kiszka wrote:
> >>>> On 2015-09-01 16:34, Michael S. Tsirkin wrote:
> >>>>> On Tue, Sep 01, 2015 at 04:09:44PM +0200, Jan Kiszka
wrote:
> >>>>>> On 2015-09-01 11:24, Michael S. Tsirkin wrote:
> >>>>>>> On Tue, Sep 01, 2015 at 11:11:52AM +0200, Jan
Kiszka wrote:
> >>>>>>>> On 2015-09-01 10:01, Michael S. Tsirkin
wrote:
> >>>>>>>>> On Tue, Sep 01, 2015 at 09:35:21AM
+0200, Jan Kiszka wrote:
> >>>>>>>>>> Leaving all the implementation and
interface details aside, this
> >>>>>>>>>> discussion is first of all about
two fundamentally different approaches:
> >>>>>>>>>> static shared memory windows vs.
dynamically remapped shared windows (a
> >>>>>>>>>> third one would be copying in the
hypervisor, but I suppose we all agree
> >>>>>>>>>> that the whole exercise is about
avoiding that). Which way do we want or
> >>>>>>>>>> have to go?
> >>>>>>>>>>
> >>>>>>>>>> Jan
> >>>>>>>>>
> >>>>>>>>> Dynamic is a superset of static: you
can always make it static if you
> >>>>>>>>> wish. Static has the advantage of
simplicity, but that's lost once you
> >>>>>>>>> realize you need to invent interfaces
to make it work.  Since we can use
> >>>>>>>>> existing IOMMU interfaces for the
dynamic one, what's the disadvantage?
> >>>>>>>>
> >>>>>>>> Complexity. Having to emulate even more of
an IOMMU in the hypervisor
> >>>>>>>> (we already have to do a bit for VT-d IR
in Jailhouse) and doing this
> >>>>>>>> per platform (AMD IOMMU, ARM SMMU, ...) is
out of scope for us. In that
> >>>>>>>> sense, generic grant tables would be more
appealing.
> >>>>>>>
> >>>>>>> That's not how we do things for KVM, PV
features need to be
> >>>>>>> modular and interchangeable with emulation.
> >>>>>>
> >>>>>> I know, and we may have to make some compromise
for Jailhouse if that
> >>>>>> brings us valuable standardization and broad guest
support. But we will
> >>>>>> surely not support an arbitrary amount of IOMMU
models for that reason.
> >>>>>>
> >>>>>>>
> >>>>>>> If you just want something that's
cross-platform and easy to
> >>>>>>> implement, just build a PV IOMMU. Maybe use
virtio for this.
> >>>>>>
> >>>>>> That is likely required to keep the complexity
manageable and to allow
> >>>>>> static preconfiguration.
> >>>>>
> >>>>> Real IOMMU allow static configuration just fine. This
is exactly
> >>>>> what VFIO uses.
> >>>>
> >>>> Please specify more precisely which feature in which IOMMU
you are
> >>>> referring to. Also, given that you refer to VFIO, I
suspect we have
> >>>> different thing in mind. I'm talking about an IOMMU
device model, like
> >>>> the one we have in QEMU now for VT-d. That one is not at
all
> >>>> preconfigured by the host for VFIO.
> >>>
> >>> I really just mean that VFIO creates a mostly static IOMMU
configuration.
> >>>
> >>> It's configured by the guest, not the host.
> >>
> >> OK, that resolves my confusion.
> >>
> >>>
> >>> I don't see host control over configuration as being
particularly important.
> >>
> >> We do, see below.
> >>
> >>>
> >>>
> >>>>>
> >>>>>> Well, we could declare our virtio-shmem device to
be an IOMMU device
> >>>>>> that controls access of a remote VM to RAM of the
one that owns the
> >>>>>> device. In the static case, this access may at
most be enabled/disabled
> >>>>>> but not moved around. The static regions would
have to be discoverable
> >>>>>> for the VM (register read-back), and the
guest's firmware will likely
> >>>>>> have to declare those ranges reserved to the guest
OS.
> >>>>>> In the dynamic case, the guest would be able to
create an alternative
> >>>>>> mapping.
> >>>>>
> >>>>>
> >>>>> I don't think we want a special device just to
support the
> >>>>> static case. It might be a bit less code to write, but
> >>>>> eventually it should be up to the guest.
> >>>>> Fundamentally, it's policy that host has no
business
> >>>>> dictating.
> >>>>
> >>>> "A bit less" is to be validated, and I doubt its
just "a bit". But if
> >>>> KVM and its guests will also support some PV-IOMMU that we
can reuse for
> >>>> our scenarios, than that is fine. KVM would not have to
mandate support
> >>>> for it while we would, that's all.
> >>>
> >>> Someone will have to do this work.
> >>>
> >>>>>
> >>>>>> We would probably have to define a generic page
table structure
> >>>>>> for that. Or do you rather have some MPU-like
control structure in mind,
> >>>>>> more similar to the memory region descriptions
vhost is already using?
> >>>>>
> >>>>> I don't care much. Page tables use less memory if
a lot of memory needs
> >>>>> to be covered. OTOH if you want to use virtio (e.g. to
allow command
> >>>>> batching) that likely means commands to manipulate the
IOMMU, and
> >>>>> maintaining it all on the host. You decide.
> >>>>
> >>>> I don't care very much about the dynamic case as we
won't support it
> >>>> anyway. However, if the configuration concept used for it
is applicable
> >>>> to static mode as well, then we could reuse it. But
preconfiguration
> >>>> will required register-based region description, I
suspect.
> >>>
> >>> I don't know what you mean by preconfiguration exactly.
> >>>
> >>> Do you want the host to configure the IOMMU? Why not let the
> >>> guest do this?
> >>
> >> We simply freeze GPA-to-HPA mappings during runtime. Avoids having
to
> >> validate and synchronize guest-triggered changes.
> > 
> > Fine, but this assumes guest does very specific things, right?
> > E.g. should guest reconfigure device's BAR, you would have
> > to change GPA to HPA mappings?
> > 
> 
> Yes, that's why we only support size exploration, not reallocation.
> 
> > 
> >>>>>
> >>>>>> Also not yet clear to me are how the vhost-pci
device and the
> >>>>>> translations it will have to do should look like
for VM2.
> >>>>>
> >>>>> I think we can use vhost-pci BAR + VM1 bus address as
the
> >>>>> VM2 physical address. In other words, all memory
exposed to
> >>>>> virtio-pci by VM1 through it's IOMMU is mapped
into BAR of
> >>>>> vhost-pci.
> >>>>>
> >>>>> Bus addresses can be validated to make sure they fit
> >>>>> in the BAR.
> >>>>
> >>>> Sounds simple but may become challenging for VMs that have
many of such
> >>>> devices (in order to connect to many possibly large VMs).
> >>>
> >>> You don't need to be able to map all guest memory if you
know
> >>> guest won't try to allow device access to all of it.
> >>> It's a question of how good is the bus address allocator.
> >>
> >> But those BARs need to allocate a guest-physical address range as
large
> >> as the other guest's RAM is, possibly even larger if that RAM
is not
> >> contiguous, and you can't put other resources into potential
holes
> >> because VM2 does not know where those holes will be.
> > 
> > No - only the RAM that you want addressable by VM2.
> 
> That's in the hand of VM1, not VM2 or the hypervisor, in case of
> reconfigurable mapping. It's indeed a non-issue in our static case.
> 
> > 
> > IOW if you wish, you actually can create a shared memory device,
> > make it accessible to the IOMMU and place some or all
> > data there.
> > 
> 
> Actually, that could also be something more sophisticated, including
> virtio-net, IF that device will be able to express its DMA window
> restrictions (a bit like 32-bit PCI devices being restricted to <4G
> addresses or ISA devices <1M).
> 
> Jan
Actually, it's the bus restriction, not the device restriction.

So if you want to use bounce buffers in the name of security or
real-time requirements, you should be able to do this if virtio uses the
DMA API.

> -- 
> Siemens AG, Corporate Technology, CT RTC ITP SES-DE
> Corporate Competence Center Embedded Linux

On 2015-09-03 10:37, Michael S. Tsirkin wrote:
> On Thu, Sep 03, 2015 at 10:21:28AM +0200, Jan Kiszka wrote:
>> On 2015-09-03 10:08, Michael S. Tsirkin wrote:
>>>
>>> IOW if you wish, you actually can create a shared memory device,
>>> make it accessible to the IOMMU and place some or all
>>> data there.
>>>
>>
>> Actually, that could also be something more sophisticated, including
>> virtio-net, IF that device will be able to express its DMA window
>> restrictions (a bit like 32-bit PCI devices being restricted to <4G
>> addresses or ISA devices <1M).
>>
>> Jan
> 
> Actually, it's the bus restriction, not the device restriction.
> 
> So if you want to use bounce buffers in the name of security or
> real-time requirements, you should be able to do this if virtio uses the
> DMA API.
Bounce buffer will only be the simplest option (though fine for low-rate
traffic that we also have in mind, like virtual consoles). Given
properly-sized regions, even if fixed, and the right communication
stacks, you can directly allocate application buffers in those regions
and avoid most to all copying.

In any case, if we manage to address this variation along with your
proposal, that would help tremendously.

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux