Xen devel - Jan 2008 - [XEN-IOMMU] Proposal of DMA protection/isolation support

hi list,
I am considering adding DMA protection/isolation support for iommu
machine:  Below are the suggested approaches to be discussed:

1) Para-virtualized IOMMU
If it is possible to integrate IOMMU driver into guest kernel, we can
just implement a set of para-virtualized interface to forward hardware
operations from guest to HV. Guest kernel will allocation IO page table
for itself, but IO-PTE updating is verified by HV through hypercall.

2) IOMMU-aware dma layer.
Currently, driver domain utilizes swiotlb to get dma_address below 4G,
which is an additional overhead to IOMMU machine. For IOMMU machine, we
can implement a new dma layer which takes "guest_domain-id",
"device_bdf", and "guest_page" information as parameters and
returns
virtual io address to guest OS. Guest OS only have very limited
knowledge/control to IOMMU. In this case, HV will allocate and update IO
page table for guest domain.

3) Hooking guest memory changes
No guest OS modification is needed in this approach.  All we need is to
update IO page table when guest physical memory changes triggered by
domain initialization, ballooning, and grant reference mapping...

Thanks for any comments, ideas, corrections... to this thread.

-Wei







_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Grant table mappings/unmappings are an obvious place where we already trap
to the hypervisor and could make correspodning changes to iommu mappings?

It depends if we want the iommu to do any more than prevent arbitrary DMA
access to foreign pages. What''s the threat model you are wanting to use
the
iommu to protect against?

 -- Keir

On 10/1/08 15:48, "Wei Wang2" <wei.wang2@amd.com> wrote:
> hi list,
> I am considering adding DMA protection/isolation support for iommu
> machine:  Below are the suggested approaches to be discussed:
> 
> 1) Para-virtualized IOMMU
> If it is possible to integrate IOMMU driver into guest kernel, we can
> just implement a set of para-virtualized interface to forward hardware
> operations from guest to HV. Guest kernel will allocation IO page table
> for itself, but IO-PTE updating is verified by HV through hypercall.
> 
> 2) IOMMU-aware dma layer.
> Currently, driver domain utilizes swiotlb to get dma_address below 4G,
> which is an additional overhead to IOMMU machine. For IOMMU machine, we
> can implement a new dma layer which takes "guest_domain-id",
> "device_bdf", and "guest_page" information as
parameters and returns
> virtual io address to guest OS. Guest OS only have very limited
> knowledge/control to IOMMU. In this case, HV will allocate and update IO
> page table for guest domain.
> 
> 3) Hooking guest memory changes
> No guest OS modification is needed in this approach.  All we need is to
> update IO page table when guest physical memory changes triggered by
> domain initialization, ballooning, and grant reference mapping...
> 
> Thanks for any comments, ideas, corrections... to this thread.
> 
> -Wei
> 
> 
> 
> 
> 
> 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Thu, 2008-01-10 at 15:54 +0000, Keir Fraser wrote:
> Grant table mappings/unmappings are an obvious place where we already trap
> to the hypervisor and could make correspodning changes to iommu mappings?
Can grant mapping cover the situation in which a device only be accessed
by a driver domain other than be shared with any remote domain? In other
word, when a device is only access by a driver domain, does grant table
mapping still happen? If yes, it is the best way to go.
> It depends if we want the iommu to do any more than prevent arbitrary DMA
> access to foreign pages. What''s the threat model you are wanting
to use the
> iommu to protect against?
I think IOMMU can help to prevent buggy driver from destroying memory content of
both
driver domain itself and foreign domain. Proper IO address which is
requested by device driver should only be provided by some pre-defined
interfaces/hypercalls.  Arbitrary dma addresses written to a device by a
buggy driver will not trigger address translations.

-Wei
>  -- Keir
> 
> On 10/1/08 15:48, "Wei Wang2" <wei.wang2@amd.com> wrote:
> 
> > hi list,
> > I am considering adding DMA protection/isolation support for iommu
> > machine:  Below are the suggested approaches to be discussed:
> > 
> > 1) Para-virtualized IOMMU
> > If it is possible to integrate IOMMU driver into guest kernel, we can
> > just implement a set of para-virtualized interface to forward hardware
> > operations from guest to HV. Guest kernel will allocation IO page
table
> > for itself, but IO-PTE updating is verified by HV through hypercall.
> > 
> > 2) IOMMU-aware dma layer.
> > Currently, driver domain utilizes swiotlb to get dma_address below 4G,
> > which is an additional overhead to IOMMU machine. For IOMMU machine,
we
> > can implement a new dma layer which takes "guest_domain-id",
> > "device_bdf", and "guest_page" information as
parameters and returns
> > virtual io address to guest OS. Guest OS only have very limited
> > knowledge/control to IOMMU. In this case, HV will allocate and update
IO
> > page table for guest domain.
> > 
> > 3) Hooking guest memory changes
> > No guest OS modification is needed in this approach.  All we need is
to
> > update IO page table when guest physical memory changes triggered by
> > domain initialization, ballooning, and grant reference mapping...
> > 
> > Thanks for any comments, ideas, corrections... to this thread.
> > 
> > -Wei
> > 
> > 
> > 
> > 
> > 
> > 
> 
> 
> 
> 



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Grant mappings will only be triggered for I/O to/from foreign domains.
I''m
not really convinced that protecting a driver domain''s own memory
against
errant DMAs is that important anyway. Firstly, there are many other ways
that a buggy driver can screw its domain, other than errant DMA. Secondly,
any driver that haflway works will request a DMA mapping from the OS before
it initiates any DMA (otherwise the driver would *never* work) and that
would probably be the point at which the OS would set up the iommu mapping.
That''s the problem -- the OS will be trusting the driver to tell it
when a
mapping should be set up, and that request will usually be co-located in the
driver code with the actual DMA initiation. So if the driver is issuing
errant DMAs, the OS is rather likely to let them happen!

 -- Keir


On 10/1/08 16:52, "Wei Wang2" <wei.wang2@amd.com> wrote:
> On Thu, 2008-01-10 at 15:54 +0000, Keir Fraser wrote:
>> Grant table mappings/unmappings are an obvious place where we already
trap
>> to the hypervisor and could make correspodning changes to iommu
mappings?
> Can grant mapping cover the situation in which a device only be accessed
> by a driver domain other than be shared with any remote domain? In other
> word, when a device is only access by a driver domain, does grant table
> mapping still happen? If yes, it is the best way to go.
> 
>> It depends if we want the iommu to do any more than prevent arbitrary
DMA
>> access to foreign pages. What''s the threat model you are
wanting to use the
>> iommu to protect against?
> I think IOMMU can help to prevent buggy driver from destroying memory
content
> of both 
> driver domain itself and foreign domain. Proper IO address which is
> requested by device driver should only be provided by some pre-defined
> interfaces/hypercalls.  Arbitrary dma addresses written to a device by a
> buggy driver will not trigger address translations.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

If a halfway work can trigger any de-initialization routines, perhaps
device driver will deallocate dma pages which are in use and iommu
unmapping can be triggered?

-Wei

On Thu, 2008-01-10 at 16:58 +0000, Keir Fraser wrote:
> Grant mappings will only be triggered for I/O to/from foreign domains.
I''m
> not really convinced that protecting a driver domain''s own memory
against
> errant DMAs is that important anyway. Firstly, there are many other ways
> that a buggy driver can screw its domain, other than errant DMA. Secondly,
> any driver that haflway works will request a DMA mapping from the OS before
> it initiates any DMA (otherwise the driver would *never* work) and that
> would probably be the point at which the OS would set up the iommu mapping.
> That''s the problem -- the OS will be trusting the driver to tell
it when a
> mapping should be set up, and that request will usually be co-located in
the
> driver code with the actual DMA initiation. So if the driver is issuing
> errant DMAs, the OS is rather likely to let them happen!
> 
>  -- Keir
> 
> 
> On 10/1/08 16:52, "Wei Wang2" <wei.wang2@amd.com> wrote:
> 
> > On Thu, 2008-01-10 at 15:54 +0000, Keir Fraser wrote:
> >> Grant table mappings/unmappings are an obvious place where we
already trap
> >> to the hypervisor and could make correspodning changes to iommu
mappings?
> > Can grant mapping cover the situation in which a device only be
accessed
> > by a driver domain other than be shared with any remote domain? In
other
> > word, when a device is only access by a driver domain, does grant
table
> > mapping still happen? If yes, it is the best way to go.
> > 
> >> It depends if we want the iommu to do any more than prevent
arbitrary DMA
> >> access to foreign pages. What''s the threat model you are
wanting to use the
> >> iommu to protect against?
> > I think IOMMU can help to prevent buggy driver from destroying memory
content
> > of both 
> > driver domain itself and foreign domain. Proper IO address which is
> > requested by device driver should only be provided by some pre-defined
> > interfaces/hypercalls.  Arbitrary dma addresses written to a device by
a
> > buggy driver will not trigger address translations.
> 
> 
> 
> 



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Thursday 10 January 2008 21:18:24 Wei Wang2 wrote:
> hi list,
> I am considering adding DMA protection/isolation support for iommu
> machine:  Below are the suggested approaches to be discussed:
>
> 1) Para-virtualized IOMMU
> If it is possible to integrate IOMMU driver into guest kernel, we can
> just implement a set of para-virtualized interface to forward hardware
> operations from guest to HV. Guest kernel will allocation IO page table
> for itself, but IO-PTE updating is verified by HV through hypercall.
>
> 2) IOMMU-aware dma layer.
> Currently, driver domain utilizes swiotlb to get dma_address below 4G,
> which is an additional overhead to IOMMU machine. For IOMMU machine, we
> can implement a new dma layer which takes "guest_domain-id",
> "device_bdf", and "guest_page" information as
parameters and returns
> virtual io address to guest OS. Guest OS only have very limited
> knowledge/control to IOMMU. In this case, HV will allocate and update IO
> page table for guest domain.
>
> 3) Hooking guest memory changes
> No guest OS modification is needed in this approach.  All we need is to
> update IO page table when guest physical memory changes triggered by
> domain initialization, ballooning, and grant reference mapping...
>
> Thanks for any comments, ideas, corrections... to this thread.
I have a few patches (quite old, I need to refresh them) for doing this. I was 
also looking at integrating this functionality with the AMD DEV for securing 
accesses.

This effort was mostly done with kvm+qemu in mind, but the DMA-level 
operations should work across any hypervisor on a PV guest.

http://lkml.org/lkml/2007/11/7/125

Let me know if these patches can be helpful and I can refresh them to the 
newer kernels.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir, 
After looking into the grant mapping code (use block device as an
example), I got some ideas to share with you and the list.  Please
correct me if I missed something. There might be 2 places to trigger
iommu mapping:

1) Before submitting io request to generic block layer, block backend
driver will invoke GNTTABOPs to map granted remote pages .  However
creating iommu mapping only in GNTTABOP not very enough because local
pages will also be submitted to block layer and they should also be
translated by iommu. 

2) When dma layer is invoked to prepare a bus address for native block
device driver. It would be better to have a "hypervisor-aware" dma
layer
which always maps virtual address to bus address via hypercall.
__gnttab_dma_map_page() might be a good place to trigger this new
hypercall. Both remote and local pages can be mapped by this way.

-Wei
 
On Thu, 2008-01-10 at 17:58 +0100, Keir Fraser wrote:
> Grant mappings will only be triggered for I/O to/from foreign domains.
> I''m
> not really convinced that protecting a driver domain''s own memory
> against
> errant DMAs is that important anyway. Firstly, there are many other
> ways
> that a buggy driver can screw its domain, other than errant DMA.
> Secondly,
> any driver that haflway works will request a DMA mapping from the OS
> before
> it initiates any DMA (otherwise the driver would *never* work) and
> that
> would probably be the point at which the OS would set up the iommu
> mapping.
> That''s the problem -- the OS will be trusting the driver to tell
it
> when a
> mapping should be set up, and that request will usually be co-located
> in the
> driver code with the actual DMA initiation. So if the driver is
> issuing
> errant DMAs, the OS is rather likely to let them happen!
> 
>  -- Keir
> 
> 
> On 10/1/08 16:52, "Wei Wang2" <wei.wang2@amd.com> wrote:
> 
> > On Thu, 2008-01-10 at 15:54 +0000, Keir Fraser wrote:
> >> Grant table mappings/unmappings are an obvious place where we
> already trap
> >> to the hypervisor and could make correspodning changes to iommu
> mappings?
> > Can grant mapping cover the situation in which a device only be
> accessed
> > by a driver domain other than be shared with any remote domain? In
> other
> > word, when a device is only access by a driver domain, does grant
> table
> > mapping still happen? If yes, it is the best way to go.
> >
> >> It depends if we want the iommu to do any more than prevent
> arbitrary DMA
> >> access to foreign pages. What''s the threat model you are
wanting to
> use the
> >> iommu to protect against?
> > I think IOMMU can help to prevent buggy driver from destroying
> memory content
> > of both
> > driver domain itself and foreign domain. Proper IO address which is
> > requested by device driver should only be provided by some
> pre-defined
> > interfaces/hypercalls.  Arbitrary dma addresses written to a device
> by a
> > buggy driver will not trigger address translations.
> 
> 
> 
> 
> 
> 



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 16/1/08 16:34, "Wei Wang2" <wei.wang2@amd.com> wrote:
> 1) Before submitting io request to generic block layer, block backend
> driver will invoke GNTTABOPs to map granted remote pages .  However
> creating iommu mapping only in GNTTABOP not very enough because local
> pages will also be submitted to block layer and they should also be
> translated by iommu.
Local pages could always be mapped into the iommu, and only remote pages
would need to be temporarily mapped via the gnttab hooks.
> 2) When dma layer is invoked to prepare a bus address for native block
> device driver. It would be better to have a "hypervisor-aware"
dma layer
> which always maps virtual address to bus address via hypercall.
> __gnttab_dma_map_page() might be a good place to trigger this new
> hypercall. Both remote and local pages can be mapped by this way.
An extra hypercall here will likely hurt performance.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Wed, 2008-01-16 at 16:45 +0000, Keir Fraser wrote:
> On 16/1/08 16:34, "Wei Wang2" <wei.wang2@amd.com> wrote:
> 
> > 1) Before submitting io request to generic block layer, block backend
> > driver will invoke GNTTABOPs to map granted remote pages .  However
> > creating iommu mapping only in GNTTABOP not very enough because local
> > pages will also be submitted to block layer and they should also be
> > translated by iommu.
> 
> Local pages could always be mapped into the iommu, and only remote pages
> would need to be temporarily mapped via the gnttab hooks.
I remembered that vtd patch used to hook page assignment to for this,
but it was soon replaced with statical 1:1 mapping. Could you suggest me
how to trace local page changes instead of mapping whole the physical
ram?
> 2) When dma layer is invoked to prepare a bus address for native block
> > device driver. It would be better to have a
"hypervisor-aware" dma layer
> > which always maps virtual address to bus address via hypercall.
> > __gnttab_dma_map_page() might be a good place to trigger this new
> > hypercall. Both remote and local pages can be mapped by this way.
> 
> An extra hypercall here will likely hurt performance.
I have implemented a prototype for this which removes 1:1 mapping and invoke a
hypercall for
dma mapping. Since block driver uses dma_map_sg heavily which is easy to batch,
I have not seen dramatic performance loss.

-Wei

 
>  -- Keir
> 
> 
> 
> 
-Wei




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 16/1/08 17:15, "Wei Wang2" <wei.wang2@amd.com> wrote:
>> Local pages could always be mapped into the iommu, and only remote
pages
>> would need to be temporarily mapped via the gnttab hooks.
> I remembered that vtd patch used to hook page assignment to for this,
> but it was soon replaced with statical 1:1 mapping. Could you suggest me
> how to trace local page changes instead of mapping whole the physical
> ram?
alloc_domheap_pages() if you want to map MFNs. populate_physmap() if you
want to map PFNs. There might be a few other places to hook also but I
expect they''re the main ones.
>> 2) When dma layer is invoked to prepare a bus address for native block
>>> device driver. It would be better to have a
"hypervisor-aware" dma layer
>>> which always maps virtual address to bus address via hypercall.
>>> __gnttab_dma_map_page() might be a good place to trigger this new
>>> hypercall. Both remote and local pages can be mapped by this way.
>> 
>> An extra hypercall here will likely hurt performance.
> I have implemented a prototype for this which removes 1:1 mapping and
invoke a
> hypercall for 
> dma mapping. Since block driver uses dma_map_sg heavily which is easy to
> batch, I have not seen dramatic performance loss.
It''s hard to slow down block-device accesses. What does it do to CPU
usage
and max bandwidth on a multi-gigabit NIC?

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> After looking into the grant mapping code (use block device as an
> example), I got some ideas to share with you and the list.  Please
> correct me if I missed something. There might be 2 places to trigger
> iommu mapping:
> 
> 1) Before submitting io request to generic block layer, block backend
> driver will invoke GNTTABOPs to map granted remote pages .  However
> creating iommu mapping only in GNTTABOP not very enough because local
> pages will also be submitted to block layer and they should also be
> translated by iommu.
> 
> 2) When dma layer is invoked to prepare a bus address for native block
> device driver. It would be better to have a "hypervisor-aware"
dma
> layer
> which always maps virtual address to bus address via hypercall.
> __gnttab_dma_map_page() might be a good place to trigger this new
> hypercall. Both remote and local pages can be mapped by this way.

It''s much better to create most mappings in #1 when the operations are
easily batchable into a single hypercall, and then fill in the missing
mappings when required at #2.

Since in many situations dom0 does very little IO for itself, this
should be a significant win, especially for networking.

Ian
 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

hi
IOMMU, could it be used in KVM ,or not? if it could , what about the 
difference between the usage in xen and the usage in KVM?

Thanks




Ian Pratt 写道:
>> After looking into the grant mapping code (use block device as an
>> example), I got some ideas to share with you and the list.  Please
>> correct me if I missed something. There might be 2 places to trigger
>> iommu mapping:
>>
>> 1) Before submitting io request to generic block layer, block backend
>> driver will invoke GNTTABOPs to map granted remote pages .  However
>> creating iommu mapping only in GNTTABOP not very enough because local
>> pages will also be submitted to block layer and they should also be
>> translated by iommu.
>>
>> 2) When dma layer is invoked to prepare a bus address for native block
>> device driver. It would be better to have a
"hypervisor-aware" dma
>> layer
>> which always maps virtual address to bus address via hypercall.
>> __gnttab_dma_map_page() might be a good place to trigger this new
>> hypercall. Both remote and local pages can be mapped by this way.
>>     
>
>
> It''s much better to create most mappings in #1 when the operations
are
> easily batchable into a single hypercall, and then fill in the missing
> mappings when required at #2.
>
> Since in many situations dom0 does very little IO for itself, this
> should be a significant win, especially for networking.
>
> Ian
>  
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>
>
>   

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel