Xen devel - Jan 2010 - [PATCH][VT-d] Dis-allow PCI device assignment if PoD is enabled

It seems that currently we don''t have any code to handle
the coexistence of VT-d and PoD. VT-d engine needs to set up
the entire page table for the domain. However if PoD is enabled,
un-populated memory is marked as populate_on_demand, and
VT-d engine won''t set up page tables for them. Therefore any
DMA towards those memory may cause DMA fault.
	So for safety concern, its better to dis-allow PCI device
assignment if PoD is enabled.

Best Regards, 
-- Dongxiao


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Seems like a good "seatbelt" for 4.0.

Looking forward, what would it take to make PoD and VT-d coexist?  We 
only need PoD during boot, until the balloon driver comes up and 
balloons down the guest''s memory.  Three solutions come to mind, but as
I don''t know the constraints of VT-d, I don''t know which is
feasible (if
any):
* Redo the VT-d mapping every time the p2m map changes as a result of PoD.
* While PoD pages exist, intercept device commands, and redo the VT-d 
map if the page was marked PoD the last time we updated the VT-d map
* Detect DMA faults, instantiate the page if necessary, update the VT-d 
map, and re-start the transaction.

How expensive is it to change the VT-d pagetable?  Is a DMA fault 
re-startable?  i.e., could we take a fault, redo the VT-d map, and 
re-issue the DMA request?

 -George

Xu, Dongxiao wrote:
> It seems that currently we don''t have any code to handle
> the coexistence of VT-d and PoD. VT-d engine needs to set up
> the entire page table for the domain. However if PoD is enabled,
> un-populated memory is marked as populate_on_demand, and
> VT-d engine won''t set up page tables for them. Therefore any
> DMA towards those memory may cause DMA fault.
> 	So for safety concern, its better to dis-allow PCI device
> assignment if PoD is enabled.
>
> Best Regards, 
> -- Dongxiao
>   

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> Looking forward, what would it take to make PoD and VT-d coexist?  We
> only need PoD during boot, until the balloon driver comes up and
> balloons down the guest''s memory.  Three solutions come to mind,
but as
> I don''t know the constraints of VT-d, I don''t know which
is feasible (if
> any):
> * Redo the VT-d mapping every time the p2m map changes as a result of PoD.
> * While PoD pages exist, intercept device commands, and redo the VT-d
> map if the page was marked PoD the last time we updated the VT-d map
> * Detect DMA faults, instantiate the page if necessary, update the VT-d
> map, and re-start the transaction.
> 
> How expensive is it to change the VT-d pagetable?  Is a DMA fault
> re-startable?  i.e., could we take a fault, redo the VT-d map, and
> re-issue the DMA request?
IOMMU faults are not restartable, at least currently.

Flushing the IOTLB is very expensive. Fortunately -ve entries are not cached,
and since PoD mainly adds new mappings we should be fine. The zeroed page
reclaimation stuff is a bit more dicey and would require syncrhonization against
an IOTLB flush before returning the pages to xen.

In many/most cases the device will not be in use that early in boot, so
it''s a bit annoying to have to do maintain the IOMMU pagetables through
PoD, but unavoidable. The key thing is that we only have to do it for domains
that actually have devices passed-through.

Ian

 



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Thu, Jan 21, 2010 at 6:02 PM, Ian Pratt <Ian.Pratt@eu.citrix.com>
wrote:
> In many/most cases the device will not be in use that early in boot, so
it''s a bit annoying to have to do maintain the IOMMU pagetables through
PoD, but unavoidable. The key thing is that we only have to do it for domains
that actually have devices passed-through.
At the moment, I can''t imagine how IOMMU/VT-d can interact well with
PoD during boot, before the balloon driver gets in and does its thing.
It''s guaranteed during that time that a high percentage of the memory
which the guest thinks it has free will be not-present in the p2m.
There''s no way we can predict which gfns will be passed to the device;
having been zeroed (and thus populated) is no help, since a
non-negligible percentage of zeroed pages will need to be reclaimed
for the PoD pool again anyway.

If it really is true that devices aren''t used during boot, then we
could simply have the balloon driver / the tools do a final "sync"
once the "target" has been reached (and outstanding PoD entries =size
of PoD memory pool).  Doing more than that (say, syncing on every
p2m update) doesn''t solve the problem (although I suppose it may be
necessary to prevent corruption).

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel