Xen devel - May 2008 - [PATCH] AMD IOMMU: Hanlde sibling device assignment correctly

Devices on AMD IOMMU system share a same set of IO page tables if they
are indexed by a same device ID in device table. To prevent these
sibling devices from being assigned to different domains, a successful
device assignment requires all sibling devices are owned by  either
pciback or the target domain. This is a trial patch with minor
modifications in the common interface to handle this situation
correctly.  VTD code has also been slightly touched please let me know
if it breaks any VTD functionality. Thanks!

Wei

Signed-off-by: Wei Wang <wei.wang2@amd.com>
-- 
AMD Saxony, Dresden, Germany
Operating System Research Center

Legal Information:
AMD Saxony Limited Liability Company & Co. KG
Sitz (Geschäftsanschrift):
   Wilschdorfer Landstr. 101, 01109 Dresden, Deutschland
Registergericht Dresden: HRA 4896
vertretungsberechtigter Komplementär:
   AMD Saxony LLC (Sitz Wilmington, Delaware, USA)
Geschäftsführer der AMD Saxony LLC:
   Dr. Hans-R. Deppe, Thomas McCoy 



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Wed, May 07, 2008 at 11:27:01AM +0200, Wei Wang2 wrote:
> Devices on AMD IOMMU system share a same set of IO page tables if
> they are indexed by a same device ID in device table.
Hi Wei,

Could you please explain in which cases two devices would end up
sharing the same device ID? Am I correct in the assumption that this
can only happen for devices on the HT fabric, rather than for
PCI/PCI-e devices?

Thanks,
Muli

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 7/5/08 10:27, "Wei Wang2" <wei.wang2@amd.com> wrote:
> Devices on AMD IOMMU system share a same set of IO page tables if they
> are indexed by a same device ID in device table. To prevent these
> sibling devices from being assigned to different domains, a successful
> device assignment requires all sibling devices are owned by  either
> pciback or the target domain. This is a trial patch with minor
> modifications in the common interface to handle this situation
> correctly.  VTD code has also been slightly touched please let me know
> if it breaks any VTD functionality. Thanks!
This patch seems to do more than you suggest, for example adding an extra
iommu hook into setup.c for dom0. Also it does get_domain_by_id() with no
corresponding put_domain(). And in fact you should be able to
rcu_lock_domain_by_id()/rcu_unlock_domain() instead which is faster.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Wed, 2008-05-07 at 12:53 +0300, Muli Ben-Yehuda
wrote:
> On Wed, May 07, 2008 at 11:27:01AM +0200, Wei Wang2 wrote:
> 
> > Devices on AMD IOMMU system share a same set of IO page tables if
> > they are indexed by a same device ID in device table.
> 
> Hi Wei,
> 
> Could you please explain in which cases two devices would end up
> sharing the same device ID? Am I correct in the assumption that this
> can only happen for devices on the HT fabric, rather than for
> PCI/PCI-e devices?
I think both could be possible. If IOMMU is implemented per PCI-to-PCI-E
bridge, then probably there will be a group of PCI device sharing a same
PCI-E requester ID. The device table can only be indexed by either PCI-E
requester ID or HT unit ID.

Wei

> Thanks,
> Muli
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Wed, 2008-05-07 at 10:53 +0100, Keir Fraser wrote:
> 
> 
> On 7/5/08 10:27, "Wei Wang2" <wei.wang2@amd.com> wrote:
> 
> > Devices on AMD IOMMU system share a same set of IO page tables if they
> > are indexed by a same device ID in device table. To prevent these
> > sibling devices from being assigned to different domains, a successful
> > device assignment requires all sibling devices are owned by  either
> > pciback or the target domain. This is a trial patch with minor
> > modifications in the common interface to handle this situation
> > correctly.  VTD code has also been slightly touched please let me know
> > if it breaks any VTD functionality. Thanks!
> 
> This patch seems to do more than you suggest, for example adding an extra
> iommu hook into setup.c for dom0. 
My idea is to let dom0 construct pci device list according to
configuration of pciback.hide=(). If a device is not hidden from dom0,
it might be in use by dom0, then it could be dangerous to assign any of
its siblings to other passthru domain. It is not very clean to hook into
setup.c but I failed to find any better way to this  :(
> Also it does get_domain_by_id() with no
> corresponding put_domain(). And in fact you should be able to
> rcu_lock_domain_by_id()/rcu_unlock_domain() instead which is faster.
Thanks, I will update that.

>  -- Keir
> 
> 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 7/5/08 11:17, "Wei Wang2" <wei.wang2@amd.com> wrote:
>> This patch seems to do more than you suggest, for example adding an
extra
>> iommu hook into setup.c for dom0.
> My idea is to let dom0 construct pci device list according to
> configuration of pciback.hide=(). If a device is not hidden from dom0,
> it might be in use by dom0, then it could be dangerous to assign any of
> its siblings to other passthru domain. It is not very clean to hook into
> setup.c but I failed to find any better way to this  :(
I might be confused about how this works. Are you saying that if a domU gets
a device passed-thru that is a sibling of a dom0-driven device, then dom0
will mistakenly have its device''s DMAs remapped according to the domU
mappings that get set up?

Is it possible to do this more dynamically from the tools? I.e, working out
sibling groups at the time we build a pass-thru domain, and at that point
fail the pass-thru if dom0 or any other domU is driving any device in the
group? Seems a nicer level to do the checks and raise errors. You might want
to some conflict checking in Xen, of course, just to sanity-check that noone
tries to set up two different mappings on the same HT or PCIe identifier.

Basically I think this patch is ugly. :-)

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 7 May 2008, at 2:09pm, Keir Fraser wrote:
> On 7/5/08 11:17, "Wei Wang2" <wei.wang2@amd.com> wrote:
>
>>> This patch seems to do more than you suggest, for example adding  
>>> an extra
>>> iommu hook into setup.c for dom0.
>> My idea is to let dom0 construct pci device list according to
>> configuration of pciback.hide=(). If a device is not hidden from  
>> dom0,
>> it might be in use by dom0, then it could be dangerous to assign  
>> any of
>> its siblings to other passthru domain. It is not very clean to hook  
>> into
>> setup.c but I failed to find any better way to this  :(
>
> I might be confused about how this works. Are you saying that if a  
> domU gets
> a device passed-thru that is a sibling of a dom0-driven device, then  
> dom0
> will mistakenly have its device''s DMAs remapped according to the
domU
> mappings that get set up?
If a device is behind a PCI-E to PCI bridge there are cases where  
transactions from the device are re-written by the bridge to use the  
requestor id of the bridge. Given that the requestor id is the token  
used by the IOMMU to determine the domain which initiated the IO, this  
effectively means that devices behind a PCI-E to PCI bridge are not  
divisible - they (and the bridge) must all be assigned to the same  
domain.

At least, that''s my understanding :-/


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

David Edmondson wrote:
> On 7 May 2008, at 2:09pm, Keir Fraser wrote:
>> On 7/5/08 11:17, "Wei Wang2" <wei.wang2@amd.com> wrote:
>> 
>>>> This patch seems to do more than you suggest, for example
adding
>>>> an extra iommu hook into setup.c for dom0.
>>> My idea is to let dom0 construct pci device list according to
>>> configuration of pciback.hide=(). If a device is not hidden from
>>> dom0, it might be in use by dom0, then it could be dangerous to
>>> assign any of its siblings to other passthru domain. It is not very
>>> clean to hook into setup.c but I failed to find any better way to
>>> this  :( 
>> 
>> I might be confused about how this works. Are you saying that if a
>> domU gets a device passed-thru that is a sibling of a dom0-driven
>> device, then dom0 will mistakenly have its device''s DMAs
remapped
>> according to the domU mappings that get set up?
> 
> If a device is behind a PCI-E to PCI bridge there are cases where
> transactions from the device are re-written by the bridge to use the
> requestor id of the bridge. Given that the requestor id is the token
> used by the IOMMU to determine the domain which initiated the IO, this
> effectively means that devices behind a PCI-E to PCI bridge are not
> divisible - they (and the bridge) must all be assigned to the same
> domain.
> 
> At least, that''s my understanding :-/
> 
For VT-d, your understanding is correct. Devices behind PCI-E to
PCI/PCI-X bridges can only be collectively assigned to a
single domain. But now it''s not implemented yet in Xen.

Randy (Weidong)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Wed, 2008-05-07 at 14:09 +0100, Keir Fraser wrote:
> On 7/5/08 11:17, "Wei Wang2" <wei.wang2@amd.com> wrote:
> 
> >> This patch seems to do more than you suggest, for example adding
an extra
> >> iommu hook into setup.c for dom0.
> > My idea is to let dom0 construct pci device list according to
> > configuration of pciback.hide=(). If a device is not hidden from dom0,
> > it might be in use by dom0, then it could be dangerous to assign any
of
> > its siblings to other passthru domain. It is not very clean to hook
into
> > setup.c but I failed to find any better way to this  :(
> 
> I might be confused about how this works. Are you saying that if a domU
gets
> a device passed-thru that is a sibling of a dom0-driven device, then dom0
> will mistakenly have its device''s DMAs remapped according to the
domU
> mappings that get set up?
I have encountered a problem on my testing machine. When I assign a usb
controller ,which shares the same device ID of a sata controller belong
to dom0, dom0 hangs immediately.  The current solution is that when
hiding a pci device from dom0, its sibling device must also be hidden.
Therefore, I need to parse pciback.hide from Xen side..
> Is it possible to do this more dynamically from the tools? I.e, working out
> sibling groups at the time we build a pass-thru domain, and at that point
> fail the pass-thru if dom0 or any other domU is driving any device in the
> group? Seems a nicer level to do the checks and raise errors. You might
want
> to some conflict checking in Xen, of course, just to sanity-check that
noone
> tries to set up two different mappings on the same HT or PCIe identifier.
Yes, that is my first thought. The question is only dom0 knows about
pciback.hide information. Current iommu code construct device list  no
matter whether devices are actually hidden from dom0. Is it better to
just allow hidden devices to be added into the device list?
> Basically I think this patch is ugly. :-)
Yes, I admit that.  the purpose of this patch is maybe just raise a problem for
further discussion. ;)

Wei
>  -- Keir
> 
> 
> 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Wed, May 07, 2008 at 12:04:51PM +0200, Wei Wang2 wrote:
> I think both could be possible. If IOMMU is implemented per
> PCI-to-PCI-E bridge, then probably there will be a group of PCI
> device sharing a same PCI-E requester ID. The device table can only
> be indexed by either PCI-E requester ID or HT unit ID.
Good point. That should be a fairly rare case. It will also apply
equally well to VT-d, so this should probably be common functionality.

Cheers,
Muli

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel