Xen devel - Sep 2013 - Multi-bridged PCIe devices (Was: Re: iommuu/vt-d issues with LSI MegaSAS (PERC5i))

I found this:

 http://lists.xen.org/archives/html/xen-devel/2010-06/msg00093.html

 while looking for a solution to a similar problem. I am
 facing a similar issue with LSI (8408E, 3081E-R) and
 Adaptec (31605) SAS cards. Was there ever a proper, more general
 fix or workaround for this issue?

 These SAS cards experience these problems in dom0. When running
 a vanilla kernel on bare metal, they work OK without intel_iommu
 set. As soon as I set intel_iommu, the same thing happens (on
 bare metal, not dom0).

 Clearly there is something badly broken with multiple layers
 of bridges when it comes to IOMMU in my setup (Intel 5520 PCIe
 root hub -> NF200 bridge -> Intel 80333 Bridge -> SAS controller)

 I tried iommu=dom0-passthrough and it doesn''t appear to have
 helped.

 I am not seeing similar problems with other PCIe devices that
 are also, in theory doing DMA (e.g. GPUs), but LSI and Adapted
 controllers appear to be affected for some reason.

 Is there anything else I could try/do to make this work?

 Gordan

It looks like I''m definitely not the first person to
 hit this problem:

 http://www.gossamer-threads.com/lists/xen/users/168557?do=post_view_threaded#168557

 No responses or workarounds suggested back then. :(

 Gordan

 On Wed, 11 Sep 2013 12:05:35 +0100, Gordan Bobic <gordan@bobich.net> 
 wrote:
> I found this:
>
> http://lists.xen.org/archives/html/xen-devel/2010-06/msg00093.html
>
> while looking for a solution to a similar problem. I am
> facing a similar issue with LSI (8408E, 3081E-R) and
> Adaptec (31605) SAS cards. Was there ever a proper, more general
> fix or workaround for this issue?
>
> These SAS cards experience these problems in dom0. When running
> a vanilla kernel on bare metal, they work OK without intel_iommu
> set. As soon as I set intel_iommu, the same thing happens (on
> bare metal, not dom0).
>
> Clearly there is something badly broken with multiple layers
> of bridges when it comes to IOMMU in my setup (Intel 5520 PCIe
> root hub -> NF200 bridge -> Intel 80333 Bridge -> SAS controller)
>
> I tried iommu=dom0-passthrough and it doesn''t appear to have
> helped.
>
> I am not seeing similar problems with other PCIe devices that
> are also, in theory doing DMA (e.g. GPUs), but LSI and Adapted
> controllers appear to be affected for some reason.
>
> Is there anything else I could try/do to make this work?
>
> Gordan
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

This got me thinking - if the problem is broken IOMMU implementation,
 is the IOMMU _actually_ required for PCI passthrough to HVM
 guests if all the memory holes and BARs are made exactly the same
 in dom0 and domU? If vBAR=pBAR, then surely there is no memory
 range remapping to be done anyway - which means that there
 is no need for the strict IOMMU requirements (over and above
 the requirements and caveats of PV domUs).

 In turn, this would enable PCI passthrough (incl. secondary VGA,
 unless I am very much mistaken) to HVM guests while running xen
 with iommu=0. It shifts the design from virtualization to
 partitioning, which I see having obvious advantages and no
 disadvantages (e.g. VM migration doesn''t work with PCI
 passthrough anyway).

 The reason I am mentioning this is because I''m working on
 a vhole=phole+vBAR=pBAR patch set anyway, and this would
 be a neat logical extension that would help me work around
 yet more problems on what appears to be a fairly common
 hardware implementation bug.

 Gordan

 On Wed, 11 Sep 2013 12:25:18 +0100, Gordan Bobic <gordan@bobich.net> 
 wrote:
> It looks like I''m definitely not the first person to
> hit this problem:
>
> 
>
http://www.gossamer-threads.com/lists/xen/users/168557?do=post_view_threaded#168557
>
> No responses or workarounds suggested back then. :(
>
> Gordan
>
> On Wed, 11 Sep 2013 12:05:35 +0100, Gordan Bobic <gordan@bobich.net> 
> wrote:
>> I found this:
>>
>> http://lists.xen.org/archives/html/xen-devel/2010-06/msg00093.html
>>
>> while looking for a solution to a similar problem. I am
>> facing a similar issue with LSI (8408E, 3081E-R) and
>> Adaptec (31605) SAS cards. Was there ever a proper, more general
>> fix or workaround for this issue?
>>
>> These SAS cards experience these problems in dom0. When running
>> a vanilla kernel on bare metal, they work OK without intel_iommu
>> set. As soon as I set intel_iommu, the same thing happens (on
>> bare metal, not dom0).
>>
>> Clearly there is something badly broken with multiple layers
>> of bridges when it comes to IOMMU in my setup (Intel 5520 PCIe
>> root hub -> NF200 bridge -> Intel 80333 Bridge -> SAS
controller)
>>
>> I tried iommu=dom0-passthrough and it doesn''t appear to have
>> helped.
>>
>> I am not seeing similar problems with other PCIe devices that
>> are also, in theory doing DMA (e.g. GPUs), but LSI and Adapted
>> controllers appear to be affected for some reason.
>>
>> Is there anything else I could try/do to make this work?
>>
>> Gordan
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xen.org
>> http://lists.xen.org/xen-devel
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

>>> On 11.09.13 at 13:05, Gordan Bobic <gordan@bobich.net> wrote:
> I found this:
> 
>  http://lists.xen.org/archives/html/xen-devel/2010-06/msg00093.html 
> 
>  while looking for a solution to a similar problem. I am
>  facing a similar issue with LSI (8408E, 3081E-R) and
>  Adaptec (31605) SAS cards. Was there ever a proper, more general
>  fix or workaround for this issue?
> 
>  These SAS cards experience these problems in dom0. When running
>  a vanilla kernel on bare metal, they work OK without intel_iommu
>  set. As soon as I set intel_iommu, the same thing happens (on
>  bare metal, not dom0).
> 
>  Clearly there is something badly broken with multiple layers
>  of bridges when it comes to IOMMU in my setup (Intel 5520 PCIe
>  root hub -> NF200 bridge -> Intel 80333 Bridge -> SAS controller)
The link above has some (hackish) workarounds - did you try
them?

The link above, however, doesn''t indicate any relationship to
multiple bridges being in between, so it may not match what
you''re observing.

In any event, seeing a hypervisor log with "iommu=debug" might
shed further light on this: For one, we might be able to see which
exact devices are present in the ACPI tables. And we would see
which device(s) eventual faults originate from.

Jan

>>> On 11.09.13 at 13:44, Gordan Bobic <gordan@bobich.net> wrote:
> This got me thinking - if the problem is broken IOMMU implementation,
>  is the IOMMU _actually_ required for PCI passthrough to HVM
>  guests if all the memory holes and BARs are made exactly the same
>  in dom0 and domU? If vBAR=pBAR, then surely there is no memory
>  range remapping to be done anyway - which means that there
>  is no need for the strict IOMMU requirements (over and above
>  the requirements and caveats of PV domUs).
But with this you ignore the need to handle device bus mastering
activities. In order to work without IOMMU, the guest''s memory
addresses would also require guest-physical = machine-physical.

Jan

On Wed, 11 Sep 2013 12:53:09 +0100, "Jan Beulich"
<JBeulich@suse.com>
 wrote:
>>>> On 11.09.13 at 13:05, Gordan Bobic <gordan@bobich.net>
wrote:
>> I found this:
>>
>>  http://lists.xen.org/archives/html/xen-devel/2010-06/msg00093.html
>>
>>  while looking for a solution to a similar problem. I am
>>  facing a similar issue with LSI (8408E, 3081E-R) and
>>  Adaptec (31605) SAS cards. Was there ever a proper, more general
>>  fix or workaround for this issue?
>>
>>  These SAS cards experience these problems in dom0. When running
>>  a vanilla kernel on bare metal, they work OK without intel_iommu
>>  set. As soon as I set intel_iommu, the same thing happens (on
>>  bare metal, not dom0).
>>
>>  Clearly there is something badly broken with multiple layers
>>  of bridges when it comes to IOMMU in my setup (Intel 5520 PCIe
>>  root hub -> NF200 bridge -> Intel 80333 Bridge -> SAS
controller)
>
> The link above has some (hackish) workarounds - did you try
> them?
 Not yet. The thing that bothers me is that he workaround
 involves hard-coding the PCI device ID which is _nasty_
 and unstable.
> The link above, however, doesn''t indicate any relationship to
> multiple bridges being in between, so it may not match what
> you''re observing.
 The impression I got was that the "invisible" devices the
 previous thread was referring to were bridges on the SAS
 card. But I may have misunderstood.
> In any event, seeing a hypervisor log with "iommu=debug" might
> shed further light on this: For one, we might be able to see which
> exact devices are present in the ACPI tables. And we would see
> which device(s) eventual faults originate from.
 The thing that bothers me is that this happens in dom0 even
 with iommu=dom0-passthrough being set.
 iommu=dom0-passthrough,workaround_bios_bug doesn''t help,
 either

 And lo and behold, I do have phantom PCI devices after all!
 lspci shows no device with ID 0000:0f:01.0


 (XEN) [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault
 (XEN) [VT-D]iommu.c:865: DMAR:[DMA Write] Request device [0000:0f:01.0] 
 fault addr 857f15000, iommu reg = ffff82c3ffd54000
 (XEN) DMAR:[fault reason 02h] Present bit in context entry is clear
 (XEN) print_vtd_entries: iommu ffff83043fff5600 dev 0000:0f:01.0 gmfn 
 857f15
 (XEN)     root_entry = ffff83043ffe5000
 (XEN)     root_entry[f] = e6f7001
 (XEN)     context = ffff83000e6f7000
 (XEN)     context[8] = 0_0
 (XEN)     ctxt_entry[8] not present
 (XEN) [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault
 (XEN) [VT-D]iommu.c:865: DMAR:[DMA Write] Request device [0000:0f:01.0] 
 fault addr 858a35000, iommu reg = ffff82c3ffd54000
 (XEN) DMAR:[fault reason 02h] Present bit in context entry is clear
 (XEN) print_vtd_entries: iommu ffff83043fff5600 dev 0000:0f:01.0 gmfn 
 858a35
 (XEN)     root_entry = ffff83043ffe5000
 (XEN)     root_entry[f] = e6f7001
 (XEN)     context = ffff83000e6f7000
 (XEN)     context[8] = 0_0
 (XEN)     ctxt_entry[8] not present
 (XEN) [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault
 (XEN) [VT-D]iommu.c:865: DMAR:[DMA Write] Request device [0000:0f:01.0] 
 fault addr 471df8000, iommu reg = ffff82c3ffd54000
 (XEN) DMAR:[fault reason 02h] Present bit in context entry is clear
 (XEN) print_vtd_entries: iommu ffff83043fff5600 dev 0000:0f:01.0 gmfn 
 471df8
 (XEN)     root_entry = ffff83043ffe5000
 (XEN)     root_entry[f] = e6f7001
 (XEN)     context = ffff83000e6f7000
 (XEN)     context[8] = 0_0
 (XEN)     ctxt_entry[8] not present
 (XEN) [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault
 (XEN) [VT-D]iommu.c:865: DMAR:[DMA Write] Request device [0000:0f:01.0] 
 fault addr 46fc22000, iommu reg = ffff82c3ffd54000
 (XEN) DMAR:[fault reason 02h] Present bit in context entry is clear
 (XEN) print_vtd_entries: iommu ffff83043fff5600 dev 0000:0f:01.0 gmfn 
 46fc22
 (XEN)     root_entry = ffff83043ffe5000
 (XEN)     root_entry[f] = e6f7001
 (XEN)     context = ffff83000e6f7000
 (XEN)     context[8] = 0_0
 (XEN)     ctxt_entry[8] not present

 Gordan

On Wed, 11 Sep 2013 12:57:10 +0100, "Jan Beulich"
<JBeulich@suse.com>
 wrote:
>>>> On 11.09.13 at 13:44, Gordan Bobic <gordan@bobich.net>
wrote:
>> This got me thinking - if the problem is broken IOMMU 
>> implementation,
>>  is the IOMMU _actually_ required for PCI passthrough to HVM
>>  guests if all the memory holes and BARs are made exactly the same
>>  in dom0 and domU? If vBAR=pBAR, then surely there is no memory
>>  range remapping to be done anyway - which means that there
>>  is no need for the strict IOMMU requirements (over and above
>>  the requirements and caveats of PV domUs).
>
> But with this you ignore the need to handle device bus mastering
> activities. In order to work without IOMMU, the guest''s memory
> addresses would also require guest-physical = machine-physical.
 Hmm... that would be harder to achieve, mainly due to legacy
 stuff like base memory. But if (fingers crossed) DMA doesn''t
 occur below 1MB, maybe the map can be bodged to emulate the
 addresses below 1MB, carve out as small a chunk of 32-bit
 memory as we can get away with for each guest, and map the
 rest with vmem=pmem in 64-bit memory range. But you are
 right - that gets way more complicated than I originally
 envisaged - all for the sake of supporting legacy
 and Steam boot-loader systems. :(

 Gordan

>>> On 11.09.13 at 14:14, Gordan Bobic <gordan@bobich.net> wrote:
> On Wed, 11 Sep 2013 12:53:09 +0100, "Jan Beulich"
<JBeulich@suse.com>
>  wrote:
>>>>> On 11.09.13 at 13:05, Gordan Bobic
<gordan@bobich.net> wrote:
>>> I found this:
>>>
>>>  http://lists.xen.org/archives/html/xen-devel/2010-06/msg00093.html
>>>
>>>  while looking for a solution to a similar problem. I am
>>>  facing a similar issue with LSI (8408E, 3081E-R) and
>>>  Adaptec (31605) SAS cards. Was there ever a proper, more general
>>>  fix or workaround for this issue?
>>>
>>>  These SAS cards experience these problems in dom0. When running
>>>  a vanilla kernel on bare metal, they work OK without intel_iommu
>>>  set. As soon as I set intel_iommu, the same thing happens (on
>>>  bare metal, not dom0).
>>>
>>>  Clearly there is something badly broken with multiple layers
>>>  of bridges when it comes to IOMMU in my setup (Intel 5520 PCIe
>>>  root hub -> NF200 bridge -> Intel 80333 Bridge -> SAS
controller)
>>
>> The link above has some (hackish) workarounds - did you try
>> them?
> 
>  Not yet. The thing that bothers me is that he workaround
>  involves hard-coding the PCI device ID which is _nasty_
>  and unstable.
I said "hackish", didn''t I? Of course such a change would not
have the slightest chance of going into any repo. But knowing
whether it helps may allow thinking of an acceptable
workaround.
>> In any event, seeing a hypervisor log with "iommu=debug"
might
>> shed further light on this: For one, we might be able to see which
>> exact devices are present in the ACPI tables. And we would see
>> which device(s) eventual faults originate from.
> 
>  The thing that bothers me is that this happens in dom0 even
>  with iommu=dom0-passthrough being set.
>  iommu=dom0-passthrough,workaround_bios_bug doesn''t help,
>  either
They''re not meant to deal with this sort of an impossible (in
theory) situation.
>  And lo and behold, I do have phantom PCI devices after all!
>  lspci shows no device with ID 0000:0f:01.0
Not exactly: Phantom functions can''t be at function 0. Irrespective
of that - do the device coordinates somehow correlate with the
problematic controller (IOW: lspci output and a full log would help)?

Jan

On Wed, 11 Sep 2013 13:31:06 +0100, "Jan Beulich"
<JBeulich@suse.com>
 wrote:
>>>> On 11.09.13 at 14:14, Gordan Bobic <gordan@bobich.net>
wrote:
>> On Wed, 11 Sep 2013 12:53:09 +0100, "Jan Beulich" 
>> <JBeulich@suse.com>
>>  wrote:
>>>>>> On 11.09.13 at 13:05, Gordan Bobic
<gordan@bobich.net> wrote:
>>>> I found this:
>>>>
>>>>  
>>>>
http://lists.xen.org/archives/html/xen-devel/2010-06/msg00093.html
>>>>
>>>>  while looking for a solution to a similar problem. I am
>>>>  facing a similar issue with LSI (8408E, 3081E-R) and
>>>>  Adaptec (31605) SAS cards. Was there ever a proper, more
general
>>>>  fix or workaround for this issue?
>>>>
>>>>  These SAS cards experience these problems in dom0. When
running
>>>>  a vanilla kernel on bare metal, they work OK without
intel_iommu
>>>>  set. As soon as I set intel_iommu, the same thing happens (on
>>>>  bare metal, not dom0).
>>>>
>>>>  Clearly there is something badly broken with multiple layers
>>>>  of bridges when it comes to IOMMU in my setup (Intel 5520 PCIe
>>>>  root hub -> NF200 bridge -> Intel 80333 Bridge -> SAS
controller)
>>>
>>> The link above has some (hackish) workarounds - did you try
>>> them?
>>
>>  Not yet. The thing that bothers me is that he workaround
>>  involves hard-coding the PCI device ID which is _nasty_
>>  and unstable.
>
> I said "hackish", didn''t I? Of course such a change
would not
> have the slightest chance of going into any repo. But knowing
> whether it helps may allow thinking of an acceptable
> workaround.
 Indeed.
>>> In any event, seeing a hypervisor log with "iommu=debug"
might
>>> shed further light on this: For one, we might be able to see which
>>> exact devices are present in the ACPI tables. And we would see
>>> which device(s) eventual faults originate from.
>>
>>  The thing that bothers me is that this happens in dom0 even
>>  with iommu=dom0-passthrough being set.
>>  iommu=dom0-passthrough,workaround_bios_bug doesn''t help,
>>  either
>
> They''re not meant to deal with this sort of an impossible (in
> theory) situation.
 It turns out that theoretically impossible things happen a lot
 more often than expected. :)
>>  And lo and behold, I do have phantom PCI devices after all!
>>  lspci shows no device with ID 0000:0f:01.0
>
> Not exactly: Phantom functions can''t be at function 0.
Irrespective
> of that - do the device coordinates somehow correlate with the
> problematic controller (IOW: lspci output and a full log would help)?
 Not necessarily a function of the same device - as I said, there is
 a PCIe bridge on the card, so there could be multiple devices hiding
 there.

 dmesg, xl dmesg, lspci -vvvnn and lspci -tvnn output is attached.

 I''ll try adding one of my LSI cards and see the comparative
 behaviour. Right now I don''t even know if the phantom device
 is on the SAS card or the motherboard.

 Gordan




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On Wed, Sep 11, 2013 at 01:19:44PM +0100, Gordan Bobic
wrote:
> On Wed, 11 Sep 2013 12:57:10 +0100, "Jan Beulich"
> <JBeulich@suse.com> wrote:
> >>>>On 11.09.13 at 13:44, Gordan Bobic
<gordan@bobich.net> wrote:
> >>This got me thinking - if the problem is broken IOMMU
> >>implementation,
> >> is the IOMMU _actually_ required for PCI passthrough to HVM
> >> guests if all the memory holes and BARs are made exactly the same
> >> in dom0 and domU? If vBAR=pBAR, then surely there is no memory
> >> range remapping to be done anyway - which means that there
> >> is no need for the strict IOMMU requirements (over and above
> >> the requirements and caveats of PV domUs).
> >
> >But with this you ignore the need to handle device bus mastering
> >activities. In order to work without IOMMU, the guest''s memory
> >addresses would also require guest-physical = machine-physical.
> 
> Hmm... that would be harder to achieve, mainly due to legacy
> stuff like base memory. But if (fingers crossed) DMA doesn''t
> occur below 1MB, maybe the map can be bodged to emulate the
> addresses below 1MB, carve out as small a chunk of 32-bit
> memory as we can get away with for each guest, and map the
> rest with vmem=pmem in 64-bit memory range. But you are
> right - that gets way more complicated than I originally
> envisaged - all for the sake of supporting legacy
> and Steam boot-loader systems. :(
> 
I don''t know the details but many years ago there was a patch
to allow PCI passthru to HVM guests without an IOMMU..

I think it''s linked on Xen pcipassthrough wiki page.
It had a limitation that only one HVM guest was supported for PCI passthru.

-- Pasi

>>> On 11.09.13 at 14:45, Gordan Bobic <gordan@bobich.net> wrote:
>  dmesg, xl dmesg, lspci -vvvnn and lspci -tvnn output is attached.
> 
>  I''ll try adding one of my LSI cards and see the comparative
>  behaviour. Right now I don''t even know if the phantom device
>  is on the SAS card or the motherboard.
The Adaptec card being the only thing on bus 0f makes it pretty
likely that this other device also is on that card.

I guess the issue is mainly because the device itself is a PCI one,
while the immediately upstream bridge (where I mean only the
visible one) is PCIe. There _must_ be a PCIe-PCI bridge between
them. And as long as firmware doesn''t know about that bridge
and the bridge doesn''t properly handle config space accesses to
it, such a device just can''t be used with an IOMMU (without some
yet to be invented workaround).

Jan

On Wed, 11 Sep 2013 14:03:14 +0100, "Jan Beulich"
<JBeulich@suse.com>
 wrote:
>>>> On 11.09.13 at 14:45, Gordan Bobic <gordan@bobich.net>
wrote:
>>  dmesg, xl dmesg, lspci -vvvnn and lspci -tvnn output is attached.
>>
>>  I''ll try adding one of my LSI cards and see the comparative
>>  behaviour. Right now I don''t even know if the phantom device
>>  is on the SAS card or the motherboard.
>
> The Adaptec card being the only thing on bus 0f makes it pretty
> likely that this other device also is on that card.
>
> I guess the issue is mainly because the device itself is a PCI one,
> while the immediately upstream bridge (where I mean only the
> visible one) is PCIe. There _must_ be a PCIe-PCI bridge between
> them. And as long as firmware doesn''t know about that bridge
> and the bridge doesn''t properly handle config space accesses to
> it, such a device just can''t be used with an IOMMU (without some
> yet to be invented workaround).
 I''m actually thinking about Konrad''s proposed hack in that
 thread from 3 years ago. If the device IDs are parameterized
 out rather than hard-coded, then this could work in nearly the
 same was as xen-pciback in terms of usage. Pass the phantom
 device IDs as parameters to the module. Done that way it
 might even be considered clean enough to be fit for public
 consumption.

 Gordan

>>> On 11.09.13 at 15:10, Gordan Bobic <gordan@bobich.net> wrote:
> On Wed, 11 Sep 2013 14:03:14 +0100, "Jan Beulich"
<JBeulich@suse.com>
>  wrote:
>>>>> On 11.09.13 at 14:45, Gordan Bobic
<gordan@bobich.net> wrote:
>>>  dmesg, xl dmesg, lspci -vvvnn and lspci -tvnn output is attached.
>>>
>>>  I''ll try adding one of my LSI cards and see the
comparative
>>>  behaviour. Right now I don''t even know if the phantom
device
>>>  is on the SAS card or the motherboard.
>>
>> The Adaptec card being the only thing on bus 0f makes it pretty
>> likely that this other device also is on that card.
>>
>> I guess the issue is mainly because the device itself is a PCI one,
>> while the immediately upstream bridge (where I mean only the
>> visible one) is PCIe. There _must_ be a PCIe-PCI bridge between
>> them. And as long as firmware doesn''t know about that bridge
>> and the bridge doesn''t properly handle config space accesses
to
>> it, such a device just can''t be used with an IOMMU (without
some
>> yet to be invented workaround).
> 
>  I''m actually thinking about Konrad''s proposed hack in
that
>  thread from 3 years ago. If the device IDs are parameterized
>  out rather than hard-coded, then this could work in nearly the
>  same was as xen-pciback in terms of usage. Pass the phantom
>  device IDs as parameters to the module. Done that way it
>  might even be considered clean enough to be fit for public
>  consumption.
Except that, short of being able to determine it via config
space reads, we also need the resulting command line option
to tell us that what kind of device that is.

Jan

On Wed, 11 Sep 2013 14:03:14 +0100, "Jan Beulich"
<JBeulich@suse.com>
 wrote:
>>>> On 11.09.13 at 14:45, Gordan Bobic <gordan@bobich.net>
wrote:
>>  dmesg, xl dmesg, lspci -vvvnn and lspci -tvnn output is attached.
>>
>>  I''ll try adding one of my LSI cards and see the comparative
>>  behaviour. Right now I don''t even know if the phantom device
>>  is on the SAS card or the motherboard.
>
> The Adaptec card being the only thing on bus 0f makes it pretty
> likely that this other device also is on that card.
 That''s plausible, but given there are Nvidia NF200 bridges between
 the Intel 5520 PCIe hubs and each of the slots on my motherboard,
 it is not a 100% certainty.
> I guess the issue is mainly because the device itself is a PCI one,
> while the immediately upstream bridge (where I mean only the
> visible one) is PCIe. There _must_ be a PCIe-PCI bridge between
> them. And as long as firmware doesn''t know about that bridge
> and the bridge doesn''t properly handle config space accesses to
> it, such a device just can''t be used with an IOMMU (without some
> yet to be invented workaround).
 You don''t think Konrad''s patch on the old thread would work
 for getting it working in dom0?

 Gordan

On Wed, 11 Sep 2013 14:22:51 +0100, "Jan Beulich"
<JBeulich@suse.com>
 wrote:
>>>> On 11.09.13 at 15:10, Gordan Bobic <gordan@bobich.net>
wrote:
>> On Wed, 11 Sep 2013 14:03:14 +0100, "Jan Beulich" 
>> <JBeulich@suse.com>
>>  wrote:
>>>>>> On 11.09.13 at 14:45, Gordan Bobic
<gordan@bobich.net> wrote:
>>>>  dmesg, xl dmesg, lspci -vvvnn and lspci -tvnn output is
attached.
>>>>
>>>>  I''ll try adding one of my LSI cards and see the
comparative
>>>>  behaviour. Right now I don''t even know if the phantom
device
>>>>  is on the SAS card or the motherboard.
>>>
>>> The Adaptec card being the only thing on bus 0f makes it pretty
>>> likely that this other device also is on that card.
>>>
>>> I guess the issue is mainly because the device itself is a PCI one,
>>> while the immediately upstream bridge (where I mean only the
>>> visible one) is PCIe. There _must_ be a PCIe-PCI bridge between
>>> them. And as long as firmware doesn''t know about that
bridge
>>> and the bridge doesn''t properly handle config space
accesses to
>>> it, such a device just can''t be used with an IOMMU
(without some
>>> yet to be invented workaround).
>>
>>  I''m actually thinking about Konrad''s proposed hack
in that
>>  thread from 3 years ago. If the device IDs are parameterized
>>  out rather than hard-coded, then this could work in nearly the
>>  same was as xen-pciback in terms of usage. Pass the phantom
>>  device IDs as parameters to the module. Done that way it
>>  might even be considered clean enough to be fit for public
>>  consumption.
>
> Except that, short of being able to determine it via config
> space reads, we also need the resulting command line option
> to tell us that what kind of device that is.
 Not sure I follow. Why do we need to know the device type?

 Gordan

>>> On 11.09.13 at 15:23, Gordan Bobic <gordan@bobich.net> wrote:
> On Wed, 11 Sep 2013 14:03:14 +0100, "Jan Beulich"
<JBeulich@suse.com> wrote:
>> I guess the issue is mainly because the device itself is a PCI one,
>> while the immediately upstream bridge (where I mean only the
>> visible one) is PCIe. There _must_ be a PCIe-PCI bridge between
>> them. And as long as firmware doesn''t know about that bridge
>> and the bridge doesn''t properly handle config space accesses
to
>> it, such a device just can''t be used with an IOMMU (without
some
>> yet to be invented workaround).
> 
>  You don''t think Konrad''s patch on the old thread would
work
>  for getting it working in dom0?
For just Dom0 this might suffice, but for passthrough it surely
won''t. And a sufficiently generic workaround patch should deal
with both imo.

Jan

>>> On 11.09.13 at 15:26, Gordan Bobic <gordan@bobich.net> wrote:
> On Wed, 11 Sep 2013 14:22:51 +0100, "Jan Beulich"
<JBeulich@suse.com>
>  wrote:
>>>>> On 11.09.13 at 15:10, Gordan Bobic
<gordan@bobich.net> wrote:
>>> On Wed, 11 Sep 2013 14:03:14 +0100, "Jan Beulich" 
>>> <JBeulich@suse.com>
>>>  wrote:
>>>>>>> On 11.09.13 at 14:45, Gordan Bobic
<gordan@bobich.net> wrote:
>>>>>  dmesg, xl dmesg, lspci -vvvnn and lspci -tvnn output is
attached.
>>>>>
>>>>>  I''ll try adding one of my LSI cards and see the
comparative
>>>>>  behaviour. Right now I don''t even know if the
phantom device
>>>>>  is on the SAS card or the motherboard.
>>>>
>>>> The Adaptec card being the only thing on bus 0f makes it pretty
>>>> likely that this other device also is on that card.
>>>>
>>>> I guess the issue is mainly because the device itself is a PCI
one,
>>>> while the immediately upstream bridge (where I mean only the
>>>> visible one) is PCIe. There _must_ be a PCIe-PCI bridge between
>>>> them. And as long as firmware doesn''t know about that
bridge
>>>> and the bridge doesn''t properly handle config space
accesses to
>>>> it, such a device just can''t be used with an IOMMU
(without some
>>>> yet to be invented workaround).
>>>
>>>  I''m actually thinking about Konrad''s proposed
hack in that
>>>  thread from 3 years ago. If the device IDs are parameterized
>>>  out rather than hard-coded, then this could work in nearly the
>>>  same was as xen-pciback in terms of usage. Pass the phantom
>>>  device IDs as parameters to the module. Done that way it
>>>  might even be considered clean enough to be fit for public
>>>  consumption.
>>
>> Except that, short of being able to determine it via config
>> space reads, we also need the resulting command line option
>> to tell us that what kind of device that is.
> 
>  Not sure I follow. Why do we need to know the device type?
Just look at set_msi_source_id() as well as
domain_context_{mapping,unmap}() (just the most prominent
examples): Behavior here heavily depends on the type of the
device itself _and_ that of the upstream bridge(s).

Jan

Jan Beulich wrote on 2013-09-11:
>>>> On 11.09.13 at 15:26, Gordan Bobic <gordan@bobich.net>
wrote:
>> On Wed, 11 Sep 2013 14:22:51 +0100, "Jan Beulich"
>> <JBeulich@suse.com>
>>  wrote:
>>>>>> On 11.09.13 at 15:10, Gordan Bobic
<gordan@bobich.net> wrote:
>>>> On Wed, 11 Sep 2013 14:03:14 +0100, "Jan Beulich"
>>>> <JBeulich@suse.com>
>>>>  wrote:
>>>>>>>> On 11.09.13 at 14:45, Gordan Bobic
<gordan@bobich.net> wrote:
>>>>>>  dmesg, xl dmesg, lspci -vvvnn and lspci -tvnn output
is attached.
>>>>>>  
>>>>>>  I''ll try adding one of my LSI cards and see
the comparative
>>>>>> behaviour. Right now I don''t even know if the
phantom device  is
>>>>>> on the SAS card or the motherboard.
>>>>> 
>>>>> The Adaptec card being the only thing on bus 0f makes it
pretty
>>>>> likely that this other device also is on that card.
>>>>> 
>>>>> I guess the issue is mainly because the device itself is a
PCI
>>>>> one, while the immediately upstream bridge (where I mean
only the
>>>>> visible one) is PCIe. There _must_ be a PCIe-PCI bridge
between
>>>>> them. And as long as firmware doesn''t know about
that bridge and
>>>>> the bridge doesn''t properly handle config space
accesses to it,
>>>>> such a device just can''t be used with an IOMMU
(without some yet
>>>>> to be invented workaround).
>>>>> 
>>>>  I''m actually thinking about Konrad''s
proposed hack in that
>>>> thread from 3 years ago. If the device IDs are parameterized 
out
>>>> rather than hard-coded, then this could work in nearly the 
same
>>>> was as xen-pciback in terms of usage. Pass the phantom  device
IDs
>>>> as parameters to the module. Done that way it  might even be
>>>> considered clean enough to be fit for public  consumption.
>>> 
>>> Except that, short of being able to determine it via config space
>>> reads, we also need the resulting command line option to tell us
>>> that what kind of device that is.
>>> 
>>  Not sure I follow. Why do we need to know the device type?
> 
> Just look at set_msi_source_id() as well as
> domain_context_{mapping,unmap}() (just the most prominent
> examples): Behavior here heavily depends on the type of the device
> itself _and_ that of the upstream bridge(s).
Looks like there are many devices are failed to work. I wonder whether the
PCI/PCIe specification tells how to detect the hidden device behind those
devices (Like detection of phantom device). If not, I think those devices are
buggy. Or we can say those devices are not really PCI/PCIe compatible. Since
VT-d only covers the PCI/PCIe device, it''s reasonable that non-PCI/PCIe
device failed to work under VT-d.

As Jan''s suggestion, we need the user to tell us whether there is a
hidden device or BDF behind anther device that the OS is unaware. We need to
pass that info to Xen before pass-thought the device.
> 
> Jan
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Best regards,
Yang

On Thu, Sep 12, 2013 at 06:20:18AM +0000, Zhang, Yang Z
wrote:
> Jan Beulich wrote on 2013-09-11:
> >>>> On 11.09.13 at 15:26, Gordan Bobic
<gordan@bobich.net> wrote:
> >> On Wed, 11 Sep 2013 14:22:51 +0100, "Jan Beulich"
> >> <JBeulich@suse.com>
> >>  wrote:
> >>>>>> On 11.09.13 at 15:10, Gordan Bobic
<gordan@bobich.net> wrote:
> >>>> On Wed, 11 Sep 2013 14:03:14 +0100, "Jan
Beulich"
> >>>> <JBeulich@suse.com>
> >>>>  wrote:
> >>>>>>>> On 11.09.13 at 14:45, Gordan Bobic
<gordan@bobich.net> wrote:
> >>>>>>  dmesg, xl dmesg, lspci -vvvnn and lspci -tvnn
output is attached.
> >>>>>>  
> >>>>>>  I''ll try adding one of my LSI cards and
see the comparative
> >>>>>> behaviour. Right now I don''t even know if
the phantom device  is
> >>>>>> on the SAS card or the motherboard.
> >>>>> 
> >>>>> The Adaptec card being the only thing on bus 0f makes
it pretty
> >>>>> likely that this other device also is on that card.
> >>>>> 
> >>>>> I guess the issue is mainly because the device itself
is a PCI
> >>>>> one, while the immediately upstream bridge (where I
mean only the
> >>>>> visible one) is PCIe. There _must_ be a PCIe-PCI
bridge between
> >>>>> them. And as long as firmware doesn''t know
about that bridge and
> >>>>> the bridge doesn''t properly handle config
space accesses to it,
> >>>>> such a device just can''t be used with an
IOMMU (without some yet
> >>>>> to be invented workaround).
> >>>>> 
> >>>>  I''m actually thinking about Konrad''s
proposed hack in that
> >>>> thread from 3 years ago. If the device IDs are
parameterized  out
> >>>> rather than hard-coded, then this could work in nearly the
same
> >>>> was as xen-pciback in terms of usage. Pass the phantom 
device IDs
> >>>> as parameters to the module. Done that way it  might even
be
> >>>> considered clean enough to be fit for public  consumption.
> >>> 
> >>> Except that, short of being able to determine it via config
space
> >>> reads, we also need the resulting command line option to tell
us
> >>> that what kind of device that is.
> >>> 
> >>  Not sure I follow. Why do we need to know the device type?
> > 
> > Just look at set_msi_source_id() as well as
> > domain_context_{mapping,unmap}() (just the most prominent
> > examples): Behavior here heavily depends on the type of the device
> > itself _and_ that of the upstream bridge(s).
> Looks like there are many devices are failed to work. I wonder whether the
PCI/PCIe specification tells how to detect the hidden device behind those
devices (Like detection of phantom device). If not, I think those devices are
buggy. Or we can say those devices are not really PCI/PCIe compatible. Since
VT-d only covers the PCI/PCIe device, it''s reasonable that non-PCI/PCIe
device failed to work under VT-d.
> 
> As Jan''s suggestion, we need the user to tell us whether there is
a hidden device or BDF behind anther device that the OS is unaware. We need to
pass that info to Xen before pass-thought the device.
> 
Interestingly enough I just hit this with my brand-new Haswell CPU and
new motherboard when passing in a capture card. It shows:

    +-1c.5-[07-09]----00.0-[08-09]--+-01.0-[09]--+-08.0  Brooktree Corporation
Bt878 Video Capture
           |                               |            +-08.1  Brooktree
Corporation Bt878 Audio Capture
           |                               |            +-09.0  Brooktree
Corporation Bt878 Video Capture
           |                               |            +-09.1  Brooktree
Corporation Bt878 Audio Capture
           |                               |            +-0a.0  Brooktree
Corporation Bt878 Video Capture
           |                               |            +-0a.1  Brooktree
Corporation Bt878 Audio Capture
           |                               |            +-0b.0  Brooktree
Corporation Bt878 Video Capture
           |                               |            \-0b.1  Brooktree
Corporation Bt878 Audio Capture
           |                               \-03.0  Texas Instruments TSB43AB22A
IEEE-1394a-2000 Controller (PHY/Link) [iOHCI-Lynx]

And Xen says:
(XEN) [VT-D]iommu.c:885: iommu_fault_status: Fault Overflow
(XEN) [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault
(XEN) [VT-D]iommu.c:865: DMAR:[DMA Read] Request device [0000:08:00.0] fault
addr 36aa3000, iommu reg = ffff82c3ffd53000
(XEN) DMAR:[fault reason 02h] Present bit in context entry is clear
(XEN) print_vtd_entries: iommu ffff83083d4939b0 dev 0000:08:00.0 gmfn 36aa3
(XEN)     root_entry = ffff83083d47e000
(XEN)     root_entry[8] = 72569a001
(XEN)     context = ffff83072569a000
(XEN)     context[0] = 0_0
(XEN)     ctxt_entry[0] not present
(XEN) [VT-D]iommu.c:885: iommu_fault_status: Fault Overflow
(XEN) [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault
(XEN) [VT-D]iommu.c:865: DMAR:[DMA Read] Request device [0000:08:00.0] fault
addr 36aa3000, iommu reg = ffff82c3ffd53000


Oddly enough it was working fine in a box with an AMD IOMMU. But
to be fair - that machine was running with Xen 4.1.

The hack I developed:
http://lists.xen.org/archives/html/xen-devel/2010-06/msg00093.html
ends up with this:

(XEN) alloc_pdev: unknown type: 0000:08:00.0
(XEN) [VT-D]iommu.c:1484: d0:unknown(0): 0000:08:00.0
(XEN) [VT-D]iommu.c:1888: d0: context mapping failed

(FYI, this Xen 4.3.1)

Let me retry on the AMD box with the same version of Xen.

On 12/11/2013 06:32 PM, Konrad Rzeszutek Wilk wrote:
> On Thu, Sep 12, 2013 at 06:20:18AM +0000, Zhang, Yang Z wrote:
>> Jan Beulich wrote on 2013-09-11:
>>>>>> On 11.09.13 at 15:26, Gordan Bobic
<gordan@bobich.net> wrote:
>>>> On Wed, 11 Sep 2013 14:22:51 +0100, "Jan Beulich"
>>>> <JBeulich@suse.com>
>>>>   wrote:
>>>>>>>> On 11.09.13 at 15:10, Gordan Bobic
<gordan@bobich.net> wrote:
>>>>>> On Wed, 11 Sep 2013 14:03:14 +0100, "Jan
Beulich"
>>>>>> <JBeulich@suse.com>
>>>>>>   wrote:
>>>>>>>>>> On 11.09.13 at 14:45, Gordan Bobic
<gordan@bobich.net> wrote:
>>>>>>>>   dmesg, xl dmesg, lspci -vvvnn and lspci -tvnn
output is attached.
>>>>>>>>
>>>>>>>>   I''ll try adding one of my LSI cards
and see the comparative
>>>>>>>> behaviour. Right now I don''t even know
if the phantom device  is
>>>>>>>> on the SAS card or the motherboard.
>>>>>>>
>>>>>>> The Adaptec card being the only thing on bus 0f
makes it pretty
>>>>>>> likely that this other device also is on that card.
>>>>>>>
>>>>>>> I guess the issue is mainly because the device
itself is a PCI
>>>>>>> one, while the immediately upstream bridge (where I
mean only the
>>>>>>> visible one) is PCIe. There _must_ be a PCIe-PCI
bridge between
>>>>>>> them. And as long as firmware doesn''t know
about that bridge and
>>>>>>> the bridge doesn''t properly handle config
space accesses to it,
>>>>>>> such a device just can''t be used with an
IOMMU (without some yet
>>>>>>> to be invented workaround).
>>>>>>>
>>>>>>   I''m actually thinking about
Konrad''s proposed hack in that
>>>>>> thread from 3 years ago. If the device IDs are
parameterized  out
>>>>>> rather than hard-coded, then this could work in nearly
the  same
>>>>>> was as xen-pciback in terms of usage. Pass the phantom 
device IDs
>>>>>> as parameters to the module. Done that way it  might
even be
>>>>>> considered clean enough to be fit for public 
consumption.
>>>>>
>>>>> Except that, short of being able to determine it via config
space
>>>>> reads, we also need the resulting command line option to
tell us
>>>>> that what kind of device that is.
>>>>>
>>>>   Not sure I follow. Why do we need to know the device type?
>>>
>>> Just look at set_msi_source_id() as well as
>>> domain_context_{mapping,unmap}() (just the most prominent
>>> examples): Behavior here heavily depends on the type of the device
>>> itself _and_ that of the upstream bridge(s).
>> Looks like there are many devices are failed to work. I wonder whether
the PCI/PCIe specification tells how to detect the hidden device behind those
devices (Like detection of phantom device). If not, I think those devices are
buggy. Or we can say those devices are not really PCI/PCIe compatible. Since
VT-d only covers the PCI/PCIe device, it''s reasonable that non-PCI/PCIe
device failed to work under VT-d.
>>
>> As Jan''s suggestion, we need the user to tell us whether there
is a hidden device or BDF behind anther device that the OS is unaware. We need
to pass that info to Xen before pass-thought the device.
>>
>
> Interestingly enough I just hit this with my brand-new Haswell CPU and
> new motherboard when passing in a capture card. It shows:
>
>      +-1c.5-[07-09]----00.0-[08-09]--+-01.0-[09]--+-08.0  Brooktree
Corporation Bt878 Video Capture
>             |                               |            +-08.1  Brooktree
Corporation Bt878 Audio Capture
>             |                               |            +-09.0  Brooktree
Corporation Bt878 Video Capture
>             |                               |            +-09.1  Brooktree
Corporation Bt878 Audio Capture
>             |                               |            +-0a.0  Brooktree
Corporation Bt878 Video Capture
>             |                               |            +-0a.1  Brooktree
Corporation Bt878 Audio Capture
>             |                               |            +-0b.0  Brooktree
Corporation Bt878 Video Capture
>             |                               |            \-0b.1  Brooktree
Corporation Bt878 Audio Capture
>             |                               \-03.0  Texas Instruments
TSB43AB22A IEEE-1394a-2000 Controller (PHY/Link) [iOHCI-Lynx]
>
> And Xen says:
> (XEN) [VT-D]iommu.c:885: iommu_fault_status: Fault Overflow
> (XEN) [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault
> (XEN) [VT-D]iommu.c:865: DMAR:[DMA Read] Request device [0000:08:00.0]
fault addr 36aa3000, iommu reg = ffff82c3ffd53000
> (XEN) DMAR:[fault reason 02h] Present bit in context entry is clear
> (XEN) print_vtd_entries: iommu ffff83083d4939b0 dev 0000:08:00.0 gmfn 36aa3
> (XEN)     root_entry = ffff83083d47e000
> (XEN)     root_entry[8] = 72569a001
> (XEN)     context = ffff83072569a000
> (XEN)     context[0] = 0_0
> (XEN)     ctxt_entry[0] not present
> (XEN) [VT-D]iommu.c:885: iommu_fault_status: Fault Overflow
> (XEN) [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault
> (XEN) [VT-D]iommu.c:865: DMAR:[DMA Read] Request device [0000:08:00.0]
fault addr 36aa3000, iommu reg = ffff82c3ffd53000
>
>
> Oddly enough it was working fine in a box with an AMD IOMMU. But
> to be fair - that machine was running with Xen 4.1.
>
> The hack I developed:
http://lists.xen.org/archives/html/xen-devel/2010-06/msg00093.html
> ends up with this:
>
> (XEN) alloc_pdev: unknown type: 0000:08:00.0
> (XEN) [VT-D]iommu.c:1484: d0:unknown(0): 0000:08:00.0
> (XEN) [VT-D]iommu.c:1888: d0: context mapping failed
>
> (FYI, this Xen 4.3.1)
>
> Let me retry on the AMD box with the same version of Xen.
I may be wrong, but this doesn''t look like the same problem (phantom
PCI
device on the bus). Or am I missing something?

As far as I can tell, the original problem was arising on cards that are 
PCIe, but based on a PCIX chipset, i.e. with a PCIe-PCIX bridge. Xen 
wasn''t the only thing affected in my case - bare metal Linux kernel was
also having problems with intel-iommu=1 in the kernel boot parameters. 
If might be worth trying that with your card to see what happens. If 
bare metal Linux with intel-iommu=1 works for your card, it''s probably 
not the same problem (of course it could be similar/related).

Out of interest, I noticed recently there is a xen parameter 
"pci-phantom", but I haven''t been able to find documentation
for it. Can
you point me in the right direction? Does it, perchance, allow 
specifying the PCI slot ID of a phantom device so that IOMMU doesn''t 
freak out when a seemingly non-existant device starts trying to do DMA?

Gordan

On Wed, Dec 11, 2013 at 09:15:17PM +0000, Gordan Bobic
wrote:
> On 12/11/2013 06:32 PM, Konrad Rzeszutek Wilk wrote:
> >On Thu, Sep 12, 2013 at 06:20:18AM +0000, Zhang, Yang Z wrote:
> >>Jan Beulich wrote on 2013-09-11:
> >>>>>>On 11.09.13 at 15:26, Gordan Bobic
<gordan@bobich.net> wrote:
> >>>>On Wed, 11 Sep 2013 14:22:51 +0100, "Jan Beulich"
> >>>><JBeulich@suse.com>
> >>>>  wrote:
> >>>>>>>>On 11.09.13 at 15:10, Gordan Bobic
<gordan@bobich.net> wrote:
> >>>>>>On Wed, 11 Sep 2013 14:03:14 +0100, "Jan
Beulich"
> >>>>>><JBeulich@suse.com>
> >>>>>>  wrote:
> >>>>>>>>>>On 11.09.13 at 14:45, Gordan Bobic
<gordan@bobich.net> wrote:
> >>>>>>>>  dmesg, xl dmesg, lspci -vvvnn and lspci
-tvnn output is attached.
> >>>>>>>>
> >>>>>>>>  I''ll try adding one of my LSI
cards and see the comparative
> >>>>>>>>behaviour. Right now I don''t even
know if the phantom device  is
> >>>>>>>>on the SAS card or the motherboard.
> >>>>>>>
> >>>>>>>The Adaptec card being the only thing on bus 0f
makes it pretty
> >>>>>>>likely that this other device also is on that
card.
> >>>>>>>
> >>>>>>>I guess the issue is mainly because the device
itself is a PCI
> >>>>>>>one, while the immediately upstream bridge
(where I mean only the
> >>>>>>>visible one) is PCIe. There _must_ be a
PCIe-PCI bridge between
> >>>>>>>them. And as long as firmware doesn''t
know about that bridge and
> >>>>>>>the bridge doesn''t properly handle
config space accesses to it,
> >>>>>>>such a device just can''t be used with
an IOMMU (without some yet
> >>>>>>>to be invented workaround).
> >>>>>>>
> >>>>>>  I''m actually thinking about
Konrad''s proposed hack in that
> >>>>>>thread from 3 years ago. If the device IDs are
parameterized  out
> >>>>>>rather than hard-coded, then this could work in
nearly the  same
> >>>>>>was as xen-pciback in terms of usage. Pass the
phantom  device IDs
> >>>>>>as parameters to the module. Done that way it 
might even be
> >>>>>>considered clean enough to be fit for public 
consumption.
> >>>>>
> >>>>>Except that, short of being able to determine it via
config space
> >>>>>reads, we also need the resulting command line option
to tell us
> >>>>>that what kind of device that is.
> >>>>>
> >>>>  Not sure I follow. Why do we need to know the device
type?
> >>>
> >>>Just look at set_msi_source_id() as well as
> >>>domain_context_{mapping,unmap}() (just the most prominent
> >>>examples): Behavior here heavily depends on the type of the
device
> >>>itself _and_ that of the upstream bridge(s).
> >>Looks like there are many devices are failed to work. I wonder
whether the PCI/PCIe specification tells how to detect the hidden device behind
those devices (Like detection of phantom device). If not, I think those devices
are buggy. Or we can say those devices are not really PCI/PCIe compatible. Since
VT-d only covers the PCI/PCIe device, it''s reasonable that non-PCI/PCIe
device failed to work under VT-d.
> >>
> >>As Jan''s suggestion, we need the user to tell us whether
there is a hidden device or BDF behind anther device that the OS is unaware. We
need to pass that info to Xen before pass-thought the device.
> >>
> >
> >Interestingly enough I just hit this with my brand-new Haswell CPU and
> >new motherboard when passing in a capture card. It shows:
> >
> >     +-1c.5-[07-09]----00.0-[08-09]--+-01.0-[09]--+-08.0  Brooktree
Corporation Bt878 Video Capture
> >            |                               |            +-08.1 
Brooktree Corporation Bt878 Audio Capture
> >            |                               |            +-09.0 
Brooktree Corporation Bt878 Video Capture
> >            |                               |            +-09.1 
Brooktree Corporation Bt878 Audio Capture
> >            |                               |            +-0a.0 
Brooktree Corporation Bt878 Video Capture
> >            |                               |            +-0a.1 
Brooktree Corporation Bt878 Audio Capture
> >            |                               |            +-0b.0 
Brooktree Corporation Bt878 Video Capture
> >            |                               |            \-0b.1 
Brooktree Corporation Bt878 Audio Capture
> >            |                               \-03.0  Texas Instruments
TSB43AB22A IEEE-1394a-2000 Controller (PHY/Link) [iOHCI-Lynx]
> >
> >And Xen says:
> >(XEN) [VT-D]iommu.c:885: iommu_fault_status: Fault Overflow
> >(XEN) [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault
> >(XEN) [VT-D]iommu.c:865: DMAR:[DMA Read] Request device [0000:08:00.0]
fault addr 36aa3000, iommu reg = ffff82c3ffd53000
> >(XEN) DMAR:[fault reason 02h] Present bit in context entry is clear
> >(XEN) print_vtd_entries: iommu ffff83083d4939b0 dev 0000:08:00.0 gmfn
36aa3
> >(XEN)     root_entry = ffff83083d47e000
> >(XEN)     root_entry[8] = 72569a001
> >(XEN)     context = ffff83072569a000
> >(XEN)     context[0] = 0_0
> >(XEN)     ctxt_entry[0] not present
> >(XEN) [VT-D]iommu.c:885: iommu_fault_status: Fault Overflow
> >(XEN) [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault
> >(XEN) [VT-D]iommu.c:865: DMAR:[DMA Read] Request device [0000:08:00.0]
fault addr 36aa3000, iommu reg = ffff82c3ffd53000
> >
> >
> >Oddly enough it was working fine in a box with an AMD IOMMU. But
> >to be fair - that machine was running with Xen 4.1.
> >
> >The hack I developed:
http://lists.xen.org/archives/html/xen-devel/2010-06/msg00093.html
> >ends up with this:
> >
> >(XEN) alloc_pdev: unknown type: 0000:08:00.0
> >(XEN) [VT-D]iommu.c:1484: d0:unknown(0): 0000:08:00.0
> >(XEN) [VT-D]iommu.c:1888: d0: context mapping failed
> >
> >(FYI, this Xen 4.3.1)
> >
> >Let me retry on the AMD box with the same version of Xen.
> 
> I may be wrong, but this doesn''t look like the same problem
(phantom
> PCI device on the bus). Or am I missing something?
It is. A phantom device as well.
> 
> As far as I can tell, the original problem was arising on cards that
> are PCIe, but based on a PCIX chipset, i.e. with a PCIe-PCIX bridge.
> Xen wasn''t the only thing affected in my case - bare metal Linux
> kernel was also having problems with intel-iommu=1 in the kernel
> boot parameters. If might be worth trying that with your card to see
> what happens. If bare metal Linux with intel-iommu=1 works for your
> card, it''s probably not the same problem (of course it could be
> similar/related).
That is a similar problem here. Except that I have a PCI devices and
it goes over an PCIe bridge (I think).
> 
> Out of interest, I noticed recently there is a xen parameter
> "pci-phantom", but I haven''t been able to find
documentation for it.
> Can you point me in the right direction? Does it, perchance, allow
> specifying the PCI slot ID of a phantom device so that IOMMU
doesn''t
> freak out when a seemingly non-existant device starts trying to do
> DMA?
I forgot about it!

t 4e3c592c93d7dbe02ca36878457515d30fe931d2
Author: Jan Beulich <jbeulich@suse.com>
Date:   Mon Jan 7 12:58:09 2013 +0100

    IOMMU: add option to specify devices behaving like ones using phantom
functions
    
    At least certain Marvell SATA controllers are known to issue bus master
    requests with a non-zero function as origin, despite themselves being
    single function devices.

Here is what the manpage says:

+### pci-phantom
+> `=[<seg>:]<bus>:<device>,<stride>`
+
+Mark a group of PCI devices as using phantom functions without actually
+advertising so, so the IOMMU can create translation contexts for them.
+
+All numbers specified must be hexadecimal ones.
+
+This option can be specified more than once (up to 8 times at present).
+

Hm, time to try it out.
> 
> Gordan

>>> On 11.12.13 at 22:30, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
> On Wed, Dec 11, 2013 at 09:15:17PM +0000, Gordan Bobic wrote:
>> On 12/11/2013 06:32 PM, Konrad Rzeszutek Wilk wrote:
>> >Interestingly enough I just hit this with my brand-new Haswell CPU
and
>> >new motherboard when passing in a capture card. It shows:
>> >
>> >     +-1c.5-[07-09]----00.0-[08-09]--+-01.0-[09]--+-08.0  Brooktree
Corporation Bt878 Video
> Capture
>> >            |                               |            +-08.1 
Brooktree
> Corporation Bt878 Audio Capture
>> >            |                               |            +-09.0 
Brooktree
> Corporation Bt878 Video Capture
>> >            |                               |            +-09.1 
Brooktree
> Corporation Bt878 Audio Capture
>> >            |                               |            +-0a.0 
Brooktree
> Corporation Bt878 Video Capture
>> >            |                               |            +-0a.1 
Brooktree
> Corporation Bt878 Audio Capture
>> >            |                               |            +-0b.0 
Brooktree
> Corporation Bt878 Video Capture
>> >            |                               |            \-0b.1 
Brooktree
> Corporation Bt878 Audio Capture
>> >            |                               \-03.0  Texas
Instruments
> TSB43AB22A IEEE-1394a-2000 Controller (PHY/Link) [iOHCI-Lynx]
>> >
>> >And Xen says:
>> >(XEN) [VT-D]iommu.c:885: iommu_fault_status: Fault Overflow
>> >(XEN) [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault
>> >(XEN) [VT-D]iommu.c:865: DMAR:[DMA Read] Request device
[0000:08:00.0] fault
> addr 36aa3000, iommu reg = ffff82c3ffd53000
>> >(XEN) DMAR:[fault reason 02h] Present bit in context entry is clear
>> >(XEN) print_vtd_entries: iommu ffff83083d4939b0 dev 0000:08:00.0
gmfn 36aa3
>> >(XEN)     root_entry = ffff83083d47e000
>> >(XEN)     root_entry[8] = 72569a001
>> >(XEN)     context = ffff83072569a000
>> >(XEN)     context[0] = 0_0
>> >(XEN)     ctxt_entry[0] not present
>> >(XEN) [VT-D]iommu.c:885: iommu_fault_status: Fault Overflow
>> >(XEN) [VT-D]iommu.c:887: iommu_fault_status: Primary Pending Fault
>> >(XEN) [VT-D]iommu.c:865: DMAR:[DMA Read] Request device
[0000:08:00.0] fault
> addr 36aa3000, iommu reg = ffff82c3ffd53000
>> >
>> >
>> >Oddly enough it was working fine in a box with an AMD IOMMU. But
>> >to be fair - that machine was running with Xen 4.1.
>> >
>> >The hack I developed: 
> http://lists.xen.org/archives/html/xen-devel/2010-06/msg00093.html 
>> >ends up with this:
>> >
>> >(XEN) alloc_pdev: unknown type: 0000:08:00.0
>> >(XEN) [VT-D]iommu.c:1484: d0:unknown(0): 0000:08:00.0
>> >(XEN) [VT-D]iommu.c:1888: d0: context mapping failed
>> >
>> >(FYI, this Xen 4.3.1)
>> >
>> >Let me retry on the AMD box with the same version of Xen.
>> 
>> I may be wrong, but this doesn''t look like the same problem
(phantom
>> PCI device on the bus). Or am I missing something?
> 
> It is. A phantom device as well.
Nothing in what you posted confirms this (because it doesn''t
show what the upstream bridge(s) is/are).
>> As far as I can tell, the original problem was arising on cards that
>> are PCIe, but based on a PCIX chipset, i.e. with a PCIe-PCIX bridge.
>> Xen wasn''t the only thing affected in my case - bare metal
Linux
>> kernel was also having problems with intel-iommu=1 in the kernel
>> boot parameters. If might be worth trying that with your card to see
>> what happens. If bare metal Linux with intel-iommu=1 works for your
>> card, it''s probably not the same problem (of course it could
be
>> similar/related).
> 
> That is a similar problem here. Except that I have a PCI devices and
> it goes over an PCIe bridge (I think).
As said above - to see that''s the case would require to see more
lspci output than what you provided.
>> Out of interest, I noticed recently there is a xen parameter
>> "pci-phantom", but I haven''t been able to find
documentation for it.
>> Can you point me in the right direction? Does it, perchance, allow
>> specifying the PCI slot ID of a phantom device so that IOMMU
doesn''t
>> freak out when a seemingly non-existant device starts trying to do
>> DMA?
> 
> I forgot about it!
> 
> t 4e3c592c93d7dbe02ca36878457515d30fe931d2
> Author: Jan Beulich <jbeulich@suse.com>
> Date:   Mon Jan 7 12:58:09 2013 +0100
> 
>     IOMMU: add option to specify devices behaving like ones using phantom
functions
Note the term "functions" here: This is about a feature of the PCI
spec that some devices apparently use without declaring they do.
This has nothing to do with entire devices being invisible, in order
for there to be phantom functions there always has to be an
ordinary device at function 0.

Jan

On Fri, Dec 13, 2013 at 11:13:11AM +0000, Jan Beulich
wrote:
> >>> On 11.12.13 at 22:30, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
> > On Wed, Dec 11, 2013 at 09:15:17PM +0000, Gordan Bobic wrote:
> >> On 12/11/2013 06:32 PM, Konrad Rzeszutek Wilk wrote:
> >> >Interestingly enough I just hit this with my brand-new Haswell
CPU and
> >> >new motherboard when passing in a capture card. It shows:
> >> >
> >> >     +-1c.5-[07-09]----00.0-[08-09]--+-01.0-[09]--+-08.0 
Brooktree Corporation Bt878 Video
> > Capture
> >> >            |                               |           
+-08.1  Brooktree
> > Corporation Bt878 Audio Capture
> >> >            |                               |           
+-09.0  Brooktree
> > Corporation Bt878 Video Capture
> >> >            |                               |           
+-09.1  Brooktree
> > Corporation Bt878 Audio Capture
> >> >            |                               |           
+-0a.0  Brooktree
> > Corporation Bt878 Video Capture
> >> >            |                               |           
+-0a.1  Brooktree
> > Corporation Bt878 Audio Capture
> >> >            |                               |           
+-0b.0  Brooktree
> > Corporation Bt878 Video Capture
> >> >            |                               |           
\-0b.1  Brooktree
> > Corporation Bt878 Audio Capture
> >> >            |                               \-03.0  Texas
Instruments
> > TSB43AB22A IEEE-1394a-2000 Controller (PHY/Link) [iOHCI-Lynx]
> >> >
> >> >And Xen says:
> >> >(XEN) [VT-D]iommu.c:885: iommu_fault_status: Fault Overflow
> >> >(XEN) [VT-D]iommu.c:887: iommu_fault_status: Primary Pending
Fault
> >> >(XEN) [VT-D]iommu.c:865: DMAR:[DMA Read] Request device
[0000:08:00.0] fault
> > addr 36aa3000, iommu reg = ffff82c3ffd53000
> >> >(XEN) DMAR:[fault reason 02h] Present bit in context entry is
clear
> >> >(XEN) print_vtd_entries: iommu ffff83083d4939b0 dev
0000:08:00.0 gmfn 36aa3
> >> >(XEN)     root_entry = ffff83083d47e000
> >> >(XEN)     root_entry[8] = 72569a001
> >> >(XEN)     context = ffff83072569a000
> >> >(XEN)     context[0] = 0_0
> >> >(XEN)     ctxt_entry[0] not present
> >> >(XEN) [VT-D]iommu.c:885: iommu_fault_status: Fault Overflow
> >> >(XEN) [VT-D]iommu.c:887: iommu_fault_status: Primary Pending
Fault
> >> >(XEN) [VT-D]iommu.c:865: DMAR:[DMA Read] Request device
[0000:08:00.0] fault
> > addr 36aa3000, iommu reg = ffff82c3ffd53000
> >> >
> >> >
> >> >Oddly enough it was working fine in a box with an AMD IOMMU.
But
> >> >to be fair - that machine was running with Xen 4.1.
> >> >
> >> >The hack I developed: 
> > http://lists.xen.org/archives/html/xen-devel/2010-06/msg00093.html 
> >> >ends up with this:
> >> >
> >> >(XEN) alloc_pdev: unknown type: 0000:08:00.0
> >> >(XEN) [VT-D]iommu.c:1484: d0:unknown(0): 0000:08:00.0
> >> >(XEN) [VT-D]iommu.c:1888: d0: context mapping failed
> >> >
> >> >(FYI, this Xen 4.3.1)
> >> >
> >> >Let me retry on the AMD box with the same version of Xen.
> >> 
> >> I may be wrong, but this doesn''t look like the same
problem (phantom
> >> PCI device on the bus). Or am I missing something?
> > 
> > It is. A phantom device as well.
> 
> Nothing in what you posted confirms this (because it doesn''t
> show what the upstream bridge(s) is/are).
Ooops. Here is an output from lspci -v and lspci -vt

00:00.0 Host bridge: Intel Corporation 4th Gen Core Processor DRAM Controller
(rev 06)
	Subsystem: Super Micro Computer Inc Device 0805
	Flags: bus master, fast devsel, latency 0
	Capabilities: [e0] Vendor Specific Information: Len=0c <?>

00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI
Express x16 Controller (rev 06) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0
	Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
	I/O behind bridge: 0000e000-0000efff
	Memory behind bridge: f0d00000-f0dfffff
	Capabilities: [88] Subsystem: Intel Corporation Device 2010
	Capabilities: [80] Power Management version 3
	Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit-
	Capabilities: [a0] Express Root Port (Slot+), MSI 00
	Capabilities: [100] Virtual Channel
	Capabilities: [140] Root Complex Link
	Capabilities: [d94] #19
	Kernel driver in use: pcieport

00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI
Express x8 Controller (rev 06) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0
	Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
	I/O behind bridge: 0000d000-0000dfff
	Memory behind bridge: f0c00000-f0cfffff
	Capabilities: [88] Subsystem: Intel Corporation Device 2010
	Capabilities: [80] Power Management version 3
	Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit-
	Capabilities: [a0] Express Root Port (Slot+), MSI 00
	Capabilities: [100] Virtual Channel
	Capabilities: [140] Root Complex Link
	Capabilities: [d94] #19
	Kernel driver in use: pcieport

00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3/4th Gen
Core Processor Integrated Graphics Controller (rev 06) (prog-if 00 [VGA
controller])
	Subsystem: Intel Corporation Device 2010
	Flags: bus master, fast devsel, latency 0, IRQ 77
	Memory at f0000000 (64-bit, non-prefetchable) [size=4M]
	Memory at e0000000 (64-bit, prefetchable) [size=256M]
	I/O ports at f000 [size=64]
	Expansion ROM at <unassigned> [disabled]
	Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit-
	Capabilities: [d0] Power Management version 2
	Capabilities: [a4] PCI Advanced Features
	Kernel driver in use: i915

00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor
HD Audio Controller (rev 06)
	Subsystem: Intel Corporation Device 2010
	Flags: fast devsel, IRQ 16
	Memory at f0e34000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [50] Power Management version 2
	Capabilities: [60] MSI: Enable- Count=1/1 Maskable- 64bit-
	Capabilities: [70] Express Root Complex Integrated Endpoint, MSI 00

00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family
USB xHCI (rev 04) (prog-if 30 [XHCI])
	Subsystem: Super Micro Computer Inc Device 0805
	Flags: bus master, medium devsel, latency 0, IRQ 71
	Memory at f0e20000 (64-bit, non-prefetchable) [size=64K]
	Capabilities: [70] Power Management version 2
	Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+
	Kernel driver in use: xhci_hcd

00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset
Family MEI Controller #1 (rev 04)
	Subsystem: Super Micro Computer Inc Device 0805
	Flags: bus master, fast devsel, latency 0, IRQ 78
	Memory at f0e40000 (64-bit, non-prefetchable) [size=16]
	Capabilities: [50] Power Management version 3
	Capabilities: [8c] MSI: Enable+ Count=1/1 Maskable- 64bit+
	Kernel driver in use: mei_me

00:16.3 Serial controller: Intel Corporation 8 Series/C220 Series Chipset Family
KT Controller (rev 04) (prog-if 02 [16550])
	Subsystem: Super Micro Computer Inc Device 0805
	Flags: bus master, 66MHz, fast devsel, latency 0, IRQ 19
	I/O ports at f0e0 [size=8]
	Memory at f0e3e000 (32-bit, non-prefetchable) [size=4K]
	Capabilities: [c8] Power Management version 3
	Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
	Kernel driver in use: serial

00:19.0 Ethernet controller: Intel Corporation Ethernet Connection I217-LM (rev
04)
	Subsystem: Super Micro Computer Inc Device 153a
	Flags: bus master, fast devsel, latency 0, IRQ 84
	Memory at f0e00000 (32-bit, non-prefetchable) [size=128K]
	Memory at f0e3d000 (32-bit, non-prefetchable) [size=4K]
	I/O ports at f080 [size=32]
	Capabilities: [c8] Power Management version 2
	Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
	Capabilities: [e0] PCI Advanced Features
	Kernel driver in use: e1000e

00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family
USB EHCI #2 (rev 04) (prog-if 20 [EHCI])
	Subsystem: Super Micro Computer Inc Device 0805
	Flags: bus master, medium devsel, latency 0, IRQ 16
	Memory at f0e3c000 (32-bit, non-prefetchable) [size=1K]
	Capabilities: [50] Power Management version 2
	Capabilities: [58] Debug port: BAR=1 offset=00a0
	Capabilities: [98] PCI Advanced Features
	Kernel driver in use: ehci-pci

00:1b.0 Audio device: Intel Corporation 8 Series/C220 Series Chipset High
Definition Audio Controller (rev 04)
	Subsystem: Super Micro Computer Inc Device 0805
	Flags: fast devsel, IRQ 22
	Memory at f0e30000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [50] Power Management version 2
	Capabilities: [60] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [70] Express Root Complex Integrated Endpoint, MSI 00
	Capabilities: [100] Virtual Channel

00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI
Express Root Port #1 (rev d4) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0
	Bus: primary=00, secondary=03, subordinate=03, sec-latency=0
	I/O behind bridge: 0000c000-0000cfff
	Memory behind bridge: f0b00000-f0bfffff
	Capabilities: [40] Express Root Port (Slot+), MSI 00
	Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
	Capabilities: [90] Subsystem: Super Micro Computer Inc Device 0805
	Capabilities: [a0] Power Management version 3
	Kernel driver in use: pcieport

00:1c.1 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI
Express Root Port #2 (rev d4) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0
	Bus: primary=00, secondary=04, subordinate=04, sec-latency=0
	I/O behind bridge: 0000b000-0000bfff
	Memory behind bridge: f0a00000-f0afffff
	Capabilities: [40] Express Root Port (Slot+), MSI 00
	Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
	Capabilities: [90] Subsystem: Super Micro Computer Inc Device 0805
	Capabilities: [a0] Power Management version 3
	Kernel driver in use: pcieport

00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI
Express Root Port #4 (rev d4) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0
	Bus: primary=00, secondary=05, subordinate=05, sec-latency=0
	I/O behind bridge: 0000a000-0000afff
	Memory behind bridge: f0900000-f09fffff
	Capabilities: [40] Express Root Port (Slot+), MSI 00
	Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
	Capabilities: [90] Subsystem: Super Micro Computer Inc Device 0805
	Capabilities: [a0] Power Management version 3
	Kernel driver in use: pcieport

00:1c.4 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI
Express Root Port #5 (rev d4) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0
	Bus: primary=00, secondary=06, subordinate=06, sec-latency=0
	I/O behind bridge: 00009000-00009fff
	Memory behind bridge: f0800000-f08fffff
	Capabilities: [40] Express Root Port (Slot+), MSI 00
	Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
	Capabilities: [90] Subsystem: Super Micro Computer Inc Device 0805
	Capabilities: [a0] Power Management version 3
	Kernel driver in use: pcieport

00:1c.5 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI
Express Root Port #6 (rev d4) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0
	Bus: primary=00, secondary=07, subordinate=09, sec-latency=0
	Memory behind bridge: f0400000-f05fffff
	Capabilities: [40] Express Root Port (Slot+), MSI 00
	Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
	Capabilities: [90] Subsystem: Super Micro Computer Inc Device 0805
	Capabilities: [a0] Power Management version 3
	Kernel driver in use: pcieport

00:1c.6 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI
Express Root Port #7 (rev d4) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0
	Bus: primary=00, secondary=0a, subordinate=0a, sec-latency=0
	Memory behind bridge: f0700000-f07fffff
	Capabilities: [40] Express Root Port (Slot+), MSI 00
	Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
	Capabilities: [90] Subsystem: Super Micro Computer Inc Device 0805
	Capabilities: [a0] Power Management version 3
	Kernel driver in use: pcieport

00:1c.7 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI
Express Root Port #8 (rev d4) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0
	Bus: primary=00, secondary=0b, subordinate=0b, sec-latency=0
	I/O behind bridge: 00008000-00008fff
	Memory behind bridge: f0600000-f06fffff
	Capabilities: [40] Express Root Port (Slot+), MSI 00
	Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
	Capabilities: [90] Subsystem: Super Micro Computer Inc Device 0805
	Capabilities: [a0] Power Management version 3
	Kernel driver in use: pcieport

00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family
USB EHCI #1 (rev 04) (prog-if 20 [EHCI])
	Subsystem: Super Micro Computer Inc Device 0805
	Flags: bus master, medium devsel, latency 0, IRQ 23
	Memory at f0e3b000 (32-bit, non-prefetchable) [size=1K]
	Capabilities: [50] Power Management version 2
	Capabilities: [58] Debug port: BAR=1 offset=00a0
	Capabilities: [98] PCI Advanced Features
	Kernel driver in use: ehci-pci

00:1f.0 ISA bridge: Intel Corporation C226 Series Chipset Family Server Advanced
SKU LPC Controller (rev 04)
	Subsystem: Super Micro Computer Inc Device 0805
	Flags: bus master, medium devsel, latency 0
	Capabilities: [e0] Vendor Specific Information: Len=0c <?>
	Kernel driver in use: lpc_ich

00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family
6-port SATA Controller 1 [AHCI mode] (rev 04) (prog-if 01 [AHCI 1.0])
	Subsystem: Super Micro Computer Inc Device 0805
	Flags: bus master, 66MHz, medium devsel, latency 0, IRQ 69
	I/O ports at f0d0 [size=8]
	I/O ports at f0c0 [size=4]
	I/O ports at f0b0 [size=8]
	I/O ports at f0a0 [size=4]
	I/O ports at f060 [size=32]
	Memory at f0e3a000 (32-bit, non-prefetchable) [size=2K]
	Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
	Capabilities: [70] Power Management version 3
	Capabilities: [a8] SATA HBA v1.0
	Kernel driver in use: ahci

00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus
Controller (rev 04)
	Subsystem: Super Micro Computer Inc Device 0805
	Flags: medium devsel, IRQ 18
	Memory at f0e39000 (64-bit, non-prefetchable) [size=256]
	I/O ports at f040 [size=32]
	Kernel driver in use: i801_smbus

00:1f.6 Signal processing controller: Intel Corporation 8 Series Chipset Family
Thermal Management Controller (rev 04)
	Subsystem: Super Micro Computer Inc Device 0805
	Flags: bus master, fast devsel, latency 0, IRQ 5
	Memory at f0e38000 (64-bit, non-prefetchable) [size=4K]
	Capabilities: [50] Power Management version 3
	Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-

01:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet
Controller (rev 06)
	Subsystem: Intel Corporation PRO/1000 PT Dual Port Server Adapter
	Flags: bus master, fast devsel, latency 0, IRQ 79
	Memory at f0da0000 (32-bit, non-prefetchable) [size=128K]
	Memory at f0d80000 (32-bit, non-prefetchable) [size=128K]
	I/O ports at e020 [size=32]
	Expansion ROM at f0d60000 [disabled] [size=128K]
	Capabilities: [c8] Power Management version 2
	Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
	Capabilities: [e0] Express Endpoint, MSI 00
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [140] Device Serial Number 00-15-17-ff-ff-3e-24-4c
	Kernel driver in use: e1000e

01:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet
Controller (rev 06)
	Subsystem: Intel Corporation PRO/1000 PT Dual Port Server Adapter
	Flags: bus master, fast devsel, latency 0, IRQ 80
	Memory at f0d40000 (32-bit, non-prefetchable) [size=128K]
	Memory at f0d20000 (32-bit, non-prefetchable) [size=128K]
	I/O ports at e000 [size=32]
	Expansion ROM at f0d00000 [disabled] [size=128K]
	Capabilities: [c8] Power Management version 2
	Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
	Capabilities: [e0] Express Endpoint, MSI 00
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [140] Device Serial Number 00-15-17-ff-ff-3e-24-4c
	Kernel driver in use: e1000e

02:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008
PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 02)
	Subsystem: LSI Logic / Symbios Logic Device 3020
	Flags: bus master, fast devsel, latency 0, IRQ 17
	I/O ports at d000 [size=256]
	Memory at f0cc0000 (64-bit, non-prefetchable) [size=16K]
	Memory at f0c80000 (64-bit, non-prefetchable) [size=256K]
	Expansion ROM at f0c00000 [disabled] [size=512K]
	Capabilities: [50] Power Management version 3
	Capabilities: [68] Express Endpoint, MSI 00
	Capabilities: [d0] Vital Product Data
	Capabilities: [a8] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [c0] MSI-X: Enable+ Count=15 Masked-
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [138] Power Budgeting <?>
	Capabilities: [150] Single Root I/O Virtualization (SR-IOV)
	Capabilities: [190] Alternative Routing-ID Interpretation (ARI)
	Kernel driver in use: mpt2sas

03:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
	Subsystem: Intel Corporation Gigabit CT Desktop Adapter
	Flags: bus master, fast devsel, latency 0, IRQ 16
	Memory at f0bc0000 (32-bit, non-prefetchable) [size=128K]
	Memory at f0b00000 (32-bit, non-prefetchable) [size=512K]
	I/O ports at c000 [size=32]
	Memory at f0be0000 (32-bit, non-prefetchable) [size=16K]
	Expansion ROM at f0b80000 [disabled] [size=256K]
	Capabilities: [c8] Power Management version 2
	Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [e0] Express Endpoint, MSI 00
	Capabilities: [a0] MSI-X: Enable+ Count=5 Masked-
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [140] Device Serial Number 00-1b-21-ff-ff-61-49-00
	Kernel driver in use: pciback

04:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
	Subsystem: Intel Corporation Gigabit CT Desktop Adapter
	Flags: bus master, fast devsel, latency 0, IRQ 17
	Memory at f0ac0000 (32-bit, non-prefetchable) [size=128K]
	Memory at f0a00000 (32-bit, non-prefetchable) [size=512K]
	I/O ports at b000 [size=32]
	Memory at f0ae0000 (32-bit, non-prefetchable) [size=16K]
	Expansion ROM at f0a80000 [disabled] [size=256K]
	Capabilities: [c8] Power Management version 2
	Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [e0] Express Endpoint, MSI 00
	Capabilities: [a0] MSI-X: Enable+ Count=5 Masked-
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [140] Device Serial Number 68-05-ca-ff-ff-02-4c-71
	Kernel driver in use: pciback

05:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection
(rev 03)
	Subsystem: Super Micro Computer Inc Device 1533
	Flags: fast devsel, IRQ 19
	Memory at f0900000 (32-bit, non-prefetchable) [disabled] [size=512K]
	I/O ports at a000 [disabled] [size=32]
	Memory at f0980000 (32-bit, non-prefetchable) [disabled] [size=16K]
	Capabilities: [40] Power Management version 3
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
	Capabilities: [70] MSI-X: Enable- Count=5 Masked-
	Capabilities: [a0] Express Endpoint, MSI 00
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [140] Device Serial Number 00-25-90-ff-ff-86-c7-57
	Capabilities: [1a0] Transaction Processing Hints
	Kernel driver in use: pciback

06:00.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet
Controller (Copper) (rev 06)
	Subsystem: Intel Corporation PRO/1000 PT Desktop Adapter
	Flags: fast devsel, IRQ 16
	Memory at f0840000 (32-bit, non-prefetchable) [disabled] [size=128K]
	Memory at f0820000 (32-bit, non-prefetchable) [disabled] [size=128K]
	I/O ports at 9000 [disabled] [size=32]
	Expansion ROM at f0800000 [disabled] [size=128K]
	Capabilities: [c8] Power Management version 2
	Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [e0] Express Endpoint, MSI 00
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [140] Device Serial Number 00-1b-21-ff-ff-00-37-02
	Kernel driver in use: pciback

07:00.0 PCI bridge: Tundra Semiconductor Corp. Device 8113 (rev 01) (prog-if 01
[Subtractive decode])
	Flags: bus master, fast devsel, latency 0
	Bus: primary=07, secondary=08, subordinate=09, sec-latency=32
	Memory behind bridge: f0400000-f05fffff
	Capabilities: [60] Subsystem: Super Micro Computer Inc Device 0805
	Capabilities: [a0] Power Management version 3

08:01.0 PCI bridge: Hint Corp HB6 Universal PCI-PCI bridge (non-transparent
mode) (rev 11) (prog-if 00 [Normal decode])
	Physical Slot: 2
	Flags: bus master, medium devsel, latency 32
	Bus: primary=08, secondary=09, subordinate=09, sec-latency=32
	Memory behind bridge: f0400000-f04fffff
	Capabilities: [80] Power Management version 2
	Capabilities: [90] CompactPCI hot-swap <?>

08:03.0 FireWire (IEEE 1394): Texas Instruments TSB43AB22A IEEE-1394a-2000
Controller (PHY/Link) [iOHCI-Lynx] (prog-if 10 [OHCI])
	Subsystem: Super Micro Computer Inc Device 0805
	Physical Slot: 4
	Flags: medium devsel, IRQ 16
	Memory at f0504000 (32-bit, non-prefetchable) [size=2K]
	Memory at f0500000 (32-bit, non-prefetchable) [size=16K]
	Capabilities: [44] Power Management version 2

09:08.0 Multimedia video controller: Brooktree Corporation Bt878 Video Capture
(rev 11)
	Subsystem: Device aa00:1460
	Flags: medium devsel, IRQ 18
	Memory at f0407000 (32-bit, prefetchable) [disabled] [size=4K]
	Capabilities: [44] Vital Product Data
	Capabilities: [4c] Power Management version 2
	Kernel driver in use: pciback

09:08.1 Multimedia controller: Brooktree Corporation Bt878 Audio Capture (rev
11)
	Subsystem: Device aa00:1460
	Flags: medium devsel, IRQ 18
	Memory at f0406000 (32-bit, prefetchable) [disabled] [size=4K]
	Capabilities: [44] Vital Product Data
	Capabilities: [4c] Power Management version 2
	Kernel driver in use: pciback

09:09.0 Multimedia video controller: Brooktree Corporation Bt878 Video Capture
(rev 11)
	Subsystem: Device aa01:1461
	Flags: medium devsel, IRQ 19
	Memory at f0405000 (32-bit, prefetchable) [disabled] [size=4K]
	Capabilities: [44] Vital Product Data
	Capabilities: [4c] Power Management version 2
	Kernel driver in use: pciback

09:09.1 Multimedia controller: Brooktree Corporation Bt878 Audio Capture (rev
11)
	Subsystem: Device aa01:1461
	Flags: medium devsel, IRQ 19
	Memory at f0404000 (32-bit, prefetchable) [disabled] [size=4K]
	Capabilities: [44] Vital Product Data
	Capabilities: [4c] Power Management version 2
	Kernel driver in use: pciback

09:0a.0 Multimedia video controller: Brooktree Corporation Bt878 Video Capture
(rev 11)
	Subsystem: Device aa02:1462
	Flags: medium devsel, IRQ 16
	Memory at f0403000 (32-bit, prefetchable) [disabled] [size=4K]
	Capabilities: [44] Vital Product Data
	Capabilities: [4c] Power Management version 2
	Kernel driver in use: pciback

09:0a.1 Multimedia controller: Brooktree Corporation Bt878 Audio Capture (rev
11)
	Subsystem: Device aa02:1462
	Flags: medium devsel, IRQ 16
	Memory at f0402000 (32-bit, prefetchable) [disabled] [size=4K]
	Capabilities: [44] Vital Product Data
	Capabilities: [4c] Power Management version 2
	Kernel driver in use: pciback

09:0b.0 Multimedia video controller: Brooktree Corporation Bt878 Video Capture
(rev 11)
	Subsystem: Device aa03:1463
	Flags: medium devsel, IRQ 17
	Memory at f0401000 (32-bit, prefetchable) [disabled] [size=4K]
	Capabilities: [44] Vital Product Data
	Capabilities: [4c] Power Management version 2
	Kernel driver in use: pciback

09:0b.1 Multimedia controller: Brooktree Corporation Bt878 Audio Capture (rev
11)
	Subsystem: Device aa03:1463
	Flags: medium devsel, IRQ 17
	Memory at f0400000 (32-bit, prefetchable) [disabled] [size=4K]
	Capabilities: [44] Vital Product Data
	Capabilities: [4c] Power Management version 2
	Kernel driver in use: pciback

0a:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host
Controller (rev 02) (prog-if 30 [XHCI])
	Subsystem: Super Micro Computer Inc Device 0805
	Flags: bus master, fast devsel, latency 0, IRQ 18
	Memory at f0700000 (64-bit, non-prefetchable) [size=8K]
	Capabilities: [50] Power Management version 3
	Capabilities: [70] MSI: Enable- Count=1/8 Maskable- 64bit+
	Capabilities: [90] MSI-X: Enable+ Count=8 Masked-
	Capabilities: [a0] Express Endpoint, MSI 00
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [150] Latency Tolerance Reporting
	Kernel driver in use: xhci_hcd

0b:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller
(rev 01) (prog-if 01 [AHCI 1.0])
	Subsystem: Super Micro Computer Inc Device 0805
	Flags: bus master, fast devsel, latency 0, IRQ 70
	I/O ports at 8050 [size=8]
	I/O ports at 8040 [size=4]
	I/O ports at 8030 [size=8]
	I/O ports at 8020 [size=4]
	I/O ports at 8000 [size=32]
	Memory at f0600000 (32-bit, non-prefetchable) [size=512]
	Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit-
	Capabilities: [78] Power Management version 3
	Capabilities: [80] Express Legacy Endpoint, MSI 00
	Capabilities: [100] Virtual Channel
	Kernel driver in use: ahci

-[0000:00]-+-00.0  Intel Corporation 4th Gen Core Processor DRAM Controller
           +-01.0-[01]--+-00.0  Intel Corporation 82571EB Gigabit Ethernet
Controller
           |            \-00.1  Intel Corporation 82571EB Gigabit Ethernet
Controller
           +-01.1-[02]----00.0  LSI Logic / Symbios Logic SAS2008 PCI-Express
Fusion-MPT SAS-2 [Falcon]
           +-02.0  Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor
Integrated Graphics Controller
           +-03.0  Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD
Audio Controller
           +-14.0  Intel Corporation 8 Series/C220 Series Chipset Family USB
xHCI
           +-16.0  Intel Corporation 8 Series/C220 Series Chipset Family MEI
Controller #1
           +-16.3  Intel Corporation 8 Series/C220 Series Chipset Family KT
Controller
           +-19.0  Intel Corporation Ethernet Connection I217-LM
           +-1a.0  Intel Corporation 8 Series/C220 Series Chipset Family USB
EHCI #2
           +-1b.0  Intel Corporation 8 Series/C220 Series Chipset High
Definition Audio Controller
           +-1c.0-[03]----00.0  Intel Corporation 82574L Gigabit Network
Connection
           +-1c.1-[04]----00.0  Intel Corporation 82574L Gigabit Network
Connection
           +-1c.3-[05]----00.0  Intel Corporation I210 Gigabit Network
Connection
           +-1c.4-[06]----00.0  Intel Corporation 82572EI Gigabit Ethernet
Controller (Copper)
           +-1c.5-[07-09]----00.0-[08-09]--+-01.0-[09]--+-08.0  Brooktree
Corporation Bt878 Video Capture
           |                               |            +-08.1  Brooktree
Corporation Bt878 Audio Capture
           |                               |            +-09.0  Brooktree
Corporation Bt878 Video Capture
           |                               |            +-09.1  Brooktree
Corporation Bt878 Audio Capture
           |                               |            +-0a.0  Brooktree
Corporation Bt878 Video Capture
           |                               |            +-0a.1  Brooktree
Corporation Bt878 Audio Capture
           |                               |            +-0b.0  Brooktree
Corporation Bt878 Video Capture
           |                               |            \-0b.1  Brooktree
Corporation Bt878 Audio Capture
           |                               \-03.0  Texas Instruments TSB43AB22A
IEEE-1394a-2000 Controller (PHY/Link) [iOHCI-Lynx]
           +-1c.6-[0a]----00.0  Renesas Technology Corp. uPD720202 USB 3.0 Host
Controller
           +-1c.7-[0b]----00.0  ASMedia Technology Inc. ASM1062 Serial ATA
Controller
           +-1d.0  Intel Corporation 8 Series/C220 Series Chipset Family USB
EHCI #1
           +-1f.0  Intel Corporation C226 Series Chipset Family Server Advanced
SKU LPC Controller
           +-1f.2  Intel Corporation 8 Series/C220 Series Chipset Family 6-port
SATA Controller 1 [AHCI mode]
           +-1f.3  Intel Corporation 8 Series/C220 Series Chipset Family SMBus
Controller
           \-1f.6  Intel Corporation 8 Series Chipset Family Thermal Management
Controller
> 
> >> As far as I can tell, the original problem was arising on cards
that
> >> are PCIe, but based on a PCIX chipset, i.e. with a PCIe-PCIX
bridge.
> >> Xen wasn''t the only thing affected in my case - bare
metal Linux
> >> kernel was also having problems with intel-iommu=1 in the kernel
> >> boot parameters. If might be worth trying that with your card to
see
> >> what happens. If bare metal Linux with intel-iommu=1 works for
your
> >> card, it''s probably not the same problem (of course it
could be
> >> similar/related).
> > 
> > That is a similar problem here. Except that I have a PCI devices and
> > it goes over an PCIe bridge (I think).
> 
> As said above - to see that''s the case would require to see more
> lspci output than what you provided.
> 
> >> Out of interest, I noticed recently there is a xen parameter
> >> "pci-phantom", but I haven''t been able to find
documentation for it.
> >> Can you point me in the right direction? Does it, perchance, allow
> >> specifying the PCI slot ID of a phantom device so that IOMMU
doesn''t
> >> freak out when a seemingly non-existant device starts trying to do
> >> DMA?
> > 
> > I forgot about it!
> > 
> > t 4e3c592c93d7dbe02ca36878457515d30fe931d2
> > Author: Jan Beulich <jbeulich@suse.com>
> > Date:   Mon Jan 7 12:58:09 2013 +0100
> > 
> >     IOMMU: add option to specify devices behaving like ones using
phantom functions
> 
> Note the term "functions" here: This is about a feature of the
PCI
> spec that some devices apparently use without declaring they do.
> This has nothing to do with entire devices being invisible, in order
> for there to be phantom functions there always has to be an
> ordinary device at function 0.
> 
> Jan

>>> On 13.12.13 at 15:43, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
> -[0000:00]-+-00.0  Intel Corporation 4th Gen Core Processor DRAM Controller
>            +-01.0-[01]--+-00.0  Intel Corporation 82571EB Gigabit Ethernet
Controller
>            |            \-00.1  Intel Corporation 82571EB Gigabit Ethernet
Controller
>            +-01.1-[02]----00.0  LSI Logic / Symbios Logic SAS2008
PCI-Express Fusion-MPT SAS-2 [Falcon]
>            +-02.0  Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor
Integrated Graphics Controller
>            +-03.0  Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor
HD Audio Controller
>            +-14.0  Intel Corporation 8 Series/C220 Series Chipset Family
USB xHCI
>            +-16.0  Intel Corporation 8 Series/C220 Series Chipset Family
MEI Controller #1
>            +-16.3  Intel Corporation 8 Series/C220 Series Chipset Family KT
Controller
>            +-19.0  Intel Corporation Ethernet Connection I217-LM
>            +-1a.0  Intel Corporation 8 Series/C220 Series Chipset Family
USB EHCI #2
>            +-1b.0  Intel Corporation 8 Series/C220 Series Chipset High
Definition Audio Controller
>            +-1c.0-[03]----00.0  Intel Corporation 82574L Gigabit Network
Connection
>            +-1c.1-[04]----00.0  Intel Corporation 82574L Gigabit Network
Connection
>            +-1c.3-[05]----00.0  Intel Corporation I210 Gigabit Network
Connection
>            +-1c.4-[06]----00.0  Intel Corporation 82572EI Gigabit Ethernet
Controller (Copper)
>            +-1c.5-[07-09]----00.0-[08-09]--+-01.0-[09]--+-08.0  Brooktree
Corporation Bt878 Video Capture
>            |                               |            +-08.1  Brooktree
Corporation Bt878 Audio Capture
>            |                               |            +-09.0  Brooktree
Corporation Bt878 Video Capture
>            |                               |            +-09.1  Brooktree
Corporation Bt878 Audio Capture
>            |                               |            +-0a.0  Brooktree
Corporation Bt878 Video Capture
>            |                               |            +-0a.1  Brooktree
Corporation Bt878 Audio Capture
>            |                               |            +-0b.0  Brooktree
Corporation Bt878 Video Capture
>            |                               |            \-0b.1  Brooktree
Corporation Bt878 Audio Capture
So 00:1c.5 is the PCIe->PCI bridge, 07:00.0 and 08:01.0 are
PCI-PCI bridges, the capture devices are ordinary ones. The only
possibly unexpected aspect I can see here is that there are two
intermediate PCI-PCI bridges, but iirc this should be handled by
the VT-d code. You''d therefore mainly want to look at
find_upstream_bridge() getting things wrong, or
pseg->bus2bridge[] not getting set up correctly.

Jan

On 12/13/2013 02:56 PM, Jan Beulich wrote:
>>>> On 13.12.13 at 15:43, Konrad Rzeszutek Wilk
<konrad.wilk@oracle.com> wrote:
>> -[0000:00]-+-00.0  Intel Corporation 4th Gen Core Processor DRAM
Controller
>>             +-01.0-[01]--+-00.0  Intel Corporation 82571EB Gigabit
Ethernet Controller
>>             |            \-00.1  Intel Corporation 82571EB Gigabit
Ethernet Controller
>>             +-01.1-[02]----00.0  LSI Logic / Symbios Logic SAS2008
PCI-Express Fusion-MPT SAS-2 [Falcon]
>>             +-02.0  Intel Corporation Xeon E3-1200 v3/4th Gen Core
Processor Integrated Graphics Controller
>>             +-03.0  Intel Corporation Xeon E3-1200 v3/4th Gen Core
Processor HD Audio Controller
>>             +-14.0  Intel Corporation 8 Series/C220 Series Chipset
Family USB xHCI
>>             +-16.0  Intel Corporation 8 Series/C220 Series Chipset
Family MEI Controller #1
>>             +-16.3  Intel Corporation 8 Series/C220 Series Chipset
Family KT Controller
>>             +-19.0  Intel Corporation Ethernet Connection I217-LM
>>             +-1a.0  Intel Corporation 8 Series/C220 Series Chipset
Family USB EHCI #2
>>             +-1b.0  Intel Corporation 8 Series/C220 Series Chipset High
Definition Audio Controller
>>             +-1c.0-[03]----00.0  Intel Corporation 82574L Gigabit
Network Connection
>>             +-1c.1-[04]----00.0  Intel Corporation 82574L Gigabit
Network Connection
>>             +-1c.3-[05]----00.0  Intel Corporation I210 Gigabit Network
Connection
>>             +-1c.4-[06]----00.0  Intel Corporation 82572EI Gigabit
Ethernet Controller (Copper)
>>             +-1c.5-[07-09]----00.0-[08-09]--+-01.0-[09]--+-08.0 
Brooktree Corporation Bt878 Video Capture
>>             |                               |            +-08.1 
Brooktree Corporation Bt878 Audio Capture
>>             |                               |            +-09.0 
Brooktree Corporation Bt878 Video Capture
>>             |                               |            +-09.1 
Brooktree Corporation Bt878 Audio Capture
>>             |                               |            +-0a.0 
Brooktree Corporation Bt878 Video Capture
>>             |                               |            +-0a.1 
Brooktree Corporation Bt878 Audio Capture
>>             |                               |            +-0b.0 
Brooktree Corporation Bt878 Video Capture
>>             |                               |            \-0b.1 
Brooktree Corporation Bt878 Audio Capture
>
> So 00:1c.5 is the PCIe->PCI bridge, 07:00.0 and 08:01.0 are
> PCI-PCI bridges, the capture devices are ordinary ones. The only
> possibly unexpected aspect I can see here is that there are two
> intermediate PCI-PCI bridges, but iirc this should be handled by
> the VT-d code.
Not sure about multiple intermediate PCI-PCI bridges, but I can confirm 
that multiuple intermediate PCIe bridge setup works fine,
e.g. Intel X58 -> Nvidia NF200 -> PLX -> GPU (GTX690)

Gordan