Xen devel - Jun 2008 - [PATCH][RFC] Support more Capability Structures and Device Specific

I am submitting the patch which supports more Capability Structures
and Device Specific Registers for passthrough device.

In Xen 3.3 unstable, qemu-dm supports Configuration Header, MSI
Capability Structure, and MSI-X Capability Structure. But qemu-dm does
not support PCI Express Capability Structure, Device Specific
Registers, etc (writing them is ignored).

To support various I/O devices, I implemented following Capability
Structures and Device Specific Registers.

    * Configuration Header Type 0
        -> emulation.
           "emulation" does not mean no accessing real I/O device.
           Access real I/O device, but guest value and real value
           might be different.
    * PCI Express Capability Structure
        -> emulation.
    * PCI Power Management Capability Structure
        -> emulation.
    * Vital Product Data Capability Structure
        -> emulation (almost passthrough).
    * Vendor Specific Capability Structure
        -> emulation (almost passthrough).
    * Device Specific Register (exclude capability structures)
        -> passthrough.
           The device drivers in guest domain are allowed to access
           Device Specific Register. So various I/O device will work.

Currently MSI Capability Structure and MSI-X Capability Structure is
not implemented, and they are hidden from guest software. I disabled
MSI and MSI-X in qemu-dm temporary. I am implementing MSI Capability
Structure and merging current MSI routines. I will release the patch
if you agree with me.

MSI-X will be after MSI. I will be very happy if anyone can help me.

Other Capability Structures are hidden from guest software. To do
this, I change Next Capability Pointer''s value to point only the
Capability Structure that need to be exported to guest software
(see emulate capabilities above). And some Capability Structures are 0
hardwired, and others are passthrough.

This patch removes "switch" statements for emulation, and introduces
table based emulation derived from pciback driver. You can implement
new Capability Structure by adding new table.
The other advantage of using this table is that you can easily change
the emulation policy of each field/bit by just simply modifying the
"emu_mask" value provided in each register table.
And for only special emulation or interacting with other components
(like hypervisor), you have to implement function corresponding to the
register.

Signed-off-by: Yuji Shimada <shimada-yxb@necst.nec.co.jp>

--
Yuji Shimada

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Yuji:
	We have a discussion in xen summit for PCI CFGS emulation, are
you in the summit too?
	Here is the slide for the discussion, we can coordinate.
Thx, eddie

Yuji Shimada wrote:
> I am submitting the patch which supports more Capability
> Structures and Device Specific Registers for passthrough
> device. 
> 
> In Xen 3.3 unstable, qemu-dm supports Configuration
> Header, MSI Capability Structure, and MSI-X Capability
> Structure. But qemu-dm does not support PCI Express
> Capability Structure, Device Specific Registers, etc
> (writing them is ignored). 
> 
> To support various I/O devices, I implemented following
> Capability Structures and Device Specific Registers.
> 
>     * Configuration Header Type 0
>         -> emulation.
>            "emulation" does not mean no accessing real
>            I/O device. Access real I/O device, but guest
>            value and real value might be different.
>     * PCI Express Capability Structure
>         -> emulation.
>     * PCI Power Management Capability Structure
>         -> emulation.
>     * Vital Product Data Capability Structure
>         -> emulation (almost passthrough).
>     * Vendor Specific Capability Structure
>         -> emulation (almost passthrough).
>     * Device Specific Register (exclude capability
>         structures) -> passthrough.
>            The device drivers in guest domain are allowed
>            to access Device Specific Register. So various
> I/O device will work. 
> 
> Currently MSI Capability Structure and MSI-X Capability
> Structure is not implemented, and they are hidden from
> guest software. I disabled MSI and MSI-X in qemu-dm
> temporary. I am implementing MSI Capability Structure and
> merging current MSI routines. I will release the patch if
> you agree with me. 
> 
> MSI-X will be after MSI. I will be very happy if anyone
> can help me. 
> 
> Other Capability Structures are hidden from guest
> software. To do this, I change Next Capability Pointer''s
> value to point only the Capability Structure that need to
> be exported to guest software (see emulate capabilities
> above). And some Capability Structures are 0 hardwired,
> and others are passthrough. 
> 
> This patch removes "switch" statements for emulation, and
> introduces table based emulation derived from pciback
> driver. You can implement new Capability Structure by
> adding new table. 
> The other advantage of using this table is that you can
> easily change the emulation policy of each field/bit by
> just simply modifying the "emu_mask" value provided in
> each register table. 
> And for only special emulation or interacting with other
> components (like hypervisor), you have to implement
> function corresponding to the register.
> 
> Signed-off-by: Yuji Shimada <shimada-yxb@necst.nec.co.jp>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

What do you think of Yuji''s patch? One thing to consider is that
perhaps the
patch should be against the upstream qemu merge now. But I''m not sure
that
PCI passthrough is even supported yet in that tree (Ian?).

 -- Keir

On 27/6/08 11:14, "Dong, Eddie" <eddie.dong@intel.com> wrote:
> Yuji:
> We have a discussion in xen summit for PCI CFGS emulation, are
> you in the summit too?
> Here is the slide for the discussion, we can coordinate.
> Thx, eddie
> 
> Yuji Shimada wrote:
>> I am submitting the patch which supports more Capability
>> Structures and Device Specific Registers for passthrough
>> device. 
>> 
>> In Xen 3.3 unstable, qemu-dm supports Configuration
>> Header, MSI Capability Structure, and MSI-X Capability
>> Structure. But qemu-dm does not support PCI Express
>> Capability Structure, Device Specific Registers, etc
>> (writing them is ignored).
>> 
>> To support various I/O devices, I implemented following
>> Capability Structures and Device Specific Registers.
>> 
>>     * Configuration Header Type 0
>>         -> emulation.
>>            "emulation" does not mean no accessing real
>>            I/O device. Access real I/O device, but guest
>>            value and real value might be different.
>>     * PCI Express Capability Structure
>>         -> emulation.
>>     * PCI Power Management Capability Structure
>>         -> emulation.
>>     * Vital Product Data Capability Structure
>>         -> emulation (almost passthrough).
>>     * Vendor Specific Capability Structure
>>         -> emulation (almost passthrough).
>>     * Device Specific Register (exclude capability
>>         structures) -> passthrough.
>>            The device drivers in guest domain are allowed
>>            to access Device Specific Register. So various
>> I/O device will work.
>> 
>> Currently MSI Capability Structure and MSI-X Capability
>> Structure is not implemented, and they are hidden from
>> guest software. I disabled MSI and MSI-X in qemu-dm
>> temporary. I am implementing MSI Capability Structure and
>> merging current MSI routines. I will release the patch if
>> you agree with me.
>> 
>> MSI-X will be after MSI. I will be very happy if anyone
>> can help me. 
>> 
>> Other Capability Structures are hidden from guest
>> software. To do this, I change Next Capability Pointer''s
>> value to point only the Capability Structure that need to
>> be exported to guest software (see emulate capabilities
>> above). And some Capability Structures are 0 hardwired,
>> and others are passthrough.
>> 
>> This patch removes "switch" statements for emulation, and
>> introduces table based emulation derived from pciback
>> driver. You can implement new Capability Structure by
>> adding new table.
>> The other advantage of using this table is that you can
>> easily change the emulation policy of each field/bit by
>> just simply modifying the "emu_mask" value provided in
>> each register table.
>> And for only special emulation or interacting with other
>> components (like hypervisor), you have to implement
>> function corresponding to the register.
>> 
>> Signed-off-by: Yuji Shimada <shimada-yxb@necst.nec.co.jp>
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser wrote:
> What do you think of Yuji''s patch? One thing to consider
> is that perhaps the patch should be against the upstream
> qemu merge now. But I''m not sure that PCI passthrough is
> even supported yet in that tree (Ian?). 
> 
If we agree the basic policy is pass through except the ones with known
behavior, I think we don''t need that many case to case handle. Dexuan
is
working on the implementation base on the summit talk and close to end,
maybe Yuji and Dexuan can coordinate first to see if the proposed policy
can server yuji''s purpose.

Thx, eddie

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser writes ("Re: [Xen-devel] [PATCH][RFC] Support more Capability
Structures andDevice Specific"):
> What do you think of Yuji''s patch? One thing to consider is that
perhaps the
> patch should be against the upstream qemu merge now. But I''m not
sure that
> PCI passthrough is even supported yet in that tree (Ian?).
In theory PCI passthrough is supposed to be supported in my merged
tree.  But, I haven''t tested it yet so I doubt it works.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Dong, Eddie writes ("RE: [Xen-devel] [PATCH][RFC] Support more Capability
Structures andDevice Specific"):
> If we agree the basic policy is pass through except the ones with known
> behavior, I think we don''t need that many case to case handle.
Dexuan is
> working on the implementation base on the summit talk and close to end,
> maybe Yuji and Dexuan can coordinate first to see if the proposed policy
> can server yuji''s purpose.
Is it really safe to pass through operations with unknown behavious ?
Particularly if the system has an iommu, the administrator may be
expecting the passthrough mechanism to defend the host from rogue
behaviour by the card and its owning guest.

(I''m no expert on PCI so forgive me if this question is stupid.)

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hello,

diff -r 926a366ca82f tools/ioemu/list.h
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/tools/ioemu/list.h	Fri Jun 27 11:58:26 2008 +0900
@@ -0,0 +1,89 @@
+#ifndef _IOEMU_LIST_H
+#define _IOEMU_LIST_H
+/* Taken from Linux kernel code, but de-kernelized for userspace. */
+#include <stddef.h>

Could you use the list implementation of qemu instead please?

Samuel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Ian Jackson writes ("Re: [Xen-devel] [PATCH][RFC] Support more Capability
Structures andDevice Specific"):
> Keir Fraser writes ("Re: [Xen-devel] [PATCH][RFC] Support more
Capability Structures andDevice Specific"):
> > What do you think of Yuji''s patch? One thing to consider is
that
> > perhaps the patch should be against the upstream qemu merge
> > now. But I''m not sure that PCI passthrough is even supported
yet
> > in that tree (Ian?).
> 
> In theory PCI passthrough is supposed to be supported in my merged
> tree.  But, I haven''t tested it yet so I doubt it works.
I should expand on that.  I''d very much like for these kind of patches
to be going into the merged qemu tree now, and I''d also like people to
help test it.

So it would be very good if people who actually use PCI passthrough
were to take a look at the merged qemu tree.

Thanks,
Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

I think it is NOT safe to pass through operations with unknown
behaviour. qemu-dm should prevent guest software setting unsafe value
to register.  We have to investigate each register and decide to
emulate(virtualize) or passthrough.

I haven''t investigated some capability structures (like PCI-X
Capability Structure).  I hide them from guest software.

Device Specific Registers (exclude capability structures) is
passthrough. In non-virtualized environment, OS does not touch device
specific registers, but device drivers configure them. In virtualized
environment, we have to allow device drivers to configure them.

--
Yuji Shimada

On Fri, 27 Jun 2008 14:34:11 +0100
Ian Jackson <Ian.Jackson@eu.citrix.com> wrote:
> Dong, Eddie writes ("RE: [Xen-devel] [PATCH][RFC] Support more
Capability Structures andDevice Specific"):
> > If we agree the basic policy is pass through except the ones with
known
> > behavior, I think we don''t need that many case to case
handle. Dexuan is
> > working on the implementation base on the summit talk and close to
end,
> > maybe Yuji and Dexuan can coordinate first to see if the proposed
policy
> > can server yuji''s purpose.
> 
> Is it really safe to pass through operations with unknown behavious ?
> Particularly if the system has an iommu, the administrator may be
> expecting the passthrough mechanism to defend the host from rogue
> behaviour by the card and its owning guest.
> 
> (I''m no expert on PCI so forgive me if this question is stupid.)
> 
> Ian.
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Yuji,
I looked at the patch.  It seems pretty good. 
Except for the (temporary) absence of MSI/MSI-X stuff, looks the passthrough
policy in the patch is almost the same as what is discussed in the PDF file
Eddie posted.

I also made some tests against the patch, and found there may be some unstable
issues:
I.e., when I boot a 32e RHEL5u1 (I add the "pci=nomsi" parameter)), it
can easily (30%~80% probable) stay for a very long (i.e., >40s) at
"Starting udev:", and after I login in shell, the NIC seems not
present (the guest has no network available), but "lspci" shows the
NIC is there.
If I use the Qemu without your patch, the issue disappears at once, and NIC in
guest works well.

I haven''t found issue in your patch yet. :)

Thanks,
-- Dexuan


-----Original Message-----
From: xen-devel-bounces@lists.xensource.com
[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Yuji Shimada
Sent: 2008年6月30日 12:32
To: Ian Jackson
Cc: xen-devel@lists.xensource.com; Dong, Eddie; Keir Fraser
Subject: Re: [Xen-devel] [PATCH][RFC] Support more Capability
StructuresandDevice Specific

I think it is NOT safe to pass through operations with unknown
behaviour. qemu-dm should prevent guest software setting unsafe value
to register.  We have to investigate each register and decide to
emulate(virtualize) or passthrough.

I haven''t investigated some capability structures (like PCI-X
Capability Structure).  I hide them from guest software.

Device Specific Registers (exclude capability structures) is
passthrough. In non-virtualized environment, OS does not touch device
specific registers, but device drivers configure them. In virtualized
environment, we have to allow device drivers to configure them.

--
Yuji Shimada

On Fri, 27 Jun 2008 14:34:11 +0100
Ian Jackson <Ian.Jackson@eu.citrix.com> wrote:
> Dong, Eddie writes ("RE: [Xen-devel] [PATCH][RFC] Support more
Capability Structures andDevice Specific"):
> > If we agree the basic policy is pass through except the ones with
known
> > behavior, I think we don''t need that many case to case
handle. Dexuan is
> > working on the implementation base on the summit talk and close to
end,
> > maybe Yuji and Dexuan can coordinate first to see if the proposed
policy
> > can server yuji''s purpose.
> 
> Is it really safe to pass through operations with unknown behavious ?
> Particularly if the system has an iommu, the administrator may be
> expecting the passthrough mechanism to defend the host from rogue
> behaviour by the card and its owning guest.
> 
> (I''m no expert on PCI so forgive me if this question is stupid.)
> 
> Ian.
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Do you mean the list implementation inside tools/ioemu/audio/sys-queue.h
in xen-unstable tree ?
I haven''t checked the upstream qemu code.

Thanks.

--
Yuji Shimada
> Hello,
> 
> diff -r 926a366ca82f tools/ioemu/list.h
> --- /dev/null	Thu Jan 01 00:00:00 1970 +0000
> +++ b/tools/ioemu/list.h	Fri Jun 27 11:58:26 2008 +0900
> @@ -0,0 +1,89 @@
> +#ifndef _IOEMU_LIST_H
> +#define _IOEMU_LIST_H
> +/* Taken from Linux kernel code, but de-kernelized for userspace. */
> +#include <stddef.h>
> 
> Could you use the list implementation of qemu instead please?
> 
> Samuel
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

I think it is better to emulate registers related to error reporting
at least.

We can consider two patterns of error handling as following.

PATTERN-1: AER is enabled on host but will not be notified to guest domain.

Actual AER is enable, but guest software is not allowed to enable AER
by _OSC in guest firmware. When error occurs, dom0 will kill guest
domain and reset the device or bus. Registers related to error
reporting should be emulated to prevent guest software turning off
actual error reporting.

PATTERN-2: AER is enabled on host and will also be notified to guest domain.

Actual AER is always enable, but guest can enable/disable its virtual
AER. When error occurs, dom0 check guest emulated AER is enabled or not.
If guest emulated AER is enabled, dom0 will notify error to guest
software. Then guest software will reset I/O device or bus.
If guest emulated AER is disabled, dom0 will not notify error to
guest software. Dom0 will kill guest domain, and reset I/O device or
bus. To do this, registers related to error reporting should be
emulated. And Root Port emulation is also required.


I''ve taken a look at "Xen - VT-D Enhance.pdf".

Is Dexuan implementing Memory Mapped Configuration Access Mechanism to
support offset 256-4095? Are following interfaces not changed ?

    pci_dev->config_read
    pci_dev->config_write

If they are not changed, I think Dexuan''s Memory Mapped Configuration
Access Mechanism and my code can be merged easily.

Thanks a lot.

--
Yuji Shimada
> Keir Fraser wrote:
> > What do you think of Yuji''s patch? One thing to consider
> > is that perhaps the patch should be against the upstream
> > qemu merge now. But I''m not sure that PCI passthrough is
> > even supported yet in that tree (Ian?). 
> > 
> 
> If we agree the basic policy is pass through except the ones with known
> behavior, I think we don''t need that many case to case handle.
Dexuan is
> working on the implementation base on the summit talk and close to end,
> maybe Yuji and Dexuan can coordinate first to see if the proposed policy
> can server yuji''s purpose.
> 
> Thx, eddie
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

I am going to release these following patches.

    1. the patch supporting MSI Capability Structure for Xen Unstable.
       (list.h will be removed. some bugs will be fixed.)

    2. the patch supporting MSI-X Capability Structure for Xen Unstable.

    3. the patch against the upstream qemu merge (including MSI/MSI-X
       Capability Structure).

Should I release patch against the upstream qemu merge prior to Xen
Unstable ?
We''d be very happy if you could help me merged that patch, so that I
can
concentrate on MSI/MSI-X patches.

Thanks.

--
Yuji Shimada

On Fri, 27 Jun 2008 14:55:01 +0100
Ian Jackson <Ian.Jackson@eu.citrix.com> wrote:
> Ian Jackson writes ("Re: [Xen-devel] [PATCH][RFC] Support more
Capability Structures andDevice Specific"):
> > Keir Fraser writes ("Re: [Xen-devel] [PATCH][RFC] Support more
Capability Structures andDevice Specific"):
> > > What do you think of Yuji''s patch? One thing to consider
is that
> > > perhaps the patch should be against the upstream qemu merge
> > > now. But I''m not sure that PCI passthrough is even
supported yet
> > > in that tree (Ian?).
> > 
> > In theory PCI passthrough is supposed to be supported in my merged
> > tree.  But, I haven''t tested it yet so I doubt it works.
> 
> I should expand on that.  I''d very much like for these kind of
patches
> to be going into the merged qemu tree now, and I''d also like
people to
> help test it.
> 
> So it would be very good if people who actually use PCI passthrough
> were to take a look at the merged qemu tree.
> 
> Thanks,
> Ian.
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Dexuan,

I''ve tested my patch with CentOS 5.1 and PCI/PCIe NIC.  In my test
environment (with "pci=nomsi" set for Dom0 boot parameter), guest
OS can use the assigned NIC and can communicate with external machine.

Does guest OS recieve interrupt? You can check via /proc/interrupts.

Thanks.

--
Yuji Shimada
> Hi Yuji,
> I looked at the patch.  It seems pretty good. 
> Except for the (temporary) absence of MSI/MSI-X stuff, looks the
passthrough policy in the patch is almost the same as what is discussed in the
PDF file Eddie posted.
> 
> I also made some tests against the patch, and found there may be some
unstable issues:
> I.e., when I boot a 32e RHEL5u1 (I add the "pci=nomsi"
parameter)), it can easily (30%~80% probable) stay for a very long (i.e.,
>40s) at "Starting udev:", and after I login in shell, the NIC
seems not present (the guest has no network available), but "lspci"
shows the NIC is there.
> If I use the Qemu without your patch, the issue disappears at once, and NIC
in guest works well.
> 
> I haven''t found issue in your patch yet. :)
> 
> Thanks,
> -- Dexuan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

I didn''t work on the memory mapped access machinism.
I''m not sure whether adding that support is OK since now Qemu emulates
an old chipset which is actually PCIe-unaware.

Thanks,
-- Dexuan


-----Original Message-----
From: xen-devel-bounces@lists.xensource.com
[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Yuji Shimada
Sent: 2008年6月30日 15:15
To: Dong, Eddie
Cc: xen-devel@lists.xensource.com; Ian Jackson; Keir Fraser
Subject: Re: [Xen-devel] [PATCH][RFC] Support more Capability
StructuresandDevice Specific

I think it is better to emulate registers related to error reporting
at least.

We can consider two patterns of error handling as following.

PATTERN-1: AER is enabled on host but will not be notified to guest domain.

Actual AER is enable, but guest software is not allowed to enable AER
by _OSC in guest firmware. When error occurs, dom0 will kill guest
domain and reset the device or bus. Registers related to error
reporting should be emulated to prevent guest software turning off
actual error reporting.

PATTERN-2: AER is enabled on host and will also be notified to guest domain.

Actual AER is always enable, but guest can enable/disable its virtual
AER. When error occurs, dom0 check guest emulated AER is enabled or not.
If guest emulated AER is enabled, dom0 will notify error to guest
software. Then guest software will reset I/O device or bus.
If guest emulated AER is disabled, dom0 will not notify error to
guest software. Dom0 will kill guest domain, and reset I/O device or
bus. To do this, registers related to error reporting should be
emulated. And Root Port emulation is also required.


I''ve taken a look at "Xen - VT-D Enhance.pdf".

Is Dexuan implementing Memory Mapped Configuration Access Mechanism to
support offset 256-4095? Are following interfaces not changed ?

    pci_dev->config_read
    pci_dev->config_write

If they are not changed, I think Dexuan''s Memory Mapped Configuration
Access Mechanism and my code can be merged easily.

Thanks a lot.

--
Yuji Shimada
> Keir Fraser wrote:
> > What do you think of Yuji''s patch? One thing to consider
> > is that perhaps the patch should be against the upstream
> > qemu merge now. But I''m not sure that PCI passthrough is
> > even supported yet in that tree (Ian?). 
> > 
> 
> If we agree the basic policy is pass through except the ones with known
> behavior, I think we don''t need that many case to case handle.
Dexuan is
> working on the implementation base on the summit talk and close to end,
> maybe Yuji and Dexuan can coordinate first to see if the proposed policy
> can server yuji''s purpose.
> 
> Thx, eddie
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

I''m using x86_64 c/s 17888: 6ace85eb96c0, and assigning a 82541PI
Gigabit Etherer NIC to guest.
I also tried  "pci=nomsi" for Dom0, and the issus is still there. 
When the issue happens, eth0 doesn''t occur in /proc/interrupt though
the device driver module is loaded.
The issue doesn''t happen every time. Really strange...

Thanks,
-- Dexuan


-----Original Message-----
From: Yuji Shimada [mailto:shimada-yxb@necst.nec.co.jp] 
Sent: 2008年6月30日 16:15
To: Cui, Dexuan
Cc: Ian Jackson; xen-devel@lists.xensource.com; Dong, Eddie; Keir Fraser
Subject: Re: [Xen-devel] [PATCH][RFC] Support more Capability
StructuresandDevice Specific

Hi Dexuan,

I''ve tested my patch with CentOS 5.1 and PCI/PCIe NIC.  In my test
environment (with "pci=nomsi" set for Dom0 boot parameter), guest
OS can use the assigned NIC and can communicate with external machine.

Does guest OS recieve interrupt? You can check via /proc/interrupts.

Thanks.

--
Yuji Shimada
> Hi Yuji,
> I looked at the patch.  It seems pretty good. 
> Except for the (temporary) absence of MSI/MSI-X stuff, looks the
passthrough policy in the patch is almost the same as what is discussed in the
PDF file Eddie posted.
> 
> I also made some tests against the patch, and found there may be some
unstable issues:
> I.e., when I boot a 32e RHEL5u1 (I add the "pci=nomsi"
parameter)), it can easily (30%~80% probable) stay for a very long (i.e.,
>40s) at "Starting udev:", and after I login in shell, the NIC
seems not present (the guest has no network available), but "lspci"
shows the NIC is there.
> If I use the Qemu without your patch, the issue disappears at once, and NIC
in guest works well.
> 
> I haven''t found issue in your patch yet. :)
> 
> Thanks,
> -- Dexuan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Yuji Shimada, le Mon 30 Jun 2008 16:12:31 +0900, a écrit
:
> Do you mean the list implementation inside tools/ioemu/audio/sys-queue.h
> in xen-unstable tree ?
Yes.
> I haven''t checked the upstream qemu code.
It''s the same, with an additional QEMU prefix.

Samuel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Yuji Shimada writes ("Re: [Xen-devel] [PATCH][RFC] Support more Capability
Structures andDevice Specific"):
> I am going to release these following patches.
...
> Should I release patch against the upstream qemu merge prior to Xen
> Unstable ?
We''re currently working to integrate the upstream qemu merge into
patchman.  I hope that we can be successful with this, so that we can
switch to this tree as the default in 3.3.

With that in mind:

Have you looked at the upstream merge at all yet ?  I haven''t yet done
any PCI passthrough testing and I think it''s very important that we do
some of that testing.  If you have any time available to do some
checking of the qemu upstream merge tree, even before we consider your
recent patches, that would be very helpful.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Ian Jackson wrote:
> Dong, Eddie writes ("RE: [Xen-devel] [PATCH][RFC] Support
> more Capability Structures andDevice Specific"): 
>> If we agree the basic policy is pass through except the
>> ones with known behavior, I think we don''t need that
>> many case to case handle. Dexuan is working on the
>> implementation base on the summit talk and close to end,
>> maybe Yuji and Dexuan can coordinate first to see if the
>> proposed policy can server yuji''s purpose. 
> 
> Is it really safe to pass through operations with unknown
> behavious ? 
> Particularly if the system has an iommu, the
> administrator may be 
> expecting the passthrough mechanism to defend the host
> from rogue 
> behaviour by the card and its owning guest.

What kind of operations in your mind will hurt host?
But yes, guest may not work properly for some cases such as some vidoe
card I mentioned in summit which may map host address to internal
register and used by drivers. For those kind of devices, CP will just
disable assignment thru kind of assignable check such as blacklist.

Other than that, I didn''t see how pass through will make things worse.
Can u specify ? All guest access to memory is protected by IOMMU and
thus no imapct to host. 


Thx, eddie

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

The PCI config space passthrough is actually crucial for Xen 3.3.

We have already found many device assignment issues caused by the lack of the
passthrough (and I think more issues will be found).
For example, the assigned BroadCom NIC can''t work because the driver
needs to access the register BNX2_PCICFG_INT_ACK_CMD (at 0x84); another example:
the USB assignment doesn''t work -- one of the reasons is: UHCI-hc needs
to access config space reigster USB_LEGKEY(at 0xC0-C1) to enable USB IRQ.

So it will be great to push the patch into xen-unstable tree.

Thanks,
-- Dexuan


-----Original Message-----
From: xen-devel-bounces@lists.xensource.com
[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Ian Jackson
Sent: 2008年7月1日 0:50
To: Yuji Shimada
Cc: xen-devel@lists.xensource.com; Dong, Eddie; Keir Fraser
Subject: Re: [Xen-devel] [PATCH][RFC] Support more Capability
StructuresandDevice Specific

Yuji Shimada writes ("Re: [Xen-devel] [PATCH][RFC] Support more Capability
Structures andDevice Specific"):
> I am going to release these following patches.
...
> Should I release patch against the upstream qemu merge prior to Xen
> Unstable ?
We''re currently working to integrate the upstream qemu merge into
patchman.  I hope that we can be successful with this, so that we can
switch to this tree as the default in 3.3.

With that in mind:

Have you looked at the upstream merge at all yet ?  I haven''t yet done
any PCI passthrough testing and I think it''s very important that we do
some of that testing.  If you have any time available to do some
checking of the qemu upstream merge tree, even before we consider your
recent patches, that would be very helpful.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Yuji Shimada wrote:
> I think it is NOT safe to pass through operations with
> unknown behaviour. qemu-dm should prevent guest software
This used to be the reason why we only pass through CMD register, but we
then suffer from that in different devices and I have to admit, without
passthrough real setting, the device won''t function correctly in many
cases.

Do u have any real data ? Device memory access is fine with IOMMU, irq
vector is virtualized now.
> setting unsafe value to register.  We have to investigate
> each register and decide to emulate(virtualize) or
> passthrough. 
Investigation is defintely good, eventually we need to know all
configuration registers, but even with that, there are still device
specifc registers we have to deal such as Vendor specific capability and
those registers directly defined by devices (not a standard PCI
capability). But hidding settings due to the reason we didn''t
investigate yet will simply make things worse, and we already observed
this with more devices tested such as UHCI mouse etc.

> 
> I haven''t investigated some capability structures (like
> PCI-X Capability Structure).  I hide them from guest
> software. 
ditto
> 
> Device Specific Registers (exclude capability structures)
> is passthrough. In non-virtualized environment, OS does
> not touch device specific registers, but device drivers
> configure them. In virtualized environment, we have to
> allow device drivers to configure them. 
We need to pass through except the device is doing IRQ vector setting
which can''t be handled in either case, those devices are simply not
assignable.

Thanks, eddie

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

In my patch, registers inside Vendor Specific Capability Structure
(Capability ID is 09h) are all passthrough, except Next Capability
Pointer Register.

Registers area defined by device are not belonging to Capability
Structure, and they are all passthrough.

Do you mean non standard Capability Structure should be passthrough?


For PCI-X Capability Structure, we have to emulate Function Number
Field, Device Number Field, and Bus Number Field in PCI-X Status
Register at least. The reason is that the value for host and guest
maybe different.

Additionally, I think we have to emulate Maximum Outstanding Split
Transactions Field in PCI-X Command Register too. The reason is that
the real value should be decided by considering all over the system,
and we should also prevent guest software writing value.

Thanks.

--
Yuji Shimada

On Tue, 1 Jul 2008 10:27:02 +0800
"Dong, Eddie" <eddie.dong@intel.com> wrote:
> Yuji Shimada wrote:
> > I think it is NOT safe to pass through operations with
> > unknown behaviour. qemu-dm should prevent guest software
> 
> This used to be the reason why we only pass through CMD register, but we
> then suffer from that in different devices and I have to admit, without
> passthrough real setting, the device won''t function correctly in
many
> cases.
> 
> Do u have any real data ? Device memory access is fine with IOMMU, irq
> vector is virtualized now.
> 
> > setting unsafe value to register.  We have to investigate
> > each register and decide to emulate(virtualize) or
> > passthrough. 
> 
> Investigation is defintely good, eventually we need to know all
> configuration registers, but even with that, there are still device
> specifc registers we have to deal such as Vendor specific capability and
> those registers directly defined by devices (not a standard PCI
> capability). But hidding settings due to the reason we didn''t
> investigate yet will simply make things worse, and we already observed
> this with more devices tested such as UHCI mouse etc.
> 
> 
> > 
> > I haven''t investigated some capability structures (like
> > PCI-X Capability Structure).  I hide them from guest
> > software. 
> 
> ditto
> 
> > 
> > Device Specific Registers (exclude capability structures)
> > is passthrough. In non-virtualized environment, OS does
> > not touch device specific registers, but device drivers
> > configure them. In virtualized environment, we have to
> > allow device drivers to configure them. 
> 
> We need to pass through except the device is doing IRQ vector setting
> which can''t be handled in either case, those devices are simply
not
> assignable.
> 
> Thanks, eddie
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Yuji Shimada writes ("Re: [Xen-devel] [PATCH][RFC] Support more Capability
Structures andDevice Specific"):
> In my patch, registers inside Vendor Specific Capability Structure
> (Capability ID is 09h) are all passthrough, except Next Capability
> Pointer Register.
My worry is that a device may say, in its vendor-specific
register-level programming documentation for these configurations,
something like:

  Do _not_ set USE_EXTERNAL_INPUT and USE_INTERNAL_INPUT
  simultaneously; this may cause damage to the Gnomovision PCI
  card and may also cause the Gnomovision PCI card to draw
  excessive current from the host power supply.

Or

  Do _not_ use the UPLOAD_FIRMWARE_* configuration.  These are for use
  by the approved Gnomovision firmware loader only.  Uploading bad
  firmware may cause damage [etc. etc.]

I haven''t read many modern PCI card specs but with the constant
shifting of functionality (even functionality which is intended to
preserve hardware integrity) to software and firmware, I would be wary
of assuming that every unknown PCI card has no register and
configuration settings which can cause hardware damage or other kinds
of unexpected and undesirable events.

If there is there a requirement written into the general PCI
specification that this won''t happen, then fine - if so please quote
chapter and verse.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> I haven''t read many modern PCI card specs but with the
> constant 
> shifting of functionality (even functionality which is
> intended to 
> preserve hardware integrity) to software and firmware, I
> would be wary 
> of assuming that every unknown PCI card has no register
> and 
> configuration settings which can cause hardware damage or
> other kinds 
> of unexpected and undesirable events.
For those vendor specific registers (in PCI cap or not), we always have
dillemma: pass through with functionality or discard with malfunction.
If we pass through them, then the opposite card are in black list that
we can''t support, if we discard them, the former class of card becomes
un-assignable.

Per current data, pass through get many known bug fixed as the case
Dexuan mentioned. But we didn''t see a HW damaging host. Some know issue
could be a device issuing tons of PCIe traffic, absorbing extra power,
issuing interrupt storm etc, but right now we didn''t see issues yet.
> 
> If there is there a requirement written into the general
> PCI 
> specification that this won''t happen, then fine - if so
> please quote 
> chapter and verse.
> 
Defintely if PCIe spec can be modified to support this one is best, but
we still have legacy devices.

Thx, eddie


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

I''ve done some bug fixes as follows.

1. correct the size calculation of MSI Capability Structure in
   pt_msi_size_init(). The next capability might be hidden due to wrong
   large size of MSI.

2. modify the decision logic for determining unused Exp ROM BAR in
   pt_bar_reg_parse(). Use PCIIORegion table instead of parsing
   BAR itself.

3. bug fix on .size_init func for PCI Express Capability Structure
   in pt_emu_reg_grp_tbl[].
   (pt_vendor_size_init ---> pt_reg_grp_size_init)

4. small bug fix on the decision logic for checking unused BAR in
   pt_pci_write_config().

5. add printf message to show overlapped device in pt_chk_bar_overlap().

6. modify pt_bar_mapping() to prevent guest software mapping memory
   resource to 00000000h

7. modify pt_bar_mapping() to map resource even if overlapping is
   detected.

I''ve tested my patch with CentOS 5.1 and PCI/PCIe NIC.  Without
"pci=nomsi", guest OS can use the assigned NIC and can communicate
with external machine.

Additionally I assigned UHCI Controller to guest domain. Guest OS can
use USB-HDD and USB-Mouse.

Could you test the patch?


I am going to remove list.h and enable MSI.

Thanks.

Signed-off-by: Yuji Shimada <shimada-yxb@necst.nec.co.jp>

--
Yuji Shimada

On Mon, 30 Jun 2008 17:29:38 +0800
"Cui, Dexuan" <dexuan.cui@intel.com> wrote:
> I''m using x86_64 c/s 17888: 6ace85eb96c0, and assigning a 82541PI
Gigabit Etherer NIC to guest.
> I also tried  "pci=nomsi" for Dom0, and the issus is still there.
> When the issue happens, eth0 doesn''t occur in /proc/interrupt
though the device driver module is loaded.
> The issue doesn''t happen every time. Really strange...
> 
> Thanks,
> -- Dexuan
> 
> 
> -----Original Message-----
> From: Yuji Shimada [mailto:shimada-yxb@necst.nec.co.jp] 
> Sent: 2008夏・花可30科苛 16:15
> To: Cui, Dexuan
> Cc: Ian Jackson; xen-devel@lists.xensource.com; Dong, Eddie; Keir Fraser
> Subject: Re: [Xen-devel] [PATCH][RFC] Support more Capability
StructuresandDevice Specific
> 
> Hi Dexuan,
> 
> I''ve tested my patch with CentOS 5.1 and PCI/PCIe NIC.  In my test
> environment (with "pci=nomsi" set for Dom0 boot parameter), guest
> OS can use the assigned NIC and can communicate with external machine.
> 
> Does guest OS recieve interrupt? You can check via /proc/interrupts.
> 
> Thanks.
> 
> --
> Yuji Shimada
> 
> > Hi Yuji,
> > I looked at the patch.  It seems pretty good. 
> > Except for the (temporary) absence of MSI/MSI-X stuff, looks the
passthrough policy in the patch is almost the same as what is discussed in the
PDF file Eddie posted.
> > 
> > I also made some tests against the patch, and found there may be some
unstable issues:
> > I.e., when I boot a 32e RHEL5u1 (I add the "pci=nomsi"
parameter)), it can easily (30%~80% probable) stay for a very long (i.e.,
>40s) at "Starting udev:", and after I login in shell, the NIC
seems not present (the guest has no network available), but "lspci"
shows the NIC is there.
> > If I use the Qemu without your patch, the issue disappears at once,
and NIC in guest works well.
> > 
> > I haven''t found issue in your patch yet. :)
> > 
> > Thanks,
> > -- Dexuan
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

I had a test with your new patch: the occasional-NIC-doesn''t-work issue
I noticed disappeared.
I''ll have a look why it disappeared.  :-)

Thanks,
-- Dexuan


-----Original Message-----
From: Yuji Shimada [mailto:shimada-yxb@necst.nec.co.jp] 
Sent: 2008年7月2日 9:03
To: Cui, Dexuan
Cc: Ian Jackson; xen-devel@lists.xensource.com; Dong, Eddie; Keir Fraser
Subject: Re: [Xen-devel] [PATCH][RFC] Support more Capability
StructuresandDevice Specific

I''ve done some bug fixes as follows.

1. correct the size calculation of MSI Capability Structure in
   pt_msi_size_init(). The next capability might be hidden due to wrong
   large size of MSI.

2. modify the decision logic for determining unused Exp ROM BAR in
   pt_bar_reg_parse(). Use PCIIORegion table instead of parsing
   BAR itself.

3. bug fix on .size_init func for PCI Express Capability Structure
   in pt_emu_reg_grp_tbl[].
   (pt_vendor_size_init ---> pt_reg_grp_size_init)

4. small bug fix on the decision logic for checking unused BAR in
   pt_pci_write_config().

5. add printf message to show overlapped device in pt_chk_bar_overlap().

6. modify pt_bar_mapping() to prevent guest software mapping memory
   resource to 00000000h

7. modify pt_bar_mapping() to map resource even if overlapping is
   detected.

I''ve tested my patch with CentOS 5.1 and PCI/PCIe NIC.  Without
"pci=nomsi", guest OS can use the assigned NIC and can communicate
with external machine.

Additionally I assigned UHCI Controller to guest domain. Guest OS can
use USB-HDD and USB-Mouse.

Could you test the patch?


I am going to remove list.h and enable MSI.

Thanks.

Signed-off-by: Yuji Shimada <shimada-yxb@necst.nec.co.jp>

--
Yuji Shimada

On Mon, 30 Jun 2008 17:29:38 +0800
"Cui, Dexuan" <dexuan.cui@intel.com> wrote:
> I''m using x86_64 c/s 17888: 6ace85eb96c0, and assigning a 82541PI
Gigabit Etherer NIC to guest.
> I also tried  "pci=nomsi" for Dom0, and the issus is still there.
> When the issue happens, eth0 doesn''t occur in /proc/interrupt
though the device driver module is loaded.
> The issue doesn''t happen every time. Really strange...
> 
> Thanks,
> -- Dexuan
> 
> 
> -----Original Message-----
> From: Yuji Shimada [mailto:shimada-yxb@necst.nec.co.jp] 
> Sent: 2008夏・花可30科苛 16:15
> To: Cui, Dexuan
> Cc: Ian Jackson; xen-devel@lists.xensource.com; Dong, Eddie; Keir Fraser
> Subject: Re: [Xen-devel] [PATCH][RFC] Support more Capability
StructuresandDevice Specific
> 
> Hi Dexuan,
> 
> I''ve tested my patch with CentOS 5.1 and PCI/PCIe NIC.  In my test
> environment (with "pci=nomsi" set for Dom0 boot parameter), guest
> OS can use the assigned NIC and can communicate with external machine.
> 
> Does guest OS recieve interrupt? You can check via /proc/interrupts.
> 
> Thanks.
> 
> --
> Yuji Shimada
> 
> > Hi Yuji,
> > I looked at the patch.  It seems pretty good. 
> > Except for the (temporary) absence of MSI/MSI-X stuff, looks the
passthrough policy in the patch is almost the same as what is discussed in the
PDF file Eddie posted.
> > 
> > I also made some tests against the patch, and found there may be some
unstable issues:
> > I.e., when I boot a 32e RHEL5u1 (I add the "pci=nomsi"
parameter)), it can easily (30%~80% probable) stay for a very long (i.e.,
>40s) at "Starting udev:", and after I login in shell, the NIC
seems not present (the guest has no network available), but "lspci"
shows the NIC is there.
> > If I use the Qemu without your patch, the issue disappears at once,
and NIC in guest works well.
> > 
> > I haven''t found issue in your patch yet. :)
> > 
> > Thanks,
> > -- Dexuan
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Dong, Eddie writes ("RE: [Xen-devel] [PATCH][RFC] Support more Capability
Structures andDevice Specific"):
> Per current data, pass through get many known bug fixed as the case
> Dexuan mentioned. But we didn''t see a HW damaging host. Some know
issue
> could be a device issuing tons of PCIe traffic, absorbing extra power,
> issuing interrupt storm etc, but right now we didn''t see issues
yet.
Most people doing PCI passthrough appear to be under the impression
that the guest cannot escape and cannot damage the host.  (Even those
currently doing PCI passthrough with current production hardware
without an iommu!)

I think it is fine to have a passthrough option which doesn''t properly
protect the host from the guest - this is a useful setup in many
situations.  But it should not be enabled by default, surely ?

Note that this is a _security_ problem.  So `data'' about
`issues''
which you have `seen'' is irrelevant.  Just because you haven''t
actually observed any misbehaviour with non-malicious guests doesn''t
mean that a malicious guest couldn''t cause the hardware to melt.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> I think it is fine to have a passthrough option which doesn''t
properly
> protect the host from the guest - this is a useful setup in many
> situations.  But it should not be enabled by default, surely ?
Agreed entirely. Note also that some implementations of an IOMMU will not
save you as they don''t fence between individual PCI devices (PCIE is
obviously a bit easier). Not fencing between devices allows you for
example to use a fairly flexible SCSI controller to reprogram another
device. 

In the general case there are also some really nasty dirty attacks you
can''t stop with an IOMMU one of which is to reflash the BIOS of the
graphics card to which you were given unrestricted access so that you
compromise the entire system next boot. These attacks appear well
understood except by IOMMU marketing people ;)

IOMMU is great for system correctness and flexibility, using it for
safely providing hardware direct access is a very very hairy business with
a complex device.

Alan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Ian Jackson wrote:
> Dong, Eddie writes ("RE: [Xen-devel] [PATCH][RFC] Support
> more Capability Structures andDevice Specific"): 
>> Per current data, pass through get many known bug fixed
>> as the case Dexuan mentioned. But we didn''t see a HW
>> damaging host. Some know issue could be a device issuing
>> tons of PCIe traffic, absorbing extra power, issuing
>> interrupt storm etc, but right now we didn''t see issues
>> yet.  
> 
> Most people doing PCI passthrough appear to be under the
> impression 
> that the guest cannot escape and cannot damage the host. 
> (Even those 
> currently doing PCI passthrough with current production
> hardware 
> without an iommu!)
What I am aware is only QoS, I didn''t know how can a guest program the
device to crash host. Interrupt storm can be blocked by hypervisor at
certain situation. Competing for unnecessary PCIe traffic is never
related to if we pass through guest setting or not. Can you give me a
specific example how host will be crashed?
> 
> I think it is fine to have a passthrough option which
> doesn''t properly 
> protect the host from the guest - this is a useful setup
> in many 
> situations.  But it should not be enabled by default,
> surely ? 
Same reason as above.
> 
> Note that this is a _security_ problem.  So `data'' about
> `issues'' 
> which you have `seen'' is irrelevant.  Just because you
> haven''t 
> actually observed any misbehaviour with non-malicious
> guests doesn''t 
> mean that a malicious guest couldn''t cause the hardware
> to melt. 
Examples even in theory?
NOTE here, current pass through logic only support devices under root
port.
> 
> Ian.
Thx, eddie

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Alan Cox wrote:
>> I think it is fine to have a passthrough option which
>> doesn''t properly protect the host from the guest - this
>> is a useful setup in many situations.  But it should not
>> be enabled by default, surely ? 
> 
> Agreed entirely. Note also that some implementations of
> an IOMMU will not save you as they don''t fence between
> individual PCI devices (PCIE is obviously a bit easier).
IOMMU, at least Intel''s IOMMU, doesn''t support pure PCI
device, only
PCIe devices can be DMA protected.
> Not fencing between devices allows you for example to use
> a fairly flexible SCSI controller to reprogram another
> device. 
Again, at least for Intel IOMMU, devices under root endpoint can never
escape from IOMMU DMA protection, right now we don''t support PCIe
devices under a switch to do assignement, but with future ATS or ACS is
implemented, we can assign devices under a switch, where ether the
switch disable peer to peer transaction or always pass up
"untranslated"
traffic to upstream.

So your concern is a not real IMO, not? Or do u mean AMD IOMMU may have
different implementation? 
> 
> In the general case there are also some really nasty
> dirty attacks you can''t stop with an IOMMU one of which
> is to reflash the BIOS of the graphics card to which you
> were given unrestricted access so that you compromise the
> entire system next boot. These attacks appear well
> understood except by IOMMU marketing people ;) 
Same with above, this is already protected by IOMMU, peer to peer DMA is
not supported right now.
> 
> IOMMU is great for system correctness and flexibility,
> using it for safely providing hardware direct access is a
> very very hairy business with a complex device.
> 
Agree, that is why we are here :)

Thx, eddie


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Acked-by: Eddie Dong <eddie.dong@intel.com>

Yuji Shimada wrote:
> I''ve done some bug fixes as follows.
> 
> 1. correct the size calculation of MSI Capability
>    Structure in pt_msi_size_init(). The next capability
>    might be hidden due to wrong large size of MSI.
> 
> 2. modify the decision logic for determining unused Exp
>    ROM BAR in pt_bar_reg_parse(). Use PCIIORegion table
>    instead of parsing BAR itself.
> 
> 3. bug fix on .size_init func for PCI Express Capability
>    Structure in pt_emu_reg_grp_tbl[].
>    (pt_vendor_size_init ---> pt_reg_grp_size_init)
> 
> 4. small bug fix on the decision logic for checking
>    unused BAR in pt_pci_write_config().
> 
> 5. add printf message to show overlapped device in
> pt_chk_bar_overlap(). 
> 
> 6. modify pt_bar_mapping() to prevent guest software
>    mapping memory resource to 00000000h
> 
> 7. modify pt_bar_mapping() to map resource even if
>    overlapping is detected.
> 
> I''ve tested my patch with CentOS 5.1 and PCI/PCIe NIC. 
> Without "pci=nomsi", guest OS can use the assigned NIC
> and can communicate 
> with external machine.
> 
> Additionally I assigned UHCI Controller to guest domain.
> Guest OS can use USB-HDD and USB-Mouse.
> 
> Could you test the patch?
> 
> 
> I am going to remove list.h and enable MSI.
> 
> Thanks.
> 
> Signed-off-by: Yuji Shimada <shimada-yxb@necst.nec.co.jp>
> 
> 
>> I''m using x86_64 c/s 17888: 6ace85eb96c0, and assigning
>> a 82541PI Gigabit Etherer NIC to guest. 
>> I also tried  "pci=nomsi" for Dom0, and the issus is
>> still there. 
>> When the issue happens, eth0 doesn''t occur in
>> /proc/interrupt though the device driver module is
>> loaded. The issue doesn''t happen every time. Really
>> strange...  
>> 
>> Thanks,
>> -- Dexuan
>> 
>> 
>> -----Original Message-----
>> From: Yuji Shimada [mailto:shimada-yxb@necst.nec.co.jp]
>> Sent: 2008夏・花可30科苛 16:15
>> To: Cui, Dexuan
>> Cc: Ian Jackson; xen-devel@lists.xensource.com; Dong,
>> Eddie; Keir Fraser 
>> Subject: Re: [Xen-devel] [PATCH][RFC] Support more
>> Capability StructuresandDevice Specific 
>> 
>> Hi Dexuan,
>> 
>> I''ve tested my patch with CentOS 5.1 and PCI/PCIe NIC. 
>> In my test 
>> environment (with "pci=nomsi" set for Dom0 boot
>> parameter), guest 
>> OS can use the assigned NIC and can communicate with
>> external machine. 
>> 
>> Does guest OS recieve interrupt? You can check via
>> /proc/interrupts. 
>> 
>> Thanks.
>> 
>> --
>> Yuji Shimada
>> 
>>> Hi Yuji,
>>> I looked at the patch.  It seems pretty good.
>>> Except for the (temporary) absence of MSI/MSI-X stuff,
>>> looks the passthrough policy in the patch is almost the
>>> same as what is discussed in the PDF file Eddie posted.
>>> 
>>> I also made some tests against the patch, and found
>>> there may be some unstable issues: 
>>> I.e., when I boot a 32e RHEL5u1 (I add the "pci=nomsi"
>>> parameter)), it can easily (30%~80% probable) stay for
>>> a very long (i.e., >40s) at "Starting udev:", and
after
>>> I login in shell, the NIC seems not present (the guest
>>> has no network available), but "lspci" shows the NIC is
>>> there. If I use the Qemu without your patch, the issue
>>> disappears at once, and NIC in guest works well.     
>>> 
>>> I haven''t found issue in your patch yet. :)
>>> 
>>> Thanks,
>>> -- Dexuan
>> 
>> 
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Dong, Eddie writes ("RE: [Xen-devel] [PATCH][RFC] Support more Capability
Structures andDevice Specific"):
> Alan Cox wrote:
> > In the general case there are also some really nasty
> > dirty attacks you can''t stop with an IOMMU one of which
> > is to reflash the BIOS of the graphics card to which you
> > were given unrestricted access so that you compromise the
> > entire system next boot. These attacks appear well
> > understood except by IOMMU marketing people ;) 
> 
> Same with above, this is already protected by IOMMU, peer to peer DMA is
> not supported right now.
You have evidently completely misunderstood Alan''s point.

I was going to explain it again but I''m not sure I know how to say it
more clearly.  Alan''s scenario doesn''t involve any peer to
peer DMA.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Ian Jackson wrote:
> Dong, Eddie writes ("RE: [Xen-devel] [PATCH][RFC] Support
> more Capability Structures andDevice Specific"): 
>> Alan Cox wrote:
>>> In the general case there are also some really nasty
>>> dirty attacks you can''t stop with an IOMMU one of which
>>> is to reflash the BIOS of the graphics card to which you
>>> were given unrestricted access so that you compromise
>>> the entire system next boot. These attacks appear well
>>> understood except by IOMMU marketing people ;)
>> 
>> Same with above, this is already protected by IOMMU,
>> peer to peer DMA is not supported right now.
> 
> You have evidently completely misunderstood Alan''s point.
> 
> I was going to explain it again but I''m not sure I know
> how to say it 
> more clearly.  Alan''s scenario doesn''t involve any peer
> to peer DMA. 
> 
> Ian.
> 
Ok, if it means guest direct MMIO to flash bios, then yes. But it is not
related with our discussion, i.e. no matter we pass through CFGS
registers or not, it may happen.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel