Yuji Shimada
2008-Jun-27 07:38 UTC
[Xen-devel] [PATCH][RFC] Support more Capability Structures and Device Specific
I am submitting the patch which supports more Capability Structures and Device Specific Registers for passthrough device. In Xen 3.3 unstable, qemu-dm supports Configuration Header, MSI Capability Structure, and MSI-X Capability Structure. But qemu-dm does not support PCI Express Capability Structure, Device Specific Registers, etc (writing them is ignored). To support various I/O devices, I implemented following Capability Structures and Device Specific Registers. * Configuration Header Type 0 -> emulation. "emulation" does not mean no accessing real I/O device. Access real I/O device, but guest value and real value might be different. * PCI Express Capability Structure -> emulation. * PCI Power Management Capability Structure -> emulation. * Vital Product Data Capability Structure -> emulation (almost passthrough). * Vendor Specific Capability Structure -> emulation (almost passthrough). * Device Specific Register (exclude capability structures) -> passthrough. The device drivers in guest domain are allowed to access Device Specific Register. So various I/O device will work. Currently MSI Capability Structure and MSI-X Capability Structure is not implemented, and they are hidden from guest software. I disabled MSI and MSI-X in qemu-dm temporary. I am implementing MSI Capability Structure and merging current MSI routines. I will release the patch if you agree with me. MSI-X will be after MSI. I will be very happy if anyone can help me. Other Capability Structures are hidden from guest software. To do this, I change Next Capability Pointer''s value to point only the Capability Structure that need to be exported to guest software (see emulate capabilities above). And some Capability Structures are 0 hardwired, and others are passthrough. This patch removes "switch" statements for emulation, and introduces table based emulation derived from pciback driver. You can implement new Capability Structure by adding new table. The other advantage of using this table is that you can easily change the emulation policy of each field/bit by just simply modifying the "emu_mask" value provided in each register table. And for only special emulation or interacting with other components (like hypervisor), you have to implement function corresponding to the register. Signed-off-by: Yuji Shimada <shimada-yxb@necst.nec.co.jp> -- Yuji Shimada _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Dong, Eddie
2008-Jun-27 10:14 UTC
RE: [Xen-devel] [PATCH][RFC] Support more Capability Structures andDevice Specific
Yuji: We have a discussion in xen summit for PCI CFGS emulation, are you in the summit too? Here is the slide for the discussion, we can coordinate. Thx, eddie Yuji Shimada wrote:> I am submitting the patch which supports more Capability > Structures and Device Specific Registers for passthrough > device. > > In Xen 3.3 unstable, qemu-dm supports Configuration > Header, MSI Capability Structure, and MSI-X Capability > Structure. But qemu-dm does not support PCI Express > Capability Structure, Device Specific Registers, etc > (writing them is ignored). > > To support various I/O devices, I implemented following > Capability Structures and Device Specific Registers. > > * Configuration Header Type 0 > -> emulation. > "emulation" does not mean no accessing real > I/O device. Access real I/O device, but guest > value and real value might be different. > * PCI Express Capability Structure > -> emulation. > * PCI Power Management Capability Structure > -> emulation. > * Vital Product Data Capability Structure > -> emulation (almost passthrough). > * Vendor Specific Capability Structure > -> emulation (almost passthrough). > * Device Specific Register (exclude capability > structures) -> passthrough. > The device drivers in guest domain are allowed > to access Device Specific Register. So various > I/O device will work. > > Currently MSI Capability Structure and MSI-X Capability > Structure is not implemented, and they are hidden from > guest software. I disabled MSI and MSI-X in qemu-dm > temporary. I am implementing MSI Capability Structure and > merging current MSI routines. I will release the patch if > you agree with me. > > MSI-X will be after MSI. I will be very happy if anyone > can help me. > > Other Capability Structures are hidden from guest > software. To do this, I change Next Capability Pointer''s > value to point only the Capability Structure that need to > be exported to guest software (see emulate capabilities > above). And some Capability Structures are 0 hardwired, > and others are passthrough. > > This patch removes "switch" statements for emulation, and > introduces table based emulation derived from pciback > driver. You can implement new Capability Structure by > adding new table. > The other advantage of using this table is that you can > easily change the emulation policy of each field/bit by > just simply modifying the "emu_mask" value provided in > each register table. > And for only special emulation or interacting with other > components (like hypervisor), you have to implement > function corresponding to the register. > > Signed-off-by: Yuji Shimada <shimada-yxb@necst.nec.co.jp>_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2008-Jun-27 10:19 UTC
Re: [Xen-devel] [PATCH][RFC] Support more Capability Structures andDevice Specific
What do you think of Yuji''s patch? One thing to consider is that perhaps the patch should be against the upstream qemu merge now. But I''m not sure that PCI passthrough is even supported yet in that tree (Ian?). -- Keir On 27/6/08 11:14, "Dong, Eddie" <eddie.dong@intel.com> wrote:> Yuji: > We have a discussion in xen summit for PCI CFGS emulation, are > you in the summit too? > Here is the slide for the discussion, we can coordinate. > Thx, eddie > > Yuji Shimada wrote: >> I am submitting the patch which supports more Capability >> Structures and Device Specific Registers for passthrough >> device. >> >> In Xen 3.3 unstable, qemu-dm supports Configuration >> Header, MSI Capability Structure, and MSI-X Capability >> Structure. But qemu-dm does not support PCI Express >> Capability Structure, Device Specific Registers, etc >> (writing them is ignored). >> >> To support various I/O devices, I implemented following >> Capability Structures and Device Specific Registers. >> >> * Configuration Header Type 0 >> -> emulation. >> "emulation" does not mean no accessing real >> I/O device. Access real I/O device, but guest >> value and real value might be different. >> * PCI Express Capability Structure >> -> emulation. >> * PCI Power Management Capability Structure >> -> emulation. >> * Vital Product Data Capability Structure >> -> emulation (almost passthrough). >> * Vendor Specific Capability Structure >> -> emulation (almost passthrough). >> * Device Specific Register (exclude capability >> structures) -> passthrough. >> The device drivers in guest domain are allowed >> to access Device Specific Register. So various >> I/O device will work. >> >> Currently MSI Capability Structure and MSI-X Capability >> Structure is not implemented, and they are hidden from >> guest software. I disabled MSI and MSI-X in qemu-dm >> temporary. I am implementing MSI Capability Structure and >> merging current MSI routines. I will release the patch if >> you agree with me. >> >> MSI-X will be after MSI. I will be very happy if anyone >> can help me. >> >> Other Capability Structures are hidden from guest >> software. To do this, I change Next Capability Pointer''s >> value to point only the Capability Structure that need to >> be exported to guest software (see emulate capabilities >> above). And some Capability Structures are 0 hardwired, >> and others are passthrough. >> >> This patch removes "switch" statements for emulation, and >> introduces table based emulation derived from pciback >> driver. You can implement new Capability Structure by >> adding new table. >> The other advantage of using this table is that you can >> easily change the emulation policy of each field/bit by >> just simply modifying the "emu_mask" value provided in >> each register table. >> And for only special emulation or interacting with other >> components (like hypervisor), you have to implement >> function corresponding to the register. >> >> Signed-off-by: Yuji Shimada <shimada-yxb@necst.nec.co.jp> > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Dong, Eddie
2008-Jun-27 10:25 UTC
RE: [Xen-devel] [PATCH][RFC] Support more Capability Structures andDevice Specific
Keir Fraser wrote:> What do you think of Yuji''s patch? One thing to consider > is that perhaps the patch should be against the upstream > qemu merge now. But I''m not sure that PCI passthrough is > even supported yet in that tree (Ian?). >If we agree the basic policy is pass through except the ones with known behavior, I think we don''t need that many case to case handle. Dexuan is working on the implementation base on the summit talk and close to end, maybe Yuji and Dexuan can coordinate first to see if the proposed policy can server yuji''s purpose. Thx, eddie _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian Jackson
2008-Jun-27 13:27 UTC
Re: [Xen-devel] [PATCH][RFC] Support more Capability Structures andDevice Specific
Keir Fraser writes ("Re: [Xen-devel] [PATCH][RFC] Support more Capability Structures andDevice Specific"):> What do you think of Yuji''s patch? One thing to consider is that perhaps the > patch should be against the upstream qemu merge now. But I''m not sure that > PCI passthrough is even supported yet in that tree (Ian?).In theory PCI passthrough is supposed to be supported in my merged tree. But, I haven''t tested it yet so I doubt it works. Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian Jackson
2008-Jun-27 13:34 UTC
RE: [Xen-devel] [PATCH][RFC] Support more Capability Structures andDevice Specific
Dong, Eddie writes ("RE: [Xen-devel] [PATCH][RFC] Support more Capability Structures andDevice Specific"):> If we agree the basic policy is pass through except the ones with known > behavior, I think we don''t need that many case to case handle. Dexuan is > working on the implementation base on the summit talk and close to end, > maybe Yuji and Dexuan can coordinate first to see if the proposed policy > can server yuji''s purpose.Is it really safe to pass through operations with unknown behavious ? Particularly if the system has an iommu, the administrator may be expecting the passthrough mechanism to defend the host from rogue behaviour by the card and its owning guest. (I''m no expert on PCI so forgive me if this question is stupid.) Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Samuel Thibault
2008-Jun-27 13:51 UTC
Re: [Xen-devel] [PATCH][RFC] Support more Capability Structures and Device Specific
Hello, diff -r 926a366ca82f tools/ioemu/list.h --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/tools/ioemu/list.h Fri Jun 27 11:58:26 2008 +0900 @@ -0,0 +1,89 @@ +#ifndef _IOEMU_LIST_H +#define _IOEMU_LIST_H +/* Taken from Linux kernel code, but de-kernelized for userspace. */ +#include <stddef.h> Could you use the list implementation of qemu instead please? Samuel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian Jackson
2008-Jun-27 13:55 UTC
Re: [Xen-devel] [PATCH][RFC] Support more Capability Structures andDevice Specific
Ian Jackson writes ("Re: [Xen-devel] [PATCH][RFC] Support more Capability Structures andDevice Specific"):> Keir Fraser writes ("Re: [Xen-devel] [PATCH][RFC] Support more Capability Structures andDevice Specific"): > > What do you think of Yuji''s patch? One thing to consider is that > > perhaps the patch should be against the upstream qemu merge > > now. But I''m not sure that PCI passthrough is even supported yet > > in that tree (Ian?). > > In theory PCI passthrough is supposed to be supported in my merged > tree. But, I haven''t tested it yet so I doubt it works.I should expand on that. I''d very much like for these kind of patches to be going into the merged qemu tree now, and I''d also like people to help test it. So it would be very good if people who actually use PCI passthrough were to take a look at the merged qemu tree. Thanks, Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Yuji Shimada
2008-Jun-30 04:31 UTC
Re: [Xen-devel] [PATCH][RFC] Support more Capability Structures andDevice Specific
I think it is NOT safe to pass through operations with unknown behaviour. qemu-dm should prevent guest software setting unsafe value to register. We have to investigate each register and decide to emulate(virtualize) or passthrough. I haven''t investigated some capability structures (like PCI-X Capability Structure). I hide them from guest software. Device Specific Registers (exclude capability structures) is passthrough. In non-virtualized environment, OS does not touch device specific registers, but device drivers configure them. In virtualized environment, we have to allow device drivers to configure them. -- Yuji Shimada On Fri, 27 Jun 2008 14:34:11 +0100 Ian Jackson <Ian.Jackson@eu.citrix.com> wrote:> Dong, Eddie writes ("RE: [Xen-devel] [PATCH][RFC] Support more Capability Structures andDevice Specific"): > > If we agree the basic policy is pass through except the ones with known > > behavior, I think we don''t need that many case to case handle. Dexuan is > > working on the implementation base on the summit talk and close to end, > > maybe Yuji and Dexuan can coordinate first to see if the proposed policy > > can server yuji''s purpose. > > Is it really safe to pass through operations with unknown behavious ? > Particularly if the system has an iommu, the administrator may be > expecting the passthrough mechanism to defend the host from rogue > behaviour by the card and its owning guest. > > (I''m no expert on PCI so forgive me if this question is stupid.) > > Ian. > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Cui, Dexuan
2008-Jun-30 05:48 UTC
RE: [Xen-devel] [PATCH][RFC] Support more Capability StructuresandDevice Specific
Hi Yuji, I looked at the patch. It seems pretty good. Except for the (temporary) absence of MSI/MSI-X stuff, looks the passthrough policy in the patch is almost the same as what is discussed in the PDF file Eddie posted. I also made some tests against the patch, and found there may be some unstable issues: I.e., when I boot a 32e RHEL5u1 (I add the "pci=nomsi" parameter)), it can easily (30%~80% probable) stay for a very long (i.e., >40s) at "Starting udev:", and after I login in shell, the NIC seems not present (the guest has no network available), but "lspci" shows the NIC is there. If I use the Qemu without your patch, the issue disappears at once, and NIC in guest works well. I haven''t found issue in your patch yet. :) Thanks, -- Dexuan -----Original Message----- From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Yuji Shimada Sent: 2008年6月30日 12:32 To: Ian Jackson Cc: xen-devel@lists.xensource.com; Dong, Eddie; Keir Fraser Subject: Re: [Xen-devel] [PATCH][RFC] Support more Capability StructuresandDevice Specific I think it is NOT safe to pass through operations with unknown behaviour. qemu-dm should prevent guest software setting unsafe value to register. We have to investigate each register and decide to emulate(virtualize) or passthrough. I haven''t investigated some capability structures (like PCI-X Capability Structure). I hide them from guest software. Device Specific Registers (exclude capability structures) is passthrough. In non-virtualized environment, OS does not touch device specific registers, but device drivers configure them. In virtualized environment, we have to allow device drivers to configure them. -- Yuji Shimada On Fri, 27 Jun 2008 14:34:11 +0100 Ian Jackson <Ian.Jackson@eu.citrix.com> wrote:> Dong, Eddie writes ("RE: [Xen-devel] [PATCH][RFC] Support more Capability Structures andDevice Specific"): > > If we agree the basic policy is pass through except the ones with known > > behavior, I think we don''t need that many case to case handle. Dexuan is > > working on the implementation base on the summit talk and close to end, > > maybe Yuji and Dexuan can coordinate first to see if the proposed policy > > can server yuji''s purpose. > > Is it really safe to pass through operations with unknown behavious ? > Particularly if the system has an iommu, the administrator may be > expecting the passthrough mechanism to defend the host from rogue > behaviour by the card and its owning guest. > > (I''m no expert on PCI so forgive me if this question is stupid.) > > Ian. > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Yuji Shimada
2008-Jun-30 07:12 UTC
Re: [Xen-devel] [PATCH][RFC] Support more Capability Structures and Device Specific
Do you mean the list implementation inside tools/ioemu/audio/sys-queue.h in xen-unstable tree ? I haven''t checked the upstream qemu code. Thanks. -- Yuji Shimada> Hello, > > diff -r 926a366ca82f tools/ioemu/list.h > --- /dev/null Thu Jan 01 00:00:00 1970 +0000 > +++ b/tools/ioemu/list.h Fri Jun 27 11:58:26 2008 +0900 > @@ -0,0 +1,89 @@ > +#ifndef _IOEMU_LIST_H > +#define _IOEMU_LIST_H > +/* Taken from Linux kernel code, but de-kernelized for userspace. */ > +#include <stddef.h> > > Could you use the list implementation of qemu instead please? > > Samuel > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Yuji Shimada
2008-Jun-30 07:14 UTC
Re: [Xen-devel] [PATCH][RFC] Support more Capability Structures andDevice Specific
I think it is better to emulate registers related to error reporting at least. We can consider two patterns of error handling as following. PATTERN-1: AER is enabled on host but will not be notified to guest domain. Actual AER is enable, but guest software is not allowed to enable AER by _OSC in guest firmware. When error occurs, dom0 will kill guest domain and reset the device or bus. Registers related to error reporting should be emulated to prevent guest software turning off actual error reporting. PATTERN-2: AER is enabled on host and will also be notified to guest domain. Actual AER is always enable, but guest can enable/disable its virtual AER. When error occurs, dom0 check guest emulated AER is enabled or not. If guest emulated AER is enabled, dom0 will notify error to guest software. Then guest software will reset I/O device or bus. If guest emulated AER is disabled, dom0 will not notify error to guest software. Dom0 will kill guest domain, and reset I/O device or bus. To do this, registers related to error reporting should be emulated. And Root Port emulation is also required. I''ve taken a look at "Xen - VT-D Enhance.pdf". Is Dexuan implementing Memory Mapped Configuration Access Mechanism to support offset 256-4095? Are following interfaces not changed ? pci_dev->config_read pci_dev->config_write If they are not changed, I think Dexuan''s Memory Mapped Configuration Access Mechanism and my code can be merged easily. Thanks a lot. -- Yuji Shimada> Keir Fraser wrote: > > What do you think of Yuji''s patch? One thing to consider > > is that perhaps the patch should be against the upstream > > qemu merge now. But I''m not sure that PCI passthrough is > > even supported yet in that tree (Ian?). > > > > If we agree the basic policy is pass through except the ones with known > behavior, I think we don''t need that many case to case handle. Dexuan is > working on the implementation base on the summit talk and close to end, > maybe Yuji and Dexuan can coordinate first to see if the proposed policy > can server yuji''s purpose. > > Thx, eddie > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Yuji Shimada
2008-Jun-30 08:00 UTC
Re: [Xen-devel] [PATCH][RFC] Support more Capability Structures andDevice Specific
I am going to release these following patches. 1. the patch supporting MSI Capability Structure for Xen Unstable. (list.h will be removed. some bugs will be fixed.) 2. the patch supporting MSI-X Capability Structure for Xen Unstable. 3. the patch against the upstream qemu merge (including MSI/MSI-X Capability Structure). Should I release patch against the upstream qemu merge prior to Xen Unstable ? We''d be very happy if you could help me merged that patch, so that I can concentrate on MSI/MSI-X patches. Thanks. -- Yuji Shimada On Fri, 27 Jun 2008 14:55:01 +0100 Ian Jackson <Ian.Jackson@eu.citrix.com> wrote:> Ian Jackson writes ("Re: [Xen-devel] [PATCH][RFC] Support more Capability Structures andDevice Specific"): > > Keir Fraser writes ("Re: [Xen-devel] [PATCH][RFC] Support more Capability Structures andDevice Specific"): > > > What do you think of Yuji''s patch? One thing to consider is that > > > perhaps the patch should be against the upstream qemu merge > > > now. But I''m not sure that PCI passthrough is even supported yet > > > in that tree (Ian?). > > > > In theory PCI passthrough is supposed to be supported in my merged > > tree. But, I haven''t tested it yet so I doubt it works. > > I should expand on that. I''d very much like for these kind of patches > to be going into the merged qemu tree now, and I''d also like people to > help test it. > > So it would be very good if people who actually use PCI passthrough > were to take a look at the merged qemu tree. > > Thanks, > Ian. > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Yuji Shimada
2008-Jun-30 08:14 UTC
Re: [Xen-devel] [PATCH][RFC] Support more Capability StructuresandDevice Specific
Hi Dexuan, I''ve tested my patch with CentOS 5.1 and PCI/PCIe NIC. In my test environment (with "pci=nomsi" set for Dom0 boot parameter), guest OS can use the assigned NIC and can communicate with external machine. Does guest OS recieve interrupt? You can check via /proc/interrupts. Thanks. -- Yuji Shimada> Hi Yuji, > I looked at the patch. It seems pretty good. > Except for the (temporary) absence of MSI/MSI-X stuff, looks the passthrough policy in the patch is almost the same as what is discussed in the PDF file Eddie posted. > > I also made some tests against the patch, and found there may be some unstable issues: > I.e., when I boot a 32e RHEL5u1 (I add the "pci=nomsi" parameter)), it can easily (30%~80% probable) stay for a very long (i.e., >40s) at "Starting udev:", and after I login in shell, the NIC seems not present (the guest has no network available), but "lspci" shows the NIC is there. > If I use the Qemu without your patch, the issue disappears at once, and NIC in guest works well. > > I haven''t found issue in your patch yet. :) > > Thanks, > -- Dexuan_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Cui, Dexuan
2008-Jun-30 09:02 UTC
RE: [Xen-devel] [PATCH][RFC] Support more Capability StructuresandDevice Specific
I didn''t work on the memory mapped access machinism. I''m not sure whether adding that support is OK since now Qemu emulates an old chipset which is actually PCIe-unaware. Thanks, -- Dexuan -----Original Message----- From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Yuji Shimada Sent: 2008年6月30日 15:15 To: Dong, Eddie Cc: xen-devel@lists.xensource.com; Ian Jackson; Keir Fraser Subject: Re: [Xen-devel] [PATCH][RFC] Support more Capability StructuresandDevice Specific I think it is better to emulate registers related to error reporting at least. We can consider two patterns of error handling as following. PATTERN-1: AER is enabled on host but will not be notified to guest domain. Actual AER is enable, but guest software is not allowed to enable AER by _OSC in guest firmware. When error occurs, dom0 will kill guest domain and reset the device or bus. Registers related to error reporting should be emulated to prevent guest software turning off actual error reporting. PATTERN-2: AER is enabled on host and will also be notified to guest domain. Actual AER is always enable, but guest can enable/disable its virtual AER. When error occurs, dom0 check guest emulated AER is enabled or not. If guest emulated AER is enabled, dom0 will notify error to guest software. Then guest software will reset I/O device or bus. If guest emulated AER is disabled, dom0 will not notify error to guest software. Dom0 will kill guest domain, and reset I/O device or bus. To do this, registers related to error reporting should be emulated. And Root Port emulation is also required. I''ve taken a look at "Xen - VT-D Enhance.pdf". Is Dexuan implementing Memory Mapped Configuration Access Mechanism to support offset 256-4095? Are following interfaces not changed ? pci_dev->config_read pci_dev->config_write If they are not changed, I think Dexuan''s Memory Mapped Configuration Access Mechanism and my code can be merged easily. Thanks a lot. -- Yuji Shimada> Keir Fraser wrote: > > What do you think of Yuji''s patch? One thing to consider > > is that perhaps the patch should be against the upstream > > qemu merge now. But I''m not sure that PCI passthrough is > > even supported yet in that tree (Ian?). > > > > If we agree the basic policy is pass through except the ones with known > behavior, I think we don''t need that many case to case handle. Dexuan is > working on the implementation base on the summit talk and close to end, > maybe Yuji and Dexuan can coordinate first to see if the proposed policy > can server yuji''s purpose. > > Thx, eddie > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Cui, Dexuan
2008-Jun-30 09:29 UTC
RE: [Xen-devel] [PATCH][RFC] Support more Capability StructuresandDevice Specific
I''m using x86_64 c/s 17888: 6ace85eb96c0, and assigning a 82541PI Gigabit Etherer NIC to guest. I also tried "pci=nomsi" for Dom0, and the issus is still there. When the issue happens, eth0 doesn''t occur in /proc/interrupt though the device driver module is loaded. The issue doesn''t happen every time. Really strange... Thanks, -- Dexuan -----Original Message----- From: Yuji Shimada [mailto:shimada-yxb@necst.nec.co.jp] Sent: 2008年6月30日 16:15 To: Cui, Dexuan Cc: Ian Jackson; xen-devel@lists.xensource.com; Dong, Eddie; Keir Fraser Subject: Re: [Xen-devel] [PATCH][RFC] Support more Capability StructuresandDevice Specific Hi Dexuan, I''ve tested my patch with CentOS 5.1 and PCI/PCIe NIC. In my test environment (with "pci=nomsi" set for Dom0 boot parameter), guest OS can use the assigned NIC and can communicate with external machine. Does guest OS recieve interrupt? You can check via /proc/interrupts. Thanks. -- Yuji Shimada> Hi Yuji, > I looked at the patch. It seems pretty good. > Except for the (temporary) absence of MSI/MSI-X stuff, looks the passthrough policy in the patch is almost the same as what is discussed in the PDF file Eddie posted. > > I also made some tests against the patch, and found there may be some unstable issues: > I.e., when I boot a 32e RHEL5u1 (I add the "pci=nomsi" parameter)), it can easily (30%~80% probable) stay for a very long (i.e., >40s) at "Starting udev:", and after I login in shell, the NIC seems not present (the guest has no network available), but "lspci" shows the NIC is there. > If I use the Qemu without your patch, the issue disappears at once, and NIC in guest works well. > > I haven''t found issue in your patch yet. :) > > Thanks, > -- Dexuan_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Samuel Thibault
2008-Jun-30 10:22 UTC
Re: [Xen-devel] [PATCH][RFC] Support more Capability Structures and Device Specific
Yuji Shimada, le Mon 30 Jun 2008 16:12:31 +0900, a écrit :> Do you mean the list implementation inside tools/ioemu/audio/sys-queue.h > in xen-unstable tree ?Yes.> I haven''t checked the upstream qemu code.It''s the same, with an additional QEMU prefix. Samuel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian Jackson
2008-Jun-30 16:50 UTC
Re: [Xen-devel] [PATCH][RFC] Support more Capability Structures andDevice Specific
Yuji Shimada writes ("Re: [Xen-devel] [PATCH][RFC] Support more Capability Structures andDevice Specific"):> I am going to release these following patches....> Should I release patch against the upstream qemu merge prior to Xen > Unstable ?We''re currently working to integrate the upstream qemu merge into patchman. I hope that we can be successful with this, so that we can switch to this tree as the default in 3.3. With that in mind: Have you looked at the upstream merge at all yet ? I haven''t yet done any PCI passthrough testing and I think it''s very important that we do some of that testing. If you have any time available to do some checking of the qemu upstream merge tree, even before we consider your recent patches, that would be very helpful. Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Dong, Eddie
2008-Jul-01 02:12 UTC
RE: [Xen-devel] [PATCH][RFC] Support more Capability Structures andDevice Specific
Ian Jackson wrote:> Dong, Eddie writes ("RE: [Xen-devel] [PATCH][RFC] Support > more Capability Structures andDevice Specific"): >> If we agree the basic policy is pass through except the >> ones with known behavior, I think we don''t need that >> many case to case handle. Dexuan is working on the >> implementation base on the summit talk and close to end, >> maybe Yuji and Dexuan can coordinate first to see if the >> proposed policy can server yuji''s purpose. > > Is it really safe to pass through operations with unknown > behavious ? > Particularly if the system has an iommu, the > administrator may be > expecting the passthrough mechanism to defend the host > from rogue > behaviour by the card and its owning guest.What kind of operations in your mind will hurt host? But yes, guest may not work properly for some cases such as some vidoe card I mentioned in summit which may map host address to internal register and used by drivers. For those kind of devices, CP will just disable assignment thru kind of assignable check such as blacklist. Other than that, I didn''t see how pass through will make things worse. Can u specify ? All guest access to memory is protected by IOMMU and thus no imapct to host. Thx, eddie _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Cui, Dexuan
2008-Jul-01 02:25 UTC
RE: [Xen-devel] [PATCH][RFC] Support more Capability StructuresandDevice Specific
The PCI config space passthrough is actually crucial for Xen 3.3. We have already found many device assignment issues caused by the lack of the passthrough (and I think more issues will be found). For example, the assigned BroadCom NIC can''t work because the driver needs to access the register BNX2_PCICFG_INT_ACK_CMD (at 0x84); another example: the USB assignment doesn''t work -- one of the reasons is: UHCI-hc needs to access config space reigster USB_LEGKEY(at 0xC0-C1) to enable USB IRQ. So it will be great to push the patch into xen-unstable tree. Thanks, -- Dexuan -----Original Message----- From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Ian Jackson Sent: 2008年7月1日 0:50 To: Yuji Shimada Cc: xen-devel@lists.xensource.com; Dong, Eddie; Keir Fraser Subject: Re: [Xen-devel] [PATCH][RFC] Support more Capability StructuresandDevice Specific Yuji Shimada writes ("Re: [Xen-devel] [PATCH][RFC] Support more Capability Structures andDevice Specific"):> I am going to release these following patches....> Should I release patch against the upstream qemu merge prior to Xen > Unstable ?We''re currently working to integrate the upstream qemu merge into patchman. I hope that we can be successful with this, so that we can switch to this tree as the default in 3.3. With that in mind: Have you looked at the upstream merge at all yet ? I haven''t yet done any PCI passthrough testing and I think it''s very important that we do some of that testing. If you have any time available to do some checking of the qemu upstream merge tree, even before we consider your recent patches, that would be very helpful. Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Dong, Eddie
2008-Jul-01 02:27 UTC
RE: [Xen-devel] [PATCH][RFC] Support more Capability Structures andDevice Specific
Yuji Shimada wrote:> I think it is NOT safe to pass through operations with > unknown behaviour. qemu-dm should prevent guest softwareThis used to be the reason why we only pass through CMD register, but we then suffer from that in different devices and I have to admit, without passthrough real setting, the device won''t function correctly in many cases. Do u have any real data ? Device memory access is fine with IOMMU, irq vector is virtualized now.> setting unsafe value to register. We have to investigate > each register and decide to emulate(virtualize) or > passthrough.Investigation is defintely good, eventually we need to know all configuration registers, but even with that, there are still device specifc registers we have to deal such as Vendor specific capability and those registers directly defined by devices (not a standard PCI capability). But hidding settings due to the reason we didn''t investigate yet will simply make things worse, and we already observed this with more devices tested such as UHCI mouse etc.> > I haven''t investigated some capability structures (like > PCI-X Capability Structure). I hide them from guest > software.ditto> > Device Specific Registers (exclude capability structures) > is passthrough. In non-virtualized environment, OS does > not touch device specific registers, but device drivers > configure them. In virtualized environment, we have to > allow device drivers to configure them.We need to pass through except the device is doing IRQ vector setting which can''t be handled in either case, those devices are simply not assignable. Thanks, eddie _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Yuji Shimada
2008-Jul-01 08:00 UTC
Re: [Xen-devel] [PATCH][RFC] Support more Capability Structures andDevice Specific
In my patch, registers inside Vendor Specific Capability Structure (Capability ID is 09h) are all passthrough, except Next Capability Pointer Register. Registers area defined by device are not belonging to Capability Structure, and they are all passthrough. Do you mean non standard Capability Structure should be passthrough? For PCI-X Capability Structure, we have to emulate Function Number Field, Device Number Field, and Bus Number Field in PCI-X Status Register at least. The reason is that the value for host and guest maybe different. Additionally, I think we have to emulate Maximum Outstanding Split Transactions Field in PCI-X Command Register too. The reason is that the real value should be decided by considering all over the system, and we should also prevent guest software writing value. Thanks. -- Yuji Shimada On Tue, 1 Jul 2008 10:27:02 +0800 "Dong, Eddie" <eddie.dong@intel.com> wrote:> Yuji Shimada wrote: > > I think it is NOT safe to pass through operations with > > unknown behaviour. qemu-dm should prevent guest software > > This used to be the reason why we only pass through CMD register, but we > then suffer from that in different devices and I have to admit, without > passthrough real setting, the device won''t function correctly in many > cases. > > Do u have any real data ? Device memory access is fine with IOMMU, irq > vector is virtualized now. > > > setting unsafe value to register. We have to investigate > > each register and decide to emulate(virtualize) or > > passthrough. > > Investigation is defintely good, eventually we need to know all > configuration registers, but even with that, there are still device > specifc registers we have to deal such as Vendor specific capability and > those registers directly defined by devices (not a standard PCI > capability). But hidding settings due to the reason we didn''t > investigate yet will simply make things worse, and we already observed > this with more devices tested such as UHCI mouse etc. > > > > > > I haven''t investigated some capability structures (like > > PCI-X Capability Structure). I hide them from guest > > software. > > ditto > > > > > Device Specific Registers (exclude capability structures) > > is passthrough. In non-virtualized environment, OS does > > not touch device specific registers, but device drivers > > configure them. In virtualized environment, we have to > > allow device drivers to configure them. > > We need to pass through except the device is doing IRQ vector setting > which can''t be handled in either case, those devices are simply not > assignable. > > Thanks, eddie > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian Jackson
2008-Jul-01 09:54 UTC
Re: [Xen-devel] [PATCH][RFC] Support more Capability Structures andDevice Specific
Yuji Shimada writes ("Re: [Xen-devel] [PATCH][RFC] Support more Capability Structures andDevice Specific"):> In my patch, registers inside Vendor Specific Capability Structure > (Capability ID is 09h) are all passthrough, except Next Capability > Pointer Register.My worry is that a device may say, in its vendor-specific register-level programming documentation for these configurations, something like: Do _not_ set USE_EXTERNAL_INPUT and USE_INTERNAL_INPUT simultaneously; this may cause damage to the Gnomovision PCI card and may also cause the Gnomovision PCI card to draw excessive current from the host power supply. Or Do _not_ use the UPLOAD_FIRMWARE_* configuration. These are for use by the approved Gnomovision firmware loader only. Uploading bad firmware may cause damage [etc. etc.] I haven''t read many modern PCI card specs but with the constant shifting of functionality (even functionality which is intended to preserve hardware integrity) to software and firmware, I would be wary of assuming that every unknown PCI card has no register and configuration settings which can cause hardware damage or other kinds of unexpected and undesirable events. If there is there a requirement written into the general PCI specification that this won''t happen, then fine - if so please quote chapter and verse. Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Dong, Eddie
2008-Jul-01 23:23 UTC
RE: [Xen-devel] [PATCH][RFC] Support more Capability Structures andDevice Specific
> I haven''t read many modern PCI card specs but with the > constant > shifting of functionality (even functionality which is > intended to > preserve hardware integrity) to software and firmware, I > would be wary > of assuming that every unknown PCI card has no register > and > configuration settings which can cause hardware damage or > other kinds > of unexpected and undesirable events.For those vendor specific registers (in PCI cap or not), we always have dillemma: pass through with functionality or discard with malfunction. If we pass through them, then the opposite card are in black list that we can''t support, if we discard them, the former class of card becomes un-assignable. Per current data, pass through get many known bug fixed as the case Dexuan mentioned. But we didn''t see a HW damaging host. Some know issue could be a device issuing tons of PCIe traffic, absorbing extra power, issuing interrupt storm etc, but right now we didn''t see issues yet.> > If there is there a requirement written into the general > PCI > specification that this won''t happen, then fine - if so > please quote > chapter and verse. >Defintely if PCIe spec can be modified to support this one is best, but we still have legacy devices. Thx, eddie _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Yuji Shimada
2008-Jul-02 01:03 UTC
Re: [Xen-devel] [PATCH][RFC] Support more Capability StructuresandDevice Specific
I''ve done some bug fixes as follows. 1. correct the size calculation of MSI Capability Structure in pt_msi_size_init(). The next capability might be hidden due to wrong large size of MSI. 2. modify the decision logic for determining unused Exp ROM BAR in pt_bar_reg_parse(). Use PCIIORegion table instead of parsing BAR itself. 3. bug fix on .size_init func for PCI Express Capability Structure in pt_emu_reg_grp_tbl[]. (pt_vendor_size_init ---> pt_reg_grp_size_init) 4. small bug fix on the decision logic for checking unused BAR in pt_pci_write_config(). 5. add printf message to show overlapped device in pt_chk_bar_overlap(). 6. modify pt_bar_mapping() to prevent guest software mapping memory resource to 00000000h 7. modify pt_bar_mapping() to map resource even if overlapping is detected. I''ve tested my patch with CentOS 5.1 and PCI/PCIe NIC. Without "pci=nomsi", guest OS can use the assigned NIC and can communicate with external machine. Additionally I assigned UHCI Controller to guest domain. Guest OS can use USB-HDD and USB-Mouse. Could you test the patch? I am going to remove list.h and enable MSI. Thanks. Signed-off-by: Yuji Shimada <shimada-yxb@necst.nec.co.jp> -- Yuji Shimada On Mon, 30 Jun 2008 17:29:38 +0800 "Cui, Dexuan" <dexuan.cui@intel.com> wrote:> I''m using x86_64 c/s 17888: 6ace85eb96c0, and assigning a 82541PI Gigabit Etherer NIC to guest. > I also tried "pci=nomsi" for Dom0, and the issus is still there. > When the issue happens, eth0 doesn''t occur in /proc/interrupt though the device driver module is loaded. > The issue doesn''t happen every time. Really strange... > > Thanks, > -- Dexuan > > > -----Original Message----- > From: Yuji Shimada [mailto:shimada-yxb@necst.nec.co.jp] > Sent: 2008夏・花可30科苛 16:15 > To: Cui, Dexuan > Cc: Ian Jackson; xen-devel@lists.xensource.com; Dong, Eddie; Keir Fraser > Subject: Re: [Xen-devel] [PATCH][RFC] Support more Capability StructuresandDevice Specific > > Hi Dexuan, > > I''ve tested my patch with CentOS 5.1 and PCI/PCIe NIC. In my test > environment (with "pci=nomsi" set for Dom0 boot parameter), guest > OS can use the assigned NIC and can communicate with external machine. > > Does guest OS recieve interrupt? You can check via /proc/interrupts. > > Thanks. > > -- > Yuji Shimada > > > Hi Yuji, > > I looked at the patch. It seems pretty good. > > Except for the (temporary) absence of MSI/MSI-X stuff, looks the passthrough policy in the patch is almost the same as what is discussed in the PDF file Eddie posted. > > > > I also made some tests against the patch, and found there may be some unstable issues: > > I.e., when I boot a 32e RHEL5u1 (I add the "pci=nomsi" parameter)), it can easily (30%~80% probable) stay for a very long (i.e., >40s) at "Starting udev:", and after I login in shell, the NIC seems not present (the guest has no network available), but "lspci" shows the NIC is there. > > If I use the Qemu without your patch, the issue disappears at once, and NIC in guest works well. > > > > I haven''t found issue in your patch yet. :) > > > > Thanks, > > -- Dexuan > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Cui, Dexuan
2008-Jul-02 02:07 UTC
RE: [Xen-devel] [PATCH][RFC] Support more Capability StructuresandDevice Specific
I had a test with your new patch: the occasional-NIC-doesn''t-work issue I noticed disappeared. I''ll have a look why it disappeared. :-) Thanks, -- Dexuan -----Original Message----- From: Yuji Shimada [mailto:shimada-yxb@necst.nec.co.jp] Sent: 2008年7月2日 9:03 To: Cui, Dexuan Cc: Ian Jackson; xen-devel@lists.xensource.com; Dong, Eddie; Keir Fraser Subject: Re: [Xen-devel] [PATCH][RFC] Support more Capability StructuresandDevice Specific I''ve done some bug fixes as follows. 1. correct the size calculation of MSI Capability Structure in pt_msi_size_init(). The next capability might be hidden due to wrong large size of MSI. 2. modify the decision logic for determining unused Exp ROM BAR in pt_bar_reg_parse(). Use PCIIORegion table instead of parsing BAR itself. 3. bug fix on .size_init func for PCI Express Capability Structure in pt_emu_reg_grp_tbl[]. (pt_vendor_size_init ---> pt_reg_grp_size_init) 4. small bug fix on the decision logic for checking unused BAR in pt_pci_write_config(). 5. add printf message to show overlapped device in pt_chk_bar_overlap(). 6. modify pt_bar_mapping() to prevent guest software mapping memory resource to 00000000h 7. modify pt_bar_mapping() to map resource even if overlapping is detected. I''ve tested my patch with CentOS 5.1 and PCI/PCIe NIC. Without "pci=nomsi", guest OS can use the assigned NIC and can communicate with external machine. Additionally I assigned UHCI Controller to guest domain. Guest OS can use USB-HDD and USB-Mouse. Could you test the patch? I am going to remove list.h and enable MSI. Thanks. Signed-off-by: Yuji Shimada <shimada-yxb@necst.nec.co.jp> -- Yuji Shimada On Mon, 30 Jun 2008 17:29:38 +0800 "Cui, Dexuan" <dexuan.cui@intel.com> wrote:> I''m using x86_64 c/s 17888: 6ace85eb96c0, and assigning a 82541PI Gigabit Etherer NIC to guest. > I also tried "pci=nomsi" for Dom0, and the issus is still there. > When the issue happens, eth0 doesn''t occur in /proc/interrupt though the device driver module is loaded. > The issue doesn''t happen every time. Really strange... > > Thanks, > -- Dexuan > > > -----Original Message----- > From: Yuji Shimada [mailto:shimada-yxb@necst.nec.co.jp] > Sent: 2008夏・花可30科苛 16:15 > To: Cui, Dexuan > Cc: Ian Jackson; xen-devel@lists.xensource.com; Dong, Eddie; Keir Fraser > Subject: Re: [Xen-devel] [PATCH][RFC] Support more Capability StructuresandDevice Specific > > Hi Dexuan, > > I''ve tested my patch with CentOS 5.1 and PCI/PCIe NIC. In my test > environment (with "pci=nomsi" set for Dom0 boot parameter), guest > OS can use the assigned NIC and can communicate with external machine. > > Does guest OS recieve interrupt? You can check via /proc/interrupts. > > Thanks. > > -- > Yuji Shimada > > > Hi Yuji, > > I looked at the patch. It seems pretty good. > > Except for the (temporary) absence of MSI/MSI-X stuff, looks the passthrough policy in the patch is almost the same as what is discussed in the PDF file Eddie posted. > > > > I also made some tests against the patch, and found there may be some unstable issues: > > I.e., when I boot a 32e RHEL5u1 (I add the "pci=nomsi" parameter)), it can easily (30%~80% probable) stay for a very long (i.e., >40s) at "Starting udev:", and after I login in shell, the NIC seems not present (the guest has no network available), but "lspci" shows the NIC is there. > > If I use the Qemu without your patch, the issue disappears at once, and NIC in guest works well. > > > > I haven''t found issue in your patch yet. :) > > > > Thanks, > > -- Dexuan > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian Jackson
2008-Jul-02 10:30 UTC
RE: [Xen-devel] [PATCH][RFC] Support more Capability Structures andDevice Specific
Dong, Eddie writes ("RE: [Xen-devel] [PATCH][RFC] Support more Capability Structures andDevice Specific"):> Per current data, pass through get many known bug fixed as the case > Dexuan mentioned. But we didn''t see a HW damaging host. Some know issue > could be a device issuing tons of PCIe traffic, absorbing extra power, > issuing interrupt storm etc, but right now we didn''t see issues yet.Most people doing PCI passthrough appear to be under the impression that the guest cannot escape and cannot damage the host. (Even those currently doing PCI passthrough with current production hardware without an iommu!) I think it is fine to have a passthrough option which doesn''t properly protect the host from the guest - this is a useful setup in many situations. But it should not be enabled by default, surely ? Note that this is a _security_ problem. So `data'' about `issues'' which you have `seen'' is irrelevant. Just because you haven''t actually observed any misbehaviour with non-malicious guests doesn''t mean that a malicious guest couldn''t cause the hardware to melt. Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Alan Cox
2008-Jul-02 11:17 UTC
Re: [Xen-devel] [PATCH][RFC] Support more Capability Structures andDevice Specific
> I think it is fine to have a passthrough option which doesn''t properly > protect the host from the guest - this is a useful setup in many > situations. But it should not be enabled by default, surely ?Agreed entirely. Note also that some implementations of an IOMMU will not save you as they don''t fence between individual PCI devices (PCIE is obviously a bit easier). Not fencing between devices allows you for example to use a fairly flexible SCSI controller to reprogram another device. In the general case there are also some really nasty dirty attacks you can''t stop with an IOMMU one of which is to reflash the BIOS of the graphics card to which you were given unrestricted access so that you compromise the entire system next boot. These attacks appear well understood except by IOMMU marketing people ;) IOMMU is great for system correctness and flexibility, using it for safely providing hardware direct access is a very very hairy business with a complex device. Alan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Dong, Eddie
2008-Jul-03 01:38 UTC
RE: [Xen-devel] [PATCH][RFC] Support more Capability Structures andDevice Specific
Ian Jackson wrote:> Dong, Eddie writes ("RE: [Xen-devel] [PATCH][RFC] Support > more Capability Structures andDevice Specific"): >> Per current data, pass through get many known bug fixed >> as the case Dexuan mentioned. But we didn''t see a HW >> damaging host. Some know issue could be a device issuing >> tons of PCIe traffic, absorbing extra power, issuing >> interrupt storm etc, but right now we didn''t see issues >> yet. > > Most people doing PCI passthrough appear to be under the > impression > that the guest cannot escape and cannot damage the host. > (Even those > currently doing PCI passthrough with current production > hardware > without an iommu!)What I am aware is only QoS, I didn''t know how can a guest program the device to crash host. Interrupt storm can be blocked by hypervisor at certain situation. Competing for unnecessary PCIe traffic is never related to if we pass through guest setting or not. Can you give me a specific example how host will be crashed?> > I think it is fine to have a passthrough option which > doesn''t properly > protect the host from the guest - this is a useful setup > in many > situations. But it should not be enabled by default, > surely ?Same reason as above.> > Note that this is a _security_ problem. So `data'' about > `issues'' > which you have `seen'' is irrelevant. Just because you > haven''t > actually observed any misbehaviour with non-malicious > guests doesn''t > mean that a malicious guest couldn''t cause the hardware > to melt.Examples even in theory? NOTE here, current pass through logic only support devices under root port.> > Ian.Thx, eddie _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Dong, Eddie
2008-Jul-03 01:46 UTC
RE: [Xen-devel] [PATCH][RFC] Support more Capability Structures andDevice Specific
Alan Cox wrote:>> I think it is fine to have a passthrough option which >> doesn''t properly protect the host from the guest - this >> is a useful setup in many situations. But it should not >> be enabled by default, surely ? > > Agreed entirely. Note also that some implementations of > an IOMMU will not save you as they don''t fence between > individual PCI devices (PCIE is obviously a bit easier).IOMMU, at least Intel''s IOMMU, doesn''t support pure PCI device, only PCIe devices can be DMA protected.> Not fencing between devices allows you for example to use > a fairly flexible SCSI controller to reprogram another > device.Again, at least for Intel IOMMU, devices under root endpoint can never escape from IOMMU DMA protection, right now we don''t support PCIe devices under a switch to do assignement, but with future ATS or ACS is implemented, we can assign devices under a switch, where ether the switch disable peer to peer transaction or always pass up "untranslated" traffic to upstream. So your concern is a not real IMO, not? Or do u mean AMD IOMMU may have different implementation?> > In the general case there are also some really nasty > dirty attacks you can''t stop with an IOMMU one of which > is to reflash the BIOS of the graphics card to which you > were given unrestricted access so that you compromise the > entire system next boot. These attacks appear well > understood except by IOMMU marketing people ;)Same with above, this is already protected by IOMMU, peer to peer DMA is not supported right now.> > IOMMU is great for system correctness and flexibility, > using it for safely providing hardware direct access is a > very very hairy business with a complex device. >Agree, that is why we are here :) Thx, eddie _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Dong, Eddie
2008-Jul-03 01:49 UTC
RE: [Xen-devel] [PATCH][RFC] Support more Capability StructuresandDevice Specific
Acked-by: Eddie Dong <eddie.dong@intel.com> Yuji Shimada wrote:> I''ve done some bug fixes as follows. > > 1. correct the size calculation of MSI Capability > Structure in pt_msi_size_init(). The next capability > might be hidden due to wrong large size of MSI. > > 2. modify the decision logic for determining unused Exp > ROM BAR in pt_bar_reg_parse(). Use PCIIORegion table > instead of parsing BAR itself. > > 3. bug fix on .size_init func for PCI Express Capability > Structure in pt_emu_reg_grp_tbl[]. > (pt_vendor_size_init ---> pt_reg_grp_size_init) > > 4. small bug fix on the decision logic for checking > unused BAR in pt_pci_write_config(). > > 5. add printf message to show overlapped device in > pt_chk_bar_overlap(). > > 6. modify pt_bar_mapping() to prevent guest software > mapping memory resource to 00000000h > > 7. modify pt_bar_mapping() to map resource even if > overlapping is detected. > > I''ve tested my patch with CentOS 5.1 and PCI/PCIe NIC. > Without "pci=nomsi", guest OS can use the assigned NIC > and can communicate > with external machine. > > Additionally I assigned UHCI Controller to guest domain. > Guest OS can use USB-HDD and USB-Mouse. > > Could you test the patch? > > > I am going to remove list.h and enable MSI. > > Thanks. > > Signed-off-by: Yuji Shimada <shimada-yxb@necst.nec.co.jp> > > >> I''m using x86_64 c/s 17888: 6ace85eb96c0, and assigning >> a 82541PI Gigabit Etherer NIC to guest. >> I also tried "pci=nomsi" for Dom0, and the issus is >> still there. >> When the issue happens, eth0 doesn''t occur in >> /proc/interrupt though the device driver module is >> loaded. The issue doesn''t happen every time. Really >> strange... >> >> Thanks, >> -- Dexuan >> >> >> -----Original Message----- >> From: Yuji Shimada [mailto:shimada-yxb@necst.nec.co.jp] >> Sent: 2008夏・花可30科苛 16:15 >> To: Cui, Dexuan >> Cc: Ian Jackson; xen-devel@lists.xensource.com; Dong, >> Eddie; Keir Fraser >> Subject: Re: [Xen-devel] [PATCH][RFC] Support more >> Capability StructuresandDevice Specific >> >> Hi Dexuan, >> >> I''ve tested my patch with CentOS 5.1 and PCI/PCIe NIC. >> In my test >> environment (with "pci=nomsi" set for Dom0 boot >> parameter), guest >> OS can use the assigned NIC and can communicate with >> external machine. >> >> Does guest OS recieve interrupt? You can check via >> /proc/interrupts. >> >> Thanks. >> >> -- >> Yuji Shimada >> >>> Hi Yuji, >>> I looked at the patch. It seems pretty good. >>> Except for the (temporary) absence of MSI/MSI-X stuff, >>> looks the passthrough policy in the patch is almost the >>> same as what is discussed in the PDF file Eddie posted. >>> >>> I also made some tests against the patch, and found >>> there may be some unstable issues: >>> I.e., when I boot a 32e RHEL5u1 (I add the "pci=nomsi" >>> parameter)), it can easily (30%~80% probable) stay for >>> a very long (i.e., >40s) at "Starting udev:", and after >>> I login in shell, the NIC seems not present (the guest >>> has no network available), but "lspci" shows the NIC is >>> there. If I use the Qemu without your patch, the issue >>> disappears at once, and NIC in guest works well. >>> >>> I haven''t found issue in your patch yet. :) >>> >>> Thanks, >>> -- Dexuan >> >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian Jackson
2008-Jul-03 09:50 UTC
RE: [Xen-devel] [PATCH][RFC] Support more Capability Structures andDevice Specific
Dong, Eddie writes ("RE: [Xen-devel] [PATCH][RFC] Support more Capability Structures andDevice Specific"):> Alan Cox wrote: > > In the general case there are also some really nasty > > dirty attacks you can''t stop with an IOMMU one of which > > is to reflash the BIOS of the graphics card to which you > > were given unrestricted access so that you compromise the > > entire system next boot. These attacks appear well > > understood except by IOMMU marketing people ;) > > Same with above, this is already protected by IOMMU, peer to peer DMA is > not supported right now.You have evidently completely misunderstood Alan''s point. I was going to explain it again but I''m not sure I know how to say it more clearly. Alan''s scenario doesn''t involve any peer to peer DMA. Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Dong, Eddie
2008-Jul-03 23:03 UTC
RE: [Xen-devel] [PATCH][RFC] Support more Capability StructuresandDevice Specific
Ian Jackson wrote:> Dong, Eddie writes ("RE: [Xen-devel] [PATCH][RFC] Support > more Capability Structures andDevice Specific"): >> Alan Cox wrote: >>> In the general case there are also some really nasty >>> dirty attacks you can''t stop with an IOMMU one of which >>> is to reflash the BIOS of the graphics card to which you >>> were given unrestricted access so that you compromise >>> the entire system next boot. These attacks appear well >>> understood except by IOMMU marketing people ;) >> >> Same with above, this is already protected by IOMMU, >> peer to peer DMA is not supported right now. > > You have evidently completely misunderstood Alan''s point. > > I was going to explain it again but I''m not sure I know > how to say it > more clearly. Alan''s scenario doesn''t involve any peer > to peer DMA. > > Ian. >Ok, if it means guest direct MMIO to flash bios, then yes. But it is not related with our discussion, i.e. no matter we pass through CFGS registers or not, it may happen. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Reasonably Related Threads
- FC-HBA assigned to guest domain does not work.
- Bug for Xen-3.3.0-rc2: libpci read error. No emulation.
- [PATCH] Improve the current FLR logic
- [PATCH] fix memory allocation from NUMA node for VT-d.
- [PATCH] dom0 linux: Reassign memory resources to device for pci passthrough.