Han, Weidong
2009-Jul-16 05:15 UTC
[Xen-devel] [pvops-dom0] [patch] Hook Linux''s PCI probe and remove callbacks
Hijack the pci_bus_type probe and remove callbacks. This option only requires modification to the Xen specific part of Linux. This is useful to add and remove pci device to Xen hypervisor when load and remove its driver. For example, when VFs are created by PF, they will be added to Xen hypervisor, and then can be assigned to guest. This patch is based on xen-tip-master branch of jeremy''s pv-ops tree. Signed-off-by: Weidong Han <weidong.han@intel.com> Signed-off-by: Allen Kay <allen.m.kay@intel.com> _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Han, Weidong
2009-Jul-16 07:18 UTC
RE: [Xen-devel] [pvops-dom0] [patch] Hook Linux''s PCI probe and remove callbacks
Forgot to mention this patch is backported from 2.6.18 dom0. Regards, Weidong Han, Weidong wrote:> Hijack the pci_bus_type probe and remove callbacks. This option only > requires modification to the Xen specific part of Linux. > > This is useful to add and remove pci device to Xen hypervisor when > load and remove its driver. For example, when VFs are created by PF, > they will be added to Xen hypervisor, and then can be assigned to > guest. > > This patch is based on xen-tip-master branch of jeremy''s pv-ops tree. > > > Signed-off-by: Weidong Han <weidong.han@intel.com> > Signed-off-by: Allen Kay <allen.m.kay@intel.com>_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Andy Burns
2009-Jul-17 06:28 UTC
Re: [Xen-devel] [pvops-dom0] [patch] Hook Linux''s PCI probe and remove callbacks
2009/7/16 Han, Weidong <weidong.han@intel.com>:> Hijack the pci_bus_type probe and remove callbacks. This option only > requires modification to the Xen specific part of Linux. > > This is useful to add and remove pci device to Xen hypervisor when > load and remove its driver. For example, when VFs are created by PF, > they will be added to Xen hypervisor, and then can be assigned to guest.Do I understand this patch is working towards adding pciback support to pv_ops dom0? Will it it allow device passthrough for CPUs without an IOMMU like 2.6.18 dom0 does? _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeremy Fitzhardinge
2009-Jul-17 23:14 UTC
Re: [Xen-devel] [pvops-dom0] [patch] Hook Linux''s PCI probe and remove callbacks
On 07/15/09 22:15, Han, Weidong wrote:> Hijack the pci_bus_type probe and remove callbacks. This option only > requires modification to the Xen specific part of Linux. > > This is useful to add and remove pci device to Xen hypervisor when > load and remove its driver. For example, when VFs are created by PF, > they will be added to Xen hypervisor, and then can be assigned to guest. > > This patch is based on xen-tip-master branch of jeremy''s pv-ops tree. >Yunhong Jiang sent me a version of this patch as part of his MSI work. This kind of interception of pci_bus_type.probe is pretty ugly, and is unlikely to be accepted upstream in this form (the patch may only touch a Xen file, but it is changing a variable belonging to the PCI subsystem). We need to work with the Linux PCI maintainers to find some other way of achieving what we need here. Can you explain what the functional requirements are here. Could we defer registering the device with Xen until some other convenient hook, or must it be done at this point in the code? Thanks, J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Kay, Allen M
2009-Jul-17 23:42 UTC
RE: [Xen-devel] [pvops-dom0] [patch] Hook Linux''s PCI probe and remove callbacks
These hooks for two purposes: 1) Enabling of a SR-IOV virtual function. The hypercall will tell xen to setup VT-d context entries for the BDF corresponding to the new virtual function. This is required for SR-IOV to work. 2) PCI hot-plug support when devices are hot added/removed. Vt-d context entries and data structures in Xen needs to be modified accordingly. These hooks needs to be enabled before either of these functions are performed - I assume after dom0 boots. Can you suggest a better way to do this? -----Original Message----- From: Jeremy Fitzhardinge [mailto:jeremy@goop.org] Sent: Friday, July 17, 2009 4:14 PM To: Han, Weidong Cc: ''xen-devel@lists.xensource.com''; ''Keir Fraser''; Kay, Allen M; Jiang, Yunhong Subject: Re: [Xen-devel] [pvops-dom0] [patch] Hook Linux''s PCI probe and remove callbacks On 07/15/09 22:15, Han, Weidong wrote:> Hijack the pci_bus_type probe and remove callbacks. This option only > requires modification to the Xen specific part of Linux. > > This is useful to add and remove pci device to Xen hypervisor when > load and remove its driver. For example, when VFs are created by PF, > they will be added to Xen hypervisor, and then can be assigned to guest. > > This patch is based on xen-tip-master branch of jeremy''s pv-ops tree. >Yunhong Jiang sent me a version of this patch as part of his MSI work. This kind of interception of pci_bus_type.probe is pretty ugly, and is unlikely to be accepted upstream in this form (the patch may only touch a Xen file, but it is changing a variable belonging to the PCI subsystem). We need to work with the Linux PCI maintainers to find some other way of achieving what we need here. Can you explain what the functional requirements are here. Could we defer registering the device with Xen until some other convenient hook, or must it be done at this point in the code? Thanks, J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jiang, Yunhong
2009-Jul-19 11:55 UTC
RE: [Xen-devel] [pvops-dom0] [patch] Hook Linux''s PCI probe and remove callbacks
One consideration to is to split them into two item: a) For all device that is not hot-pluged, we don't need this hypercall, instead, Xen will do the scan and setup the mapping. b) For those hot-plug device and SR-IOV devices, maybe we can use the BUS_NOTIFY_ADD_DEVICE notifier to achieve ths purpose. We can just add a notifier to it. I assume this is simlar to IOMMU in kernel, which should requires this notify also. So a Xen specific notifier should achieve our purpose without change to the PCI bus layer. I checked the kernel code, and noticed that arch/x86/kernel/amd_iommu.c register the notifier for BUS_NOTIFY_ADD_DEVICE, but a bit strange why Intel IOMMU has no such support, do you know if VT-d in kernel support device hotplug already? Allen/Jeremy, any suggestion? Thanks Yunhong Jiang>-----Original Message----- >From: Kay, Allen M >Sent: 2009年7月18日 7:43 >To: Jeremy Fitzhardinge; Han, Weidong >Cc: 'xen-devel@lists.xensource.com'; 'Keir Fraser'; Jiang, Yunhong >Subject: RE: [Xen-devel] [pvops-dom0] [patch] Hook Linux's PCI >probe and remove callbacks > >These hooks for two purposes: > >1) Enabling of a SR-IOV virtual function. The hypercall will >tell xen to setup VT-d context entries for the BDF >corresponding to the new virtual function. This is required >for SR-IOV to work. > >2) PCI hot-plug support when devices are hot added/removed. >Vt-d context entries and data structures in Xen needs to be >modified accordingly. > >These hooks needs to be enabled before either of these >functions are performed - I assume after dom0 boots. > >Can you suggest a better way to do this? > >-----Original Message----- >From: Jeremy Fitzhardinge [mailto:jeremy@goop.org] >Sent: Friday, July 17, 2009 4:14 PM >To: Han, Weidong >Cc: 'xen-devel@lists.xensource.com'; 'Keir Fraser'; Kay, Allen >M; Jiang, Yunhong >Subject: Re: [Xen-devel] [pvops-dom0] [patch] Hook Linux's PCI >probe and remove callbacks > >On 07/15/09 22:15, Han, Weidong wrote: >> Hijack the pci_bus_type probe and remove callbacks. This option only >> requires modification to the Xen specific part of Linux. >> >> This is useful to add and remove pci device to Xen hypervisor when >> load and remove its driver. For example, when VFs are created by PF, >> they will be added to Xen hypervisor, and then can be >assigned to guest. >> >> This patch is based on xen-tip-master branch of jeremy's pv-ops tree. >> > >Yunhong Jiang sent me a version of this patch as part of his MSI work. >This kind of interception of pci_bus_type.probe is pretty ugly, and is >unlikely to be accepted upstream in this form (the patch may only touch >a Xen file, but it is changing a variable belonging to the PCI >subsystem). We need to work with the Linux PCI maintainers to >find some >other way of achieving what we need here. > >Can you explain what the functional requirements are here. Could we >defer registering the device with Xen until some other convenient hook, >or must it be done at this point in the code? > >Thanks, > J >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Han, Weidong
2009-Jul-20 09:03 UTC
RE: [Xen-devel] [pvops-dom0] [patch] Hook Linux''s PCI probe and remove callbacks
Jiang, Yunhong wrote:> One consideration to is to split them into two item: > a) For all device that is not hot-pluged, we don't need this > hypercall, instead, Xen will do the scan and setup the mapping. > b) For those hot-plug device and SR-IOV devices, maybe we can use the > BUS_NOTIFY_ADD_DEVICE notifier to achieve ths purpose. We can just > add a notifier to it. I assume this is simlar to IOMMU in kernel, > which should requires this notify also. So a Xen specific notifier > should achieve our purpose without change to the PCI bus layer.This method looks good. It only needs to register a notifier call for Xen. It should be more possible to be accepted by upstream. Regards, Weidong> > I checked the kernel code, and noticed that > arch/x86/kernel/amd_iommu.c register the notifier for > BUS_NOTIFY_ADD_DEVICE, but a bit strange why Intel IOMMU has no such > support, do you know if VT-d in kernel support device hotplug > already? > > Allen/Jeremy, any suggestion? > > Thanks > Yunhong Jiang > >> -----Original Message----- >> From: Kay, Allen M >> Sent: 2009年7月18日 7:43 >> To: Jeremy Fitzhardinge; Han, Weidong >> Cc: 'xen-devel@lists.xensource.com'; 'Keir Fraser'; Jiang, Yunhong >> Subject: RE: [Xen-devel] [pvops-dom0] [patch] Hook Linux's PCI >> probe and remove callbacks >> >> These hooks for two purposes: >> >> 1) Enabling of a SR-IOV virtual function. The hypercall will >> tell xen to setup VT-d context entries for the BDF >> corresponding to the new virtual function. This is required >> for SR-IOV to work. >> >> 2) PCI hot-plug support when devices are hot added/removed. >> Vt-d context entries and data structures in Xen needs to be >> modified accordingly. >> >> These hooks needs to be enabled before either of these >> functions are performed - I assume after dom0 boots. >> >> Can you suggest a better way to do this? >> >> -----Original Message----- >> From: Jeremy Fitzhardinge [mailto:jeremy@goop.org] >> Sent: Friday, July 17, 2009 4:14 PM >> To: Han, Weidong >> Cc: 'xen-devel@lists.xensource.com'; 'Keir Fraser'; Kay, Allen >> M; Jiang, Yunhong >> Subject: Re: [Xen-devel] [pvops-dom0] [patch] Hook Linux's PCI >> probe and remove callbacks >> >> On 07/15/09 22:15, Han, Weidong wrote: >>> Hijack the pci_bus_type probe and remove callbacks. This option only >>> requires modification to the Xen specific part of Linux. >>> >>> This is useful to add and remove pci device to Xen hypervisor when >>> load and remove its driver. For example, when VFs are created by PF, >>> they will be added to Xen hypervisor, and then can be assigned to >>> guest. >>> >>> This patch is based on xen-tip-master branch of jeremy's pv-ops >>> tree. >>> >> >> Yunhong Jiang sent me a version of this patch as part of his MSI >> work. This kind of interception of pci_bus_type.probe is pretty >> ugly, and is unlikely to be accepted upstream in this form (the >> patch may only touch a Xen file, but it is changing a variable >> belonging to the PCI subsystem). We need to work with the Linux PCI >> maintainers to >> find some >> other way of achieving what we need here. >> >> Can you explain what the functional requirements are here. Could we >> defer registering the device with Xen until some other convenient >> hook, or must it be done at this point in the code? >> >> Thanks, >> J_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeremy Fitzhardinge
2009-Jul-20 17:00 UTC
Re: [Xen-devel] [pvops-dom0] [patch] Hook Linux''s PCI probe and remove callbacks
On 07/17/09 16:42, Kay, Allen M wrote:> These hooks for two purposes: > > 1) Enabling of a SR-IOV virtual function. The hypercall will tell xen to setup VT-d context entries for the BDF corresponding to the new virtual function. This is required for SR-IOV to work. > > 2) PCI hot-plug support when devices are hot added/removed. Vt-d context entries and data structures in Xen needs to be modified accordingly. > > These hooks needs to be enabled before either of these functions are performed - I assume after dom0 boots. > > Can you suggest a better way to do this? >The best person to ask is probably Jesse Barnes (cc:d). In general the way to approach these things is to either find something in the kernel which does something analogous and use the same interfaces, possibly extending them a bit if necessary. J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeremy Fitzhardinge
2009-Jul-20 17:01 UTC
Re: [Xen-devel] [pvops-dom0] [patch] Hook Linux''s PCI probe and remove callbacks
On 07/19/09 04:55, Jiang, Yunhong wrote:> One consideration to is to split them into two item: > a) For all device that is not hot-pluged, we don''t need this hypercall, instead, Xen will do the scan and setup the mapping. > b) For those hot-plug device and SR-IOV devices, maybe we can use the BUS_NOTIFY_ADD_DEVICE notifier to achieve ths purpose. We can just add a notifier to it. I assume this is simlar to IOMMU in kernel, which should requires this notify also. So a Xen specific notifier should achieve our purpose without change to the PCI bus layer. > > I checked the kernel code, and noticed that arch/x86/kernel/amd_iommu.c register the notifier for BUS_NOTIFY_ADD_DEVICE, but a bit strange why Intel IOMMU has no such support, do you know if VT-d in kernel support device hotplug already? > > Allen/Jeremy, any suggestion? >Yes, this is exactly the kind of answer I was looking for. J> Thanks > Yunhong Jiang > > >> -----Original Message----- >> From: Kay, Allen M >> Sent: 2009年7月18日 7:43 >> To: Jeremy Fitzhardinge; Han, Weidong >> Cc: ''xen-devel@lists.xensource.com''; ''Keir Fraser''; Jiang, Yunhong >> Subject: RE: [Xen-devel] [pvops-dom0] [patch] Hook Linux''s PCI >> probe and remove callbacks >> >> These hooks for two purposes: >> >> 1) Enabling of a SR-IOV virtual function. The hypercall will >> tell xen to setup VT-d context entries for the BDF >> corresponding to the new virtual function. This is required >> for SR-IOV to work. >> >> 2) PCI hot-plug support when devices are hot added/removed. >> Vt-d context entries and data structures in Xen needs to be >> modified accordingly. >> >> These hooks needs to be enabled before either of these >> functions are performed - I assume after dom0 boots. >> >> Can you suggest a better way to do this? >> >> -----Original Message----- >> From: Jeremy Fitzhardinge [mailto:jeremy@goop.org] >> Sent: Friday, July 17, 2009 4:14 PM >> To: Han, Weidong >> Cc: ''xen-devel@lists.xensource.com''; ''Keir Fraser''; Kay, Allen >> M; Jiang, Yunhong >> Subject: Re: [Xen-devel] [pvops-dom0] [patch] Hook Linux''s PCI >> probe and remove callbacks >> >> On 07/15/09 22:15, Han, Weidong wrote: >> >>> Hijack the pci_bus_type probe and remove callbacks. This option only >>> requires modification to the Xen specific part of Linux. >>> >>> This is useful to add and remove pci device to Xen hypervisor when >>> load and remove its driver. For example, when VFs are created by PF, >>> they will be added to Xen hypervisor, and then can be >>> >> assigned to guest. >> >>> This patch is based on xen-tip-master branch of jeremy''s pv-ops tree. >>> >>> >> Yunhong Jiang sent me a version of this patch as part of his MSI work. >> This kind of interception of pci_bus_type.probe is pretty ugly, and is >> unlikely to be accepted upstream in this form (the patch may only touch >> a Xen file, but it is changing a variable belonging to the PCI >> subsystem). We need to work with the Linux PCI maintainers to >> find some >> other way of achieving what we need here. >> >> Can you explain what the functional requirements are here. Could we >> defer registering the device with Xen until some other convenient hook, >> or must it be done at this point in the code? >> >> Thanks, >> J >> > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jesse Barnes
2009-Jul-20 17:12 UTC
Re: [Xen-devel] [pvops-dom0] [patch] Hook Linux''s PCI probe and remove callbacks
On Mon, 20 Jul 2009 10:01:37 -0700 Jeremy Fitzhardinge <jeremy@goop.org> wrote:> On 07/19/09 04:55, Jiang, Yunhong wrote: > > One consideration to is to split them into two item: > > a) For all device that is not hot-pluged, we don''t need this > > hypercall, instead, Xen will do the scan and setup the mapping. b) > > For those hot-plug device and SR-IOV devices, maybe we can use the > > BUS_NOTIFY_ADD_DEVICE notifier to achieve ths purpose. We can just > > add a notifier to it. I assume this is simlar to IOMMU in kernel, > > which should requires this notify also. So a Xen specific notifier > > should achieve our purpose without change to the PCI bus layer. > > > > I checked the kernel code, and noticed that > > arch/x86/kernel/amd_iommu.c register the notifier for > > BUS_NOTIFY_ADD_DEVICE, but a bit strange why Intel IOMMU has no > > such support, do you know if VT-d in kernel support device hotplug > > already? > > > > Allen/Jeremy, any suggestion? > > > > Yes, this is exactly the kind of answer I was looking for.Yeah if you can make that work for you it sounds like the best approach. If you have further questions or want some review on it, please cc linux-pci@vger.kernel.org. There''s an active hotplug development community who may be willing to help. Thanks, -- Jesse Barnes, Intel Open Source Technology Center _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel