Lin, Ray
2010-Oct-13 16:49 UTC
[Xen-devel] PCI devices remap from dom0 to domU with pv-ops kernel
Konrad,
We try to remap the pci devices from dom0 to domU by assigning the pci in
the domU configuration file. Such as
pci="[''01:00.0@11'', ''01:00.1@10'',
''04:00.0@13'', ''04:00.1@12'']"
In hvm kernel, the domU would see the pci devices remapped like this,
dom0 pci device 01:00.0 => domU pci device 00:11.0
dom0 pci device 01:00.1 => domU pci device 00:10.0
dom0 pci device 04:00.0 => domU pci device 00:13.0
dom0 pci device 04:00.1 => domU pci device 00:12.0
But pv-ops kernel seems ignore the remapping. Is there any way (kernel
configuration setting ??) to make it work ?
Thanks,
Ray
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2010-Oct-13 18:51 UTC
[Xen-devel] Re: PCI devices remap from dom0 to domU with pv-ops kernel
On Wed, Oct 13, 2010 at 10:49:12AM -0600, Lin, Ray wrote:> > Konrad, > > We try to remap the pci devices from dom0 to domU by assigning the pci in the domU configuration file. Such as > > pci="[''01:00.0@11'', ''01:00.1@10'', ''04:00.0@13'', ''04:00.1@12'']" > > In hvm kernel, the domU would see the pci devices remapped like this, > > dom0 pci device 01:00.0 => domU pci device 00:11.0 > dom0 pci device 01:00.1 => domU pci device 00:10.0 > dom0 pci device 04:00.0 => domU pci device 00:13.0 > dom0 pci device 04:00.1 => domU pci device 00:12.0 > > But pv-ops kernel seems ignore the remapping. Is there any way (kernel configuration setting ??) to make it work ?Not completly. When you compile your Xen PCI backend, you should see an option whether you want Virtual PCI, Passthrough or some other type. By default it will pick the Virtual one so that it just enumerates from zero the PCI devices. You could write your own plugin that would do remaping per your desire.> > Thanks, > Ray > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Lin, Ray
2010-Oct-14 17:26 UTC
[Xen-devel] RE: PCI devices remap from dom0 to domU with pv-ops kernel
Konrad, We choose "Virtual" and the dom0/domU show like this. It doesn''t remap to what we want. You mentioned about "write your own plugin that would do remapping per your desire". Could you elaborate it a bit ? Thanks. Dom0: kaan-18:~# lspci -n | grep 11f8 07:00.0 0c04: 11f8:8032 (rev 08) 07:00.1 0c04: 11f8:8032 (rev 08) 07:00.2 0c04: 11f8:8032 (rev 08) 07:00.3 0c04: 11f8:8032 (rev 08) DomU: kaan-18-dpm:~# lspci -n 00:00.0 0c04: 11f8:8032 (rev 08) 00:00.1 0c04: 11f8:8032 (rev 08) 00:00.2 0c04: 11f8:8032 (rev 08) 00:00.3 0c04: 11f8:8032 (rev 08) -Ray -----Original Message----- From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com] Sent: Wednesday, October 13, 2010 11:52 AM To: Lin, Ray Cc: bruce.edge@gmail.com; xen-devel@lists.xensource.com Subject: Re: PCI devices remap from dom0 to domU with pv-ops kernel On Wed, Oct 13, 2010 at 10:49:12AM -0600, Lin, Ray wrote:> > Konrad, > > We try to remap the pci devices from dom0 to domU by assigning the > pci in the domU configuration file. Such as > > pci="[''01:00.0@11'', ''01:00.1@10'', ''04:00.0@13'', ''04:00.1@12'']" > > In hvm kernel, the domU would see the pci devices remapped like this, > > dom0 pci device 01:00.0 => domU pci device 00:11.0 > dom0 pci device 01:00.1 => domU pci device 00:10.0 > dom0 pci device 04:00.0 => domU pci device 00:13.0 > dom0 pci device 04:00.1 => domU pci device 00:12.0 > > But pv-ops kernel seems ignore the remapping. Is there any way (kernel configuration setting ??) to make it work ?Not completly. When you compile your Xen PCI backend, you should see an option whether you want Virtual PCI, Passthrough or some other type. By default it will pick the Virtual one so that it just enumerates from zero the PCI devices. You could write your own plugin that would do remaping per your desire.> > Thanks, > Ray > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Konrad Rzeszutek Wilk
2010-Oct-14 17:49 UTC
Re: [Xen-devel] RE: PCI devices remap from dom0 to domU with pv-ops kernel
On Thu, Oct 14, 2010 at 11:26:03AM -0600, Lin, Ray wrote:> > Konrad, > > We choose "Virtual" and the dom0/domU show like this. It doesn''t remap to what we want. You mentioned about "write your own plugin that would do remapping per your desire". Could you elaborate it a bit ? Thanks.Sure. Look in drivers/xen/pciback. There are four files of interest: vpci.c slot.c passthrough.c controller.c By default, the vpci.c is selected, where in the domU the devices start from 00:00.0. The passthrough is obvious. The slot.c is where we have only two bus and all the devices are stuck under the slots. So a maximum of 64 devices, and the device function is ignored: This is what it would show up (I think, you might want to double-check this): 00:01.0:0c04: 11f8:8032 (rev 08) 00:02.0:0c04: 11f8:8032 (rev 08) 00:03.0:0c04: 11f8:8032 (rev 08) 00:04.0:0c04: 11f8:8032 (rev 08) The controller, is "PCI "Controller" Backend - virtualize PCI bus topology based on PCI controllers. Devices under the same PCI controller are exposed on the same virtual domain:bus. Within a bus, device slots are virtualized to compact the bus. " (from controller.c) I think based on what you said earlier, you just want to use the "slot" backend: CONFIG_XEN_PCIDEV_BACKEND_SLOT=y> > Dom0: > kaan-18:~# lspci -n | grep 11f8 > 07:00.0 0c04: 11f8:8032 (rev 08) > 07:00.1 0c04: 11f8:8032 (rev 08) > 07:00.2 0c04: 11f8:8032 (rev 08) > 07:00.3 0c04: 11f8:8032 (rev 08) > > DomU: > kaan-18-dpm:~# lspci -n > 00:00.0 0c04: 11f8:8032 (rev 08) > 00:00.1 0c04: 11f8:8032 (rev 08) > 00:00.2 0c04: 11f8:8032 (rev 08) > 00:00.3 0c04: 11f8:8032 (rev 08) > > > -Ray > > -----Original Message----- > From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com] > Sent: Wednesday, October 13, 2010 11:52 AM > To: Lin, Ray > Cc: bruce.edge@gmail.com; xen-devel@lists.xensource.com > Subject: Re: PCI devices remap from dom0 to domU with pv-ops kernel > > On Wed, Oct 13, 2010 at 10:49:12AM -0600, Lin, Ray wrote: > > > > Konrad, > > > > We try to remap the pci devices from dom0 to domU by assigning the > > pci in the domU configuration file. Such as > > > > pci="[''01:00.0@11'', ''01:00.1@10'', ''04:00.0@13'', ''04:00.1@12'']" > > > > In hvm kernel, the domU would see the pci devices remapped like this, > > > > dom0 pci device 01:00.0 => domU pci device 00:11.0 > > dom0 pci device 01:00.1 => domU pci device 00:10.0 > > dom0 pci device 04:00.0 => domU pci device 00:13.0 > > dom0 pci device 04:00.1 => domU pci device 00:12.0 > > > > But pv-ops kernel seems ignore the remapping. Is there any way (kernel configuration setting ??) to make it work ? > > Not completly. > > When you compile your Xen PCI backend, you should see an option whether you want Virtual PCI, Passthrough or some other type. By default it will pick the Virtual one so that it just enumerates from zero the PCI devices. You could write your own plugin that would do remaping per your desire. > > > > Thanks, > > Ray > > > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Lin, Ray
2010-Oct-14 17:54 UTC
RE: [Xen-devel] RE: PCI devices remap from dom0 to domU with pv-ops kernel
Thanks. I''ll look into it. -Ray -----Original Message----- From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com] Sent: Thursday, October 14, 2010 10:49 AM To: Lin, Ray Cc: xen-devel@lists.xensource.com; bruce.edge@gmail.com Subject: Re: [Xen-devel] RE: PCI devices remap from dom0 to domU with pv-ops kernel On Thu, Oct 14, 2010 at 11:26:03AM -0600, Lin, Ray wrote:> > Konrad, > > We choose "Virtual" and the dom0/domU show like this. It doesn''t remap to what we want. You mentioned about "write your own plugin that would do remapping per your desire". Could you elaborate it a bit ? Thanks.Sure. Look in drivers/xen/pciback. There are four files of interest: vpci.c slot.c passthrough.c controller.c By default, the vpci.c is selected, where in the domU the devices start from 00:00.0. The passthrough is obvious. The slot.c is where we have only two bus and all the devices are stuck under the slots. So a maximum of 64 devices, and the device function is ignored: This is what it would show up (I think, you might want to double-check this): 00:01.0:0c04: 11f8:8032 (rev 08) 00:02.0:0c04: 11f8:8032 (rev 08) 00:03.0:0c04: 11f8:8032 (rev 08) 00:04.0:0c04: 11f8:8032 (rev 08) The controller, is "PCI "Controller" Backend - virtualize PCI bus topology based on PCI controllers. Devices under the same PCI controller are exposed on the same virtual domain:bus. Within a bus, device slots are virtualized to compact the bus. " (from controller.c) I think based on what you said earlier, you just want to use the "slot" backend: CONFIG_XEN_PCIDEV_BACKEND_SLOT=y> > Dom0: > kaan-18:~# lspci -n | grep 11f8 > 07:00.0 0c04: 11f8:8032 (rev 08) > 07:00.1 0c04: 11f8:8032 (rev 08) > 07:00.2 0c04: 11f8:8032 (rev 08) > 07:00.3 0c04: 11f8:8032 (rev 08) > > DomU: > kaan-18-dpm:~# lspci -n > 00:00.0 0c04: 11f8:8032 (rev 08) > 00:00.1 0c04: 11f8:8032 (rev 08) > 00:00.2 0c04: 11f8:8032 (rev 08) > 00:00.3 0c04: 11f8:8032 (rev 08) > > > -Ray > > -----Original Message----- > From: Konrad Rzeszutek Wilk [mailto:konrad.wilk@oracle.com] > Sent: Wednesday, October 13, 2010 11:52 AM > To: Lin, Ray > Cc: bruce.edge@gmail.com; xen-devel@lists.xensource.com > Subject: Re: PCI devices remap from dom0 to domU with pv-ops kernel > > On Wed, Oct 13, 2010 at 10:49:12AM -0600, Lin, Ray wrote: > > > > Konrad, > > > > We try to remap the pci devices from dom0 to domU by assigning > > the pci in the domU configuration file. Such as > > > > pci="[''01:00.0@11'', ''01:00.1@10'', ''04:00.0@13'', ''04:00.1@12'']" > > > > In hvm kernel, the domU would see the pci devices remapped like > > this, > > > > dom0 pci device 01:00.0 => domU pci device 00:11.0 > > dom0 pci device 01:00.1 => domU pci device 00:10.0 > > dom0 pci device 04:00.0 => domU pci device 00:13.0 > > dom0 pci device 04:00.1 => domU pci device 00:12.0 > > > > But pv-ops kernel seems ignore the remapping. Is there any way (kernel configuration setting ??) to make it work ? > > Not completly. > > When you compile your Xen PCI backend, you should see an option whether you want Virtual PCI, Passthrough or some other type. By default it will pick the Virtual one so that it just enumerates from zero the PCI devices. You could write your own plugin that would do remaping per your desire. > > > > Thanks, > > Ray > > > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel