Xen devel - Apr 2009 - [PATCH 0/3] RFC: PCIe IO space multiplexing for bootable pass through HVM domain

Hi all. This is for PCIe IO space multiplexing. Please comment.

It is not uncommon that a big iron for server consolidation has
many (e.g. > 16) PCIe slots. It will hold many domains, and
the administrator wants them to boot from pass through devices.
But currently up to 16 hvm domains can boot from pass through device.

This patch series addresses this issue by multiplexing IO space access.
The patches are composed of
	Linux part: IO space ressignment code and multiplexing driver
	xen part:   udev script for the driver
	ioemu part: make use of the PCIe io space multiplexing driver


Usage:
Add dom0 kernel command line specifying the pass through devices by
 
guestiomuldev=[<segment>:]<bus>:<dev>[,[<segment:><bus>:dev]][,...]
(Don''t forget add related options. pciback.hide, reassign_resources,
reassigndev or guestdev.)


Details:
PCI expansion ROM BIOS often uses IO port access to boot from its device
and Linux as dom0 exclusively assigns IO space to downstream PCI bridges
and the assignment unit of PCI bridge IO space is 4K. So the only up to
16 PCIe device can be accessed via IO space within 64K IO ports.
So on virtualized environment, it means only up to 16 guest domains
can boot from such pass-through devices.

Limitation:
PCI devices or root complex integrated endpoints aren''t supported.
IO port of IO shared devices can''t be accessed from dom0 Linux device
driver.
But this wouldn''t be a big issue because PCIe specification discourages
the use of IO space and recommends that IO space should be used only
for bootable device with ROM code. OS device driver should work without
IO space access.

Test:
At present I have tested with only single multifunction PCIe because
I don''t have a machine with complicated PCIe topology nor many PCIe
cards.

thanks	

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Thu, 2 Apr 2009 13:01:28 +0900
Isaku Yamahata <yamahata@valinux.co.jp> wrote:
> 
> Hi all. This is for PCIe IO space multiplexing. Please comment.
> 
> It is not uncommon that a big iron for server consolidation has
> many (e.g. > 16) PCIe slots. It will hold many domains, and
> the administrator wants them to boot from pass through devices.
> But currently up to 16 hvm domains can boot from pass through device.
> 
> This patch series addresses this issue by multiplexing IO space access.
> The patches are composed of
> 	Linux part: IO space ressignment code and multiplexing driver
> 	xen part:   udev script for the driver
> 	ioemu part: make use of the PCIe io space multiplexing driver
> 
> 
> Usage:
> Add dom0 kernel command line specifying the pass through devices by
>  
guestiomuldev=[<segment>:]<bus>:<dev>[,[<segment:><bus>:dev]][,...]
> (Don''t forget add related options. pciback.hide,
reassign_resources,
> reassigndev or guestdev.)
It is nice to add "iomul" flag to "guestdev=" boot
parameter, because
segment# and bus# can be changed on rebooting. In addition to this,
specifying pciback.hide, reassigndev and guestioemuldev is redundant.

This is a example.

        guestdev=PNP0A03:0-02.0+iomul,PNP0A03:0-1e.0+iomul


Incidentally, I will submit the patch to use device path to assign
devices to guest domain as discussed in xen-devel, after xen 3.4 is
released.

Thanks,
--
Yuji Shimada


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Thu, Apr 02, 2009 at 04:12:56PM +0900, Yuji Shimada
wrote:
> On Thu, 2 Apr 2009 13:01:28 +0900
> Isaku Yamahata <yamahata@valinux.co.jp> wrote:
> 
> > 
> > Hi all. This is for PCIe IO space multiplexing. Please comment.
> > 
> > It is not uncommon that a big iron for server consolidation has
> > many (e.g. > 16) PCIe slots. It will hold many domains, and
> > the administrator wants them to boot from pass through devices.
> > But currently up to 16 hvm domains can boot from pass through device.
> > 
> > This patch series addresses this issue by multiplexing IO space
access.
> > The patches are composed of
> > 	Linux part: IO space ressignment code and multiplexing driver
> > 	xen part:   udev script for the driver
> > 	ioemu part: make use of the PCIe io space multiplexing driver
> > 
> > 
> > Usage:
> > Add dom0 kernel command line specifying the pass through devices by
> >  
guestiomuldev=[<segment>:]<bus>:<dev>[,[<segment:><bus>:dev]][,...]
> > (Don''t forget add related options. pciback.hide,
reassign_resources,
> > reassigndev or guestdev.)
> 
> It is nice to add "iomul" flag to "guestdev=" boot
parameter, because
> segment# and bus# can be changed on rebooting. In addition to this,
> specifying pciback.hide, reassigndev and guestioemuldev is redundant.
> 
> This is a example.
> 
>         guestdev=PNP0A03:0-02.0+iomul,PNP0A03:0-1e.0+iomul
That sounds a good idea.
I haven''t addressed the option integration yet because I''d
like to verify
the functionality itself first.

My IO sharing code handles devices in the unit of device (not function),
so ".<func>" isn''t necessary. The parser will be
modified slightly.

thanks,
> Incidentally, I will submit the patch to use device path to assign
> devices to guest domain as discussed in xen-devel, after xen 3.4 is
> released.
> 
> Thanks,
-- 
yamahata

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel