All,
I would like to run the following issue past the list to confirm that my
problem analysis makes sense or whether I missed something:
We have a HP ML370G6 server with VT-x and VT-d enabled in the BIOS. It is
equipped with two Smart Array HBAs - one connected to the HDD array
(SmartArray P800) and one connected to an LTO tape drive (SmartArray
P212). Both controllers use the same driver on the dom0 (cciss).
dom0 OS = CentOS 5.3 Linux 2.6.18-164.el5xen x86_64
Xen = 3.4.1
My plan was to use PCI passthrough to give a domU unrestricted access to
the tape drive. However, I have been unable to get pciback to seize the
P212 controller at boot time.
Firstly, I modified the initrd to preload the pciback module:
# mkinitrd -f --preload=pciback /boot/initrd-$(uname -r).img $(uname -r)
Then I added the following lines to modprobe.conf:
options pciback hide=(0000:10:00.0)
install cciss /sbin/modprobe pciback ; /sbin/modprobe --first-time
--ignore-install cciss
grub.conf:
title CentOS (2.6.18-164.el5xen)
root (hd0,0)
kernel /boot/xen.gz-3.4.1 iommu=1
module /boot/vmlinuz-2.6.18-164.el5xen ro root=LABEL=/
module /boot/initrd-2.6.18-164.el5xen.img
NB: Passing pciback.hide=(0000:10:00.0) as a kernel parameter does not
work:
kernel: Unknown boot option `pciback.hide=(10:00.0)'': ignoring
I believe this is because the pciback driver is compiled as a module in
the CentOS stock kernel.
After booting with the above config the cciss driver has taken control of
PCI device 000:10:00.0 instead of the pciback module.
It seems the pciback module is loaded after the HBA driver (cciss) and
therefore does not get a chance to seize the Smart Array P212 controller.
This is most likely because the pciback driver is compiled as a module
instead of into the kernel. The cciss driver is loaded first to gain
access to the hard drive array (via the P800 controller) and to load the
pciback module and its configuration. At that point it is too late for
pciback to gain control of the Smart Array Controller as the cciss driver
has already bound to it.
I belive to fix this I would have to recompile the CentOS kernel with the
pciback compiled in statically. Is that correct or is there any other way?
Thanks
Matthias
SOUTH PACIFIC SEEDS
PO Box 934, GRIFFITH N.S.W. 2680. AUSTRALIA
Phone: 02 6962 7333 Fax: 02 6964 1311
CAUTION: The information contained in this e-mail is privileged and
confidential. If you are reading this message and you are not the intended
recipient, you are hereby notified that any use, dissemination,
distribution or reproduction of all or part of the contents is prohibited.
If you receive this message in error, please notify the sender
immediately. Any opinions or views expressed in this message are those of
the individual sender and may not represent those of their employer.
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
Matthias Reif wrote:> > All, > > I would like to run the following issue past the list to confirm that my > problem analysis makes sense or whether I missed something: > > We have a HP ML370G6 server with VT-x and VT-d enabled in the BIOS. It > is equipped with two Smart Array HBAs - one connected to the HDD array > (SmartArray P800) and one connected to an LTO tape drive (SmartArray > P212). Both controllers use the same driver on the dom0 (cciss). > > dom0 OS = CentOS 5.3 Linux 2.6.18-164.el5xen x86_64 > Xen = 3.4.1 > > My plan was to use PCI passthrough to give a domU unrestricted access to > the tape drive. However, I have been unable to get pciback to seize the > P212 controller at boot time. > > Firstly, I modified the initrd to preload the pciback module: > > # mkinitrd -f --preload=pciback /boot/initrd-$(uname -r).img $(uname -r) > > Then I added the following lines to modprobe.conf: > > options pciback hide=(0000:10:00.0) > install cciss /sbin/modprobe pciback ; /sbin/modprobe --first-time > --ignore-install cciss > > grub.conf: > > title CentOS (2.6.18-164.el5xen) > root (hd0,0) > kernel /boot/xen.gz-3.4.1 iommu=1 > module /boot/vmlinuz-2.6.18-164.el5xen ro root=LABEL=/ > module /boot/initrd-2.6.18-164.el5xen.img > > > NB: Passing pciback.hide=(0000:10:00.0) as a kernel parameter does not > work: > > kernel: Unknown boot option `pciback.hide=(10:00.0)'': ignoring > > I believe this is because the pciback driver is compiled as a module in > the CentOS stock kernel. > > > After booting with the above config the cciss driver has taken control > of PCI device 000:10:00.0 instead of the pciback module. > > > It seems the pciback module is loaded after the HBA driver (cciss) and > therefore does not get a chance to seize the Smart Array P212 > controller. This is most likely because the pciback driver is compiled > as a module instead of into the kernel. The cciss driver is loaded first > to gain access to the hard drive array (via the P800 controller) and to > load the pciback module and its configuration. At that point it is too > late for pciback to gain control of the Smart Array Controller as the > cciss driver has already bound to it. > > I belive to fix this I would have to recompile the CentOS kernel with > the pciback compiled in statically. Is that correct or is there any > other way? > > Thanks > MatthiasUsing the pciback kernel or module option is not the only way to have pciback seize a device. You could use calls to the sysfs structure to unbind the device from the cciss driver and then assign it to pciback. Something along those lines might work after loading pciback: echo -n ''0000:10:00.0'' > /sys/bus/pci/drivers/cciss/unbind echo -n ''0000:10:00.0'' > /sys/bus/pci/drivers/pciback/new_slot echo -n ''0000:10:00.0'' > /sys/bus/pci/drivers/pciback/bind Make sure you are not using any of the resources (i.e. exclude this is vgscan or automounts) to avoid unexpected results... Best regards, Christian _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
xen-users-bounces@lists.xensource.com wrote on 30/09/2009 07:05:39 PM:> Matthias Reif wrote: > > > > All, > > > > I would like to run the following issue past the list to confirm thatmy> > problem analysis makes sense or whether I missed something: > > > > We have a HP ML370G6 server with VT-x and VT-d enabled in the BIOS. It> > is equipped with two Smart Array HBAs - one connected to the HDD array> > (SmartArray P800) and one connected to an LTO tape drive (SmartArray > > P212). Both controllers use the same driver on the dom0 (cciss). > > > > dom0 OS = CentOS 5.3 Linux 2.6.18-164.el5xen x86_64 > > Xen = 3.4.1 > > > > My plan was to use PCI passthrough to give a domU unrestricted accessto> > the tape drive. However, I have been unable to get pciback to seizethe> > P212 controller at boot time. > > > > Firstly, I modified the initrd to preload the pciback module: > > > > # mkinitrd -f --preload=pciback /boot/initrd-$(uname -r).img $(uname-r)> > > > Then I added the following lines to modprobe.conf: > > > > options pciback hide=(0000:10:00.0) > > install cciss /sbin/modprobe pciback ; /sbin/modprobe --first-time > > --ignore-install cciss > > > > grub.conf: > > > > title CentOS (2.6.18-164.el5xen) > > root (hd0,0) > > kernel /boot/xen.gz-3.4.1 iommu=1 > > module /boot/vmlinuz-2.6.18-164.el5xen ro root=LABEL=/ > > module /boot/initrd-2.6.18-164.el5xen.img > > > > > > NB: Passing pciback.hide=(0000:10:00.0) as a kernel parameter does not> > work: > > > > kernel: Unknown boot option `pciback.hide=(10:00.0)'': ignoring > > > > I believe this is because the pciback driver is compiled as a modulein> > the CentOS stock kernel. > > > > > > After booting with the above config the cciss driver has taken control> > of PCI device 000:10:00.0 instead of the pciback module. > > > > > > It seems the pciback module is loaded after the HBA driver (cciss) and> > therefore does not get a chance to seize the Smart Array P212 > > controller. This is most likely because the pciback driver is compiled> > as a module instead of into the kernel. The cciss driver is loadedfirst> > to gain access to the hard drive array (via the P800 controller) andto> > load the pciback module and its configuration. At that point it is too> > late for pciback to gain control of the Smart Array Controller as the > > cciss driver has already bound to it. > > > > I belive to fix this I would have to recompile the CentOS kernel with > > the pciback compiled in statically. Is that correct or is there any > > other way? > > > > > Using the pciback kernel or module option is not the only way to have > pciback seize a device. You could use calls to the sysfs structure to > unbind the device from the cciss driver and then assign it to pciback. > > Something along those lines might work after loading pciback: > > echo -n ''0000:10:00.0'' > /sys/bus/pci/drivers/cciss/unbind > echo -n ''0000:10:00.0'' > /sys/bus/pci/drivers/pciback/new_slot > echo -n ''0000:10:00.0'' > /sys/bus/pci/drivers/pciback/bind > > Make sure you are not using any of the resources (i.e. exclude this is > vgscan or automounts) to avoid unexpected results... >Thanks for your suggestion. I tried your approach, but for some reason the unbind command freezes. There is nothing in the logs that indicates why. Thanks Matthias SOUTH PACIFIC SEEDS PO Box 934, GRIFFITH N.S.W. 2680. AUSTRALIA Phone: 02 6962 7333 Fax: 02 6964 1311 CAUTION: The information contained in this e-mail is privileged and confidential. If you are reading this message and you are not the intended recipient, you are hereby notified that any use, dissemination, distribution or reproduction of all or part of the contents is prohibited. If you receive this message in error, please notify the sender immediately. Any opinions or views expressed in this message are those of the individual sender and may not represent those of their employer. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users