So yesterday I got was able to get a domU successfully using a PCI SCSI card. This morning I restarted the host server. When I try to start up the same domU I get this error: Error: pci: improper device assignment specified: pci: 0000:0e:04.0 must be co-assigned to the same guest with 0000:0e:04.0, but it is not owned by pciback. How do I make this card owned by pciback on reboot. Do I need to use the hide parameter on the kernel line on boot? Thanks, James _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Tue, 2010-04-27 at 09:05 -0400, James Pifer wrote:> So yesterday I got was able to get a domU successfully using a PCI SCSI > card. This morning I restarted the host server. When I try to start up > the same domU I get this error: > > Error: pci: improper device assignment specified: pci: 0000:0e:04.0 must > be co-assigned to the same guest with 0000:0e:04.0, but it is not owned > by pciback. > > How do I make this card owned by pciback on reboot. Do I need to use the > hide parameter on the kernel line on boot? >Looking at this document: http://wiki.xensource.com/xenwiki/VTdHowTo It says to add something like this to the module line: pciback.hide=(01:00.0)(00:02.0) My kernel version is: 2.6.27.19-5-xen #1 SMP 2009-02-28 04:40:21 +0100 x86_64 x86_64 x86_64 GNU/Linux This page talks about VT and my domU is a PV. Does this same pciback apply? Thanks, James _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Sergio Charpinel Jr.
2010-Apr-27 14:56 UTC
Re: [Xen-users] pci device not owned by pciback.
James, PV machines do not need VT-d to passthrough. You need to use pciback.hide in kernel boot line (if pciback is compiled inside the kernel), or make a script to unbind the device. This site explains how. 2010/4/27 James Pifer <jep@obrien-pifer.com>> On Tue, 2010-04-27 at 09:05 -0400, James Pifer wrote: > > So yesterday I got was able to get a domU successfully using a PCI SCSI > > card. This morning I restarted the host server. When I try to start up > > the same domU I get this error: > > > > Error: pci: improper device assignment specified: pci: 0000:0e:04.0 must > > be co-assigned to the same guest with 0000:0e:04.0, but it is not owned > > by pciback. > > > > How do I make this card owned by pciback on reboot. Do I need to use the > > hide parameter on the kernel line on boot? > > > > Looking at this document: > http://wiki.xensource.com/xenwiki/VTdHowTo > > It says to add something like this to the module line: > pciback.hide=(01:00.0)(00:02.0) > > My kernel version is: > 2.6.27.19-5-xen #1 SMP 2009-02-28 04:40:21 +0100 x86_64 x86_64 x86_64 > GNU/Linux > > This page talks about VT and my domU is a PV. Does this same pciback apply? > > Thanks, > James > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >-- Sergio Roberto Charpinel Jr. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Tue, 2010-04-27 at 11:56 -0300, Sergio Charpinel Jr. wrote:> James, > > > PV machines do not need VT-d to passthrough. > You need to use pciback.hide in kernel boot line (if pciback is > compiled inside the kernel), or make a script to unbind the device. > This site explains how.How can I tell if it''s compiled inside the kernel already? That''s the way I would like to do it. Thanks, James _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi James,>> Error: pci: improper device assignment specified: pci: 0000:0e:04.0 must >> be co-assigned to the same guest with 0000:0e:04.0, but it is not owned >> by pciback.What Xen version are you using? Are you using a pvops based kernel? Or a ''classic'' xen kernel based on 2.6.18+patches? Are you sure it mentions the exact same id twice? It is more likely to report two almost identical ids, with a small difference in the last digit... (e.g. 0000:0e:04.1 must be co-assigned with 0000:0e:04.0)>> How do I make this card owned by pciback on reboot. Do I need to use the >> hide parameter on the kernel line on boot? > > It says to add something like this to the module line: > pciback.hide=(01:00.0)(00:02.0)For older kernel versions, you should use the command as stated above. For newer versions, the module is renamed to xen-pciback, and you''d state: xen-pciback.hide=(01:00.0)(00:02.0)> This page talks about VT and my domU is a PV. Does this same pciback > apply?Hiding is the same for VTd and for PV. However, the ''co-assigned'' error message is typical for VTd, it doesn''t allow devices on the same PCI bus to be divided over more than one vm. If you are passing through a single device to a PV machine, and xen gives you the above mentioned ''co-assigned'' error, it is probably due to a bug in xen (which appeared a while ago, but has since been fixed). Regards, Mark _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
>>> On 2010/04/27 at 07:05, James Pifer <jep@obrien-pifer.com> wrote: > So yesterday I got was able to get a domU successfully using a PCI SCSI > card. This morning I restarted the host server. When I try to start up > the same domU I get this error: > > Error: pci: improper device assignment specified: pci: 0000:0e:04.0 must > be co-assigned to the same guest with 0000:0e:04.0, but it is not owned > by pciback. > > How do I make this card owned by pciback on reboot. Do I need to use the > hide parameter on the kernel line on boot? >Yes, you need to use the hide parameter on the command line, and you need to make sure that the pciback module is loaded. Also take a look at dmesg and /var/log/messages output and see if there is any indication in those locations why pciback is having trouble taking ownership of the device. You also need to make sure that dom0 does not have any drivers loaded for the device - you should be able to use the verbose output of lspci to see what drivers are loaded for a particular card and unload them. -Nick -------- This e-mail may contain confidential and privileged material for the sole use of the intended recipient. If this email is not intended for you, or you are not responsible for the delivery of this message to the intended recipient, please note that this message may contain SEAKR Engineering (SEAKR) Privileged/Proprietary Information. In such a case, you are strictly prohibited from downloading, photocopying, distributing or otherwise using this message, its contents or attachments in any way. If you have received this message in error, please notify us immediately by replying to this e-mail and delete the message from your mailbox. Information contained in this message that does not relate to the business of SEAKR is neither endorsed by nor attributable to SEAKR. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> What Xen version are you using? Are you using a pvops based kernel? Or a > ''classic'' xen kernel based on 2.6.18+patches? > Are you sure it mentions the exact same id twice? It is more likely to > report two almost identical ids, with a small difference in the last > digit... (e.g. 0000:0e:04.1 must be co-assigned with 0000:0e:04.0)That was a copy and paste of the output, and double checked. It says the same one. There is another as you suspected. Running xen 3.4 for SLES11. Not sure this is enough to answer your question: Linux test03 2.6.27.19-5-xen #1 SMP 2009-02-28 04:40:21 +0100 x86_64 x86_64 x86_64 GNU/Linux> Hiding is the same for VTd and for PV. However, the ''co-assigned'' error > message is typical for VTd, it doesn''t allow devices on the same PCI bus > to be divided over more than one vm. > If you are passing through a single device to a PV machine, and xen gives > you the above mentioned ''co-assigned'' error, it is probably due to a bug > in xen (which appeared a while ago, but has since been fixed).This sounds like the bug. Thanks, James _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Tue, 2010-04-27 at 17:04 +0200, Mark Hurenkamp wrote:> Hi James, > > > >> Error: pci: improper device assignment specified: pci: 0000:0e:04.0 must > >> be co-assigned to the same guest with 0000:0e:04.0, but it is not owned > >> by pciback. > What Xen version are you using? Are you using a pvops based kernel? Or a > ''classic'' xen kernel based on 2.6.18+patches? > Are you sure it mentions the exact same id twice? It is more likely to > report two almost identical ids, with a small difference in the last > digit... (e.g. 0000:0e:04.1 must be co-assigned with 0000:0e:04.0) > > >> How do I make this card owned by pciback on reboot. Do I need to use the > >> hide parameter on the kernel line on boot? > > > > It says to add something like this to the module line: > > pciback.hide=(01:00.0)(00:02.0) > For older kernel versions, you should use the command as stated above. For > newer versions, the module is renamed to xen-pciback, and you''d state: > xen-pciback.hide=(01:00.0)(00:02.0) > > > This page talks about VT and my domU is a PV. Does this same pciback > > apply? > Hiding is the same for VTd and for PV. However, the ''co-assigned'' error > message is typical for VTd, it doesn''t allow devices on the same PCI bus > to be divided over more than one vm. > If you are passing through a single device to a PV machine, and xen gives > you the above mentioned ''co-assigned'' error, it is probably due to a bug > in xen (which appeared a while ago, but has since been fixed).I tried adding this to my modules line: module /boot/vmlinuz-2.6.27.19-5-xen root=/dev/disk/by-id/cciss-3600508b1001030364643393433300000-part2 resume=/dev/disk/by-id/cciss-3600508b1001030364643393433300000-part1 splash=silent showopts vga=0x317 pciback.hide=(0e:04.0)(0e:04.1) When the system came up it looks like the host server still took control of the scsi card: 0e:04.0 SCSI storage controller: Atto Technology Ultra320 SCSI Host Adapter (rev 08) Subsystem: Atto Technology ExpressPCI UL5D Low Profile Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 30 I/O ports at 5000 [size=256] Memory at fafc0000 (64-bit, non-prefetchable) [size=256K] Memory at faf80000 (64-bit, non-prefetchable) [size=256K] [virtual] Expansion ROM at e8100000 [disabled] [size=1M] Capabilities: [50] Power Management version 2 Capabilities: [58] Message Signalled Interrupts: Mask- 64bit+ Count=1/1 Enable- Capabilities: [68] PCI-X non-bridge device Kernel driver in use: mptspi Kernel modules: mptspi 0e:04.1 SCSI storage controller: Atto Technology Ultra320 SCSI Host Adapter (rev 08) Subsystem: Atto Technology ExpressPCI UL5D Low Profile Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 37 I/O ports at 5400 [size=256] Memory at faf40000 (64-bit, non-prefetchable) [size=256K] Memory at faf00000 (64-bit, non-prefetchable) [size=256K] [virtual] Expansion ROM at e8200000 [disabled] [size=1M] Capabilities: [50] Power Management version 2 Capabilities: [58] Message Signalled Interrupts: Mask- 64bit+ Count=1/1 Enable- Capabilities: [68] PCI-X non-bridge device Kernel driver in use: mptspi Kernel modules: mptspi I tried unloading mptspi and it unloads, but I still can''t start the virtual machine assigned those cards. Same error as before: Error: pci: improper device assignment specified: pci: 0000:0e:04.0 must be co-assigned to the same guest with 0000:0e:04.0, but it is not owned by pciback. It''s weird that it worked yesterday, but the difference was we plugged in the DLT drive AFTER the hosts system was already running. Looked at dmesg: # dmesg | grep pciback Command line: root=/dev/disk/by-id/cciss-3600508b1001030364643393433300000-part2 resume=/dev/disk/by-id/cciss-3600508b1001030364643393433300000-part1 splash=silent vga=0x317 pciback.hide=(0e:04.0)(0e:04.1) Kernel command line: root=/dev/disk/by-id/cciss-3600508b1001030364643393433300000-part2 resume=/dev/disk/by-id/cciss-3600508b1001030364643393433300000-part1 splash=silent vga=0x317 pciback.hide=(0e:04.0)(0e:04.1) Unknown boot option `pciback.hide=(0e:04.0)(0e:04.1)'': ignoring So then I tried xen-pciback: # dmesg | grep pciback Command line: root=/dev/disk/by-id/cciss-3600508b1001030364643393433300000-part2 resume=/dev/disk/by-id/cciss-3600508b1001030364643393433300000-part1 splash=silent vga=0x317 xen-pciback.hide=(0e:04.0)(0e:04.1) Kernel command line: root=/dev/disk/by-id/cciss-3600508b1001030364643393433300000-part2 resume=/dev/disk/by-id/cciss-3600508b1001030364643393433300000-part1 splash=silent vga=0x317 xen-pciback.hide=(0e:04.0)(0e:04.1) Unknown boot option `xen-pciback.hide=(0e:04.0)(0e:04.1)'': ignoring Does this mean it is not compiled in the kernel? That would really suck. What would my next step be? Thanks, James _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
search keyword co-assigned here -> http://wiki.xensource.com/xenwiki/XenPCIpassthrough James Pifer wrote:> On Tue, 2010-04-27 at 17:04 +0200, Mark Hurenkamp wrote: > >> Hi James, >> >> >> >>>> Error: pci: improper device assignment specified: pci: 0000:0e:04.0 must >>>> be co-assigned to the same guest with 0000:0e:04.0, but it is not owned >>>> by pciback. >>>> >> What Xen version are you using? Are you using a pvops based kernel? Or a >> ''classic'' xen kernel based on 2.6.18+patches? >> Are you sure it mentions the exact same id twice? It is more likely to >> report two almost identical ids, with a small difference in the last >> digit... (e.g. 0000:0e:04.1 must be co-assigned with 0000:0e:04.0) >> >> >>>> How do I make this card owned by pciback on reboot. Do I need to use the >>>> hide parameter on the kernel line on boot? >>>> >>> It says to add something like this to the module line: >>> pciback.hide=(01:00.0)(00:02.0) >>> >> For older kernel versions, you should use the command as stated above. For >> newer versions, the module is renamed to xen-pciback, and you''d state: >> xen-pciback.hide=(01:00.0)(00:02.0) >> >> >>> This page talks about VT and my domU is a PV. Does this same pciback >>> apply? >>> >> Hiding is the same for VTd and for PV. However, the ''co-assigned'' error >> message is typical for VTd, it doesn''t allow devices on the same PCI bus >> to be divided over more than one vm. >> If you are passing through a single device to a PV machine, and xen gives >> you the above mentioned ''co-assigned'' error, it is probably due to a bug >> in xen (which appeared a while ago, but has since been fixed). >> > > > I tried adding this to my modules line: > module /boot/vmlinuz-2.6.27.19-5-xen root=/dev/disk/by-id/cciss-3600508b1001030364643393433300000-part2 resume=/dev/disk/by-id/cciss-3600508b1001030364643393433300000-part1 splash=silent showopts vga=0x317 pciback.hide=(0e:04.0)(0e:04.1) > > When the system came up it looks like the host server still took control > of the scsi card: > 0e:04.0 SCSI storage controller: Atto Technology Ultra320 SCSI Host Adapter (rev 08) > Subsystem: Atto Technology ExpressPCI UL5D Low Profile > Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 30 > I/O ports at 5000 [size=256] > Memory at fafc0000 (64-bit, non-prefetchable) [size=256K] > Memory at faf80000 (64-bit, non-prefetchable) [size=256K] > [virtual] Expansion ROM at e8100000 [disabled] [size=1M] > Capabilities: [50] Power Management version 2 > Capabilities: [58] Message Signalled Interrupts: Mask- 64bit+ Count=1/1 Enable- > Capabilities: [68] PCI-X non-bridge device > Kernel driver in use: mptspi > Kernel modules: mptspi > > 0e:04.1 SCSI storage controller: Atto Technology Ultra320 SCSI Host Adapter (rev 08) > Subsystem: Atto Technology ExpressPCI UL5D Low Profile > Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 37 > I/O ports at 5400 [size=256] > Memory at faf40000 (64-bit, non-prefetchable) [size=256K] > Memory at faf00000 (64-bit, non-prefetchable) [size=256K] > [virtual] Expansion ROM at e8200000 [disabled] [size=1M] > Capabilities: [50] Power Management version 2 > Capabilities: [58] Message Signalled Interrupts: Mask- 64bit+ Count=1/1 Enable- > Capabilities: [68] PCI-X non-bridge device > Kernel driver in use: mptspi > Kernel modules: mptspi > > I tried unloading mptspi and it unloads, but I still can''t start the > virtual machine assigned those cards. Same error as before: > Error: pci: improper device assignment specified: pci: 0000:0e:04.0 must be co-assigned to the same guest with 0000:0e:04.0, but it is not owned by pciback. > > It''s weird that it worked yesterday, but the difference was we plugged > in the DLT drive AFTER the hosts system was already running. > > Looked at dmesg: > # dmesg | grep pciback > Command line: root=/dev/disk/by-id/cciss-3600508b1001030364643393433300000-part2 resume=/dev/disk/by-id/cciss-3600508b1001030364643393433300000-part1 splash=silent vga=0x317 pciback.hide=(0e:04.0)(0e:04.1) > Kernel command line: root=/dev/disk/by-id/cciss-3600508b1001030364643393433300000-part2 resume=/dev/disk/by-id/cciss-3600508b1001030364643393433300000-part1 splash=silent vga=0x317 pciback.hide=(0e:04.0)(0e:04.1) > Unknown boot option `pciback.hide=(0e:04.0)(0e:04.1)'': ignoring > > So then I tried xen-pciback: > # dmesg | grep pciback > Command line: root=/dev/disk/by-id/cciss-3600508b1001030364643393433300000-part2 resume=/dev/disk/by-id/cciss-3600508b1001030364643393433300000-part1 splash=silent vga=0x317 xen-pciback.hide=(0e:04.0)(0e:04.1) > Kernel command line: root=/dev/disk/by-id/cciss-3600508b1001030364643393433300000-part2 resume=/dev/disk/by-id/cciss-3600508b1001030364643393433300000-part1 splash=silent vga=0x317 xen-pciback.hide=(0e:04.0)(0e:04.1) > Unknown boot option `xen-pciback.hide=(0e:04.0)(0e:04.1)'': ignoring > > Does this mean it is not compiled in the kernel? That would really suck. > > What would my next step be? > > Thanks, > James > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Tue, 2010-04-27 at 12:58 -0400, listmail wrote:> search keyword co-assigned here -> > http://wiki.xensource.com/xenwiki/XenPCIpassthrough >I''m looking at what you sent, but... I''m actually passing (or trying to) pass both ports on the card to the same domU. Plus, this worked yesterday, but we plugged the DLT drive in after the host was already booted. Could this have made a difference? Plus, the domU is a PV not an HVM, so why does VT come into play? Or is that because passthrough is a VT "thing"? Here''s what I did next. 1) We unplugged the DLT drive and rebooted the host again. When it came up I got the same error. 2) Next from virt-manager I removed and re-added the PCI cards to passthrough. Now the system starts, but bombs right at the scsi card part of the boot. 3) Next I removed the domU, added the following to the config, then readded the domU: extra="iommu=soft swiotlb=128 " Boot the domU and get the same thing. 4) Next I removed the domU, added the following to the config, then readded the domU: extra="iommu=soft swiotlb=force " Now I boot the domU and everything works. Does that makes sense? Do you still think that fits into the stuff you linked me to? Thanks, James _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi James,>> What Xen version are you using? Are you using a pvops based kernel? >> Or a >> ''classic'' xen kernel based on 2.6.18+patches?Could you please answer this? It would help to know what your Xen and kernel are based on. Do you build your own Xen & kernel? Or are you using a Xen and kernel version provided by your distro?> I tried unloading mptspi and it unloads, but I still can''t start the > virtual machine assigned those cards. Same error as before: > Error: pci: improper device assignment specified: pci: 0000:0e:04.0 > must be co-assigned to the same guest with 0000:0e:04.0, but it is > not owned by pciback.Yeah, it is not sufficient to unload only. You still have to assign the devices to pciback: echo -n "0000:0e:04.0" > /sys/bus/pci/drivers/pciback/new_slot echo -n "0000:0e:04.1" > /sys/bus/pci/drivers/pciback/new_slot echo -n "0000:0e:04.0" > /sys/bus/pci/drivers/pciback/bind echo -n "0000:0e:04.1" > /sys/bus/pci/drivers/pciback/bind> Unknown boot option `pciback.hide=(0e:04.0)(0e:04.1)'': ignoring > Unknown boot option `xen-pciback.hide=(0e:04.0)(0e:04.1)'': ignoring > > Does this mean it is not compiled in the kernel? That would really > suck.It does seem that way... or worse, perhaps your kernel doesn''t have pciback at all... You can try to load it manually and then run the echo commands listed above.> What would my next step be?If you can''t manually load pciback, and the echo commands above fail, then all i can think of is to try a different kernel... Regards, Mark. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> Could you please answer this? It would help to know what your Xen and > kernel are based on. Do you build your own Xen & kernel? Or are you > using > a Xen and kernel version provided by your distro?Yes, provided by the distro. Distro comes with xen 3.3, but I''m running xen 3.4 as provided for sles11.> > > I tried unloading mptspi and it unloads, but I still can''t start the > > virtual machine assigned those cards. Same error as before: > > Error: pci: improper device assignment specified: pci: 0000:0e:04.0 > > must be co-assigned to the same guest with 0000:0e:04.0, but it is > > not owned by pciback. > Yeah, it is not sufficient to unload only. You still have to assign > the devices to pciback: > echo -n "0000:0e:04.0" > /sys/bus/pci/drivers/pciback/new_slot > echo -n "0000:0e:04.1" > /sys/bus/pci/drivers/pciback/new_slot > echo -n "0000:0e:04.0" > /sys/bus/pci/drivers/pciback/bind > echo -n "0000:0e:04.1" > /sys/bus/pci/drivers/pciback/bindI will try these, but see my other response. The whole thing seems screwy to me. Thanks, James _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> Yeah, getting device passthrough to work properly for pvops kernel > can be tricky. The iommu and swiotlb magic sound familiar, it took > some time before i got my ivtv cards working properly as well... > > Good to hear you got it to work now.Well it''s working, but not solved. I can''t go through all those steps every time. I will need to figure out how to get this to work on boot. I will try the commands you wrote in the other response and see what happens. Maybe I will open a ticket with Novell too. Thanks, James _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Sergio Charpinel Jr.
2010-Apr-28 14:49 UTC
Re: [Xen-users] pci device not owned by pciback.
James, To get this working on boot, you need to set the pciback.hide parameter in kernel boot line. But pciback must but compiled inside the kernel (not as module). If you are using CentOS or redhat, probably it is compiled as module. You could check $(uname -r)-config file and see the options. CONFIG_XEN_PCIDEV_BACKEND=y in xen kernel (dont know in pv_ops). If these commands are working for you, just make a script for it. 2010/4/27 James Pifer <jep@obrien-pifer.com>> > Yeah, getting device passthrough to work properly for pvops kernel > > can be tricky. The iommu and swiotlb magic sound familiar, it took > > some time before i got my ivtv cards working properly as well... > > > > Good to hear you got it to work now. > > Well it''s working, but not solved. I can''t go through all those steps > every time. I will need to figure out how to get this to work on boot. I > will try the commands you wrote in the other response and see what > happens. Maybe I will open a ticket with Novell too. > > Thanks, > James > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >-- Sergio Roberto Charpinel Jr. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Wed, 2010-04-28 at 11:49 -0300, Sergio Charpinel Jr. wrote:> James, > > > To get this working on boot, you need to set the pciback.hide > parameter in kernel boot line. But pciback must but compiled inside > the kernel (not as module). If you are using CentOS or redhat, > probably it is compiled as module. > You could check $(uname -r)-config file and see the options. > CONFIG_XEN_PCIDEV_BACKEND=y in xen kernel (dont know in pv_ops). >How do I check this? # echo $(uname -r)-config 2.6.27.45-0.1-xen-config I cannot find a file on my system by this name. Thanks, James _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Wed, 2010-04-28 at 11:49 -0300, Sergio Charpinel Jr. wrote:> James, > > > To get this working on boot, you need to set the pciback.hide > parameter in kernel boot line. But pciback must but compiled inside > the kernel (not as module). If you are using CentOS or redhat, > probably it is compiled as module. > You could check $(uname -r)-config file and see the options. > CONFIG_XEN_PCIDEV_BACKEND=y in xen kernel (dont know in pv_ops). >Ok, I think I found it: # cat /boot/config-2.6.27.45-0.1-xen | grep CONFIG_XEN_PCIDEV_BACKEND CONFIG_XEN_PCIDEV_BACKEND=m CONFIG_XEN_PCIDEV_BACKEND_VPCI=y # CONFIG_XEN_PCIDEV_BACKEND_PASS is not set # CONFIG_XEN_PCIDEV_BACKEND_SLOT is not set # CONFIG_XEN_PCIDEV_BACKEND_CONTROLLER is not set So it looks like it''s compiled as a module. I added this to /etc/modprobe.conf.local (which is what SLES wants you to use): # hide pci card for xen passthrough options pciback hide=(0000:0e:04.0) options pciback hide=(0000:0e:04.1) install mptspi /sbin/modprobe pciback ; /sbin/modprobe --first-time --ignore-install mptspi mptspi is the driver that was being used on the card at the host level. Just need to try rebooting and see what happens. Please let me know if anyone sees any issues with this. Thanks, James _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> Ok, I think I found it: > > # cat /boot/config-2.6.27.45-0.1-xen | grep CONFIG_XEN_PCIDEV_BACKEND > CONFIG_XEN_PCIDEV_BACKEND=m > CONFIG_XEN_PCIDEV_BACKEND_VPCI=y > # CONFIG_XEN_PCIDEV_BACKEND_PASS is not set > # CONFIG_XEN_PCIDEV_BACKEND_SLOT is not set > # CONFIG_XEN_PCIDEV_BACKEND_CONTROLLER is not set > > > So it looks like it''s compiled as a module. I added this > to /etc/modprobe.conf.local (which is what SLES wants you to use): > > # hide pci card for xen passthrough > options pciback hide=(0000:0e:04.0) > options pciback hide=(0000:0e:04.1) > install mptspi /sbin/modprobe pciback ; /sbin/modprobe --first-time --ignore-install mptspi > > mptspi is the driver that was being used on the card at the host level. > Just need to try rebooting and see what happens. > > Please let me know if anyone sees any issues with this.Tried it. Doesn''t look like it worked. When we rebooted the server mptspi is still the kernel driver on the card, not pciback. Anyone see what I may have missed? Thanks, James 0e:04.0 SCSI storage controller: Atto Technology Ultra320 SCSI Host Adapter (rev 08) Subsystem: Atto Technology ExpressPCI UL5D Low Profile Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 30 I/O ports at 5000 [size=256] Memory at fafc0000 (64-bit, non-prefetchable) [size=256K] Memory at faf80000 (64-bit, non-prefetchable) [size=256K] [virtual] Expansion ROM at e8100000 [disabled] [size=1M] Capabilities: [50] Power Management version 2 Capabilities: [58] Message Signalled Interrupts: Mask- 64bit+ Count=1/1 Enable- Capabilities: [68] PCI-X non-bridge device Kernel driver in use: mptspi Kernel modules: mptspi 0e:04.1 SCSI storage controller: Atto Technology Ultra320 SCSI Host Adapter (rev 08) Subsystem: Atto Technology ExpressPCI UL5D Low Profile Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 37 I/O ports at 5400 [size=256] Memory at faf40000 (64-bit, non-prefetchable) [size=256K] Memory at faf00000 (64-bit, non-prefetchable) [size=256K] [virtual] Expansion ROM at e8200000 [disabled] [size=1M] Capabilities: [50] Power Management version 2 Capabilities: [58] Message Signalled Interrupts: Mask- 64bit+ Count=1/1 Enable- Capabilities: [68] PCI-X non-bridge device Kernel driver in use: mptspi Kernel modules: mptspi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Sergio Charpinel Jr.
2010-Apr-28 22:14 UTC
Re: [Xen-users] pci device not owned by pciback.
I think you forgot to make the new initrd with this options. A time ago, I tried something similar to what you did, but i didnt work here. I didnt get deep into this, so I recompiled the kernel. Maybe I missed something too. 2010/4/28 James Pifer <jep@obrien-pifer.com>> > Ok, I think I found it: > > > > # cat /boot/config-2.6.27.45-0.1-xen | grep CONFIG_XEN_PCIDEV_BACKEND > > CONFIG_XEN_PCIDEV_BACKEND=m > > CONFIG_XEN_PCIDEV_BACKEND_VPCI=y > > # CONFIG_XEN_PCIDEV_BACKEND_PASS is not set > > # CONFIG_XEN_PCIDEV_BACKEND_SLOT is not set > > # CONFIG_XEN_PCIDEV_BACKEND_CONTROLLER is not set > > > > > > So it looks like it''s compiled as a module. I added this > > to /etc/modprobe.conf.local (which is what SLES wants you to use): > > > > # hide pci card for xen passthrough > > options pciback hide=(0000:0e:04.0) > > options pciback hide=(0000:0e:04.1) > > install mptspi /sbin/modprobe pciback ; /sbin/modprobe --first-time > --ignore-install mptspi > > > > mptspi is the driver that was being used on the card at the host level. > > Just need to try rebooting and see what happens. > > > > Please let me know if anyone sees any issues with this. > > Tried it. Doesn''t look like it worked. When we rebooted the > server mptspi is still the kernel driver on the card, not pciback. > Anyone see what I may have missed? > > Thanks, > James > > 0e:04.0 SCSI storage controller: Atto Technology Ultra320 SCSI Host Adapter > (rev 08) > Subsystem: Atto Technology ExpressPCI UL5D Low Profile > Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 30 > I/O ports at 5000 [size=256] > Memory at fafc0000 (64-bit, non-prefetchable) [size=256K] > Memory at faf80000 (64-bit, non-prefetchable) [size=256K] > [virtual] Expansion ROM at e8100000 [disabled] [size=1M] > Capabilities: [50] Power Management version 2 > Capabilities: [58] Message Signalled Interrupts: Mask- 64bit+ > Count=1/1 Enable- > Capabilities: [68] PCI-X non-bridge device > Kernel driver in use: mptspi > Kernel modules: mptspi > > 0e:04.1 SCSI storage controller: Atto Technology Ultra320 SCSI Host Adapter > (rev 08) > Subsystem: Atto Technology ExpressPCI UL5D Low Profile > Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 37 > I/O ports at 5400 [size=256] > Memory at faf40000 (64-bit, non-prefetchable) [size=256K] > Memory at faf00000 (64-bit, non-prefetchable) [size=256K] > [virtual] Expansion ROM at e8200000 [disabled] [size=1M] > Capabilities: [50] Power Management version 2 > Capabilities: [58] Message Signalled Interrupts: Mask- 64bit+ > Count=1/1 Enable- > Capabilities: [68] PCI-X non-bridge device > Kernel driver in use: mptspi > Kernel modules: mptspi > > > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >-- Sergio Roberto Charpinel Jr. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Wed, 2010-04-28 at 19:14 -0300, Sergio Charpinel Jr. wrote:> I think you forgot to make the new initrd with this options. > A time ago, I tried something similar to what you did, but i didnt > work here. I didnt get deep into this, so I recompiled the kernel. > Maybe I missed something too. >I thought looking at that config file that it meant that''s the way it was configured, so updating modules.conf was all I needed. Can you elaborate on making the new initrd? Thanks, James _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Wed, 2010-04-28 at 20:42 -0400, James Pifer wrote:> On Wed, 2010-04-28 at 19:14 -0300, Sergio Charpinel Jr. wrote: > > I think you forgot to make the new initrd with this options. > > A time ago, I tried something similar to what you did, but i didnt > > work here. I didnt get deep into this, so I recompiled the kernel. > > Maybe I missed something too. > > > > > I thought looking at that config file that it meant that''s the way it > was configured, so updating modules.conf was all I needed. Can you > elaborate on making the new initrd?Still not sure what to do here. In my /boot directory I have this: -rw-r--r-- 1 root root 88117 2009-02-28 02:28 config-2.6.27.19-5-xen -rw-r--r-- 1 root root 6255788 2010-04-19 14:13 initrd-2.6.27.19-5-xen lrwxrwxrwx 1 root root 22 2010-04-19 14:13 initrd-xen -> initrd-2.6.27.19-5-xen In /boot/grub/menu.lst I have: title Xen -- SUSE Linux Enterprise Server 11 - 2.6.27.19-5 root (hd0,1) kernel /boot/xen.gz dom0_mem=2048M module /boot/vmlinuz-2.6.27.19-5-xen root=/dev/disk/by-id/cciss-3600508b1001030364643393433300000-part2 resume=/dev/disk/by-id/cciss-3600508b1001030364643393433300000-part1 splash=silent showopts vga=0x317 module /boot/initrd-2.6.27.19-5-xen uname -a gives: Linux test03 2.6.27.19-5-xen #1 SMP 2009-02-28 04:40:21 +0100 x86_64 x86_64 x86_64 GNU/Linux Inside config-2.6.27.19-5-xen have the lines: CONFIG_XEN_PCIDEV_BACKEND=m CONFIG_XEN_PCIDEV_BACKEND_VPCI=y # CONFIG_XEN_PCIDEV_BACKEND_PASS is not set # CONFIG_XEN_PCIDEV_BACKEND_SLOT is not set # CONFIG_XEN_PCIDEV_BACKEND_CONTROLLER is not set /etc/modprobe.conf.local has: # hide pci card for xen passthrough options pciback hide=(0000:0e:04.0) options pciback hide=(0000:0e:04.1) install mptspi /sbin/modprobe pciback ; /sbin/modprobe --first-time --ignore-install mptspi So, doesn''t this mean that the initrd being used is already configured to use pciback as a module, so updating modules.conf (or modules.conf.local on sles11) like I have should be enough? Help is appreciated. Boss is breathing down my neck! Thanks, James _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Sergio Charpinel Jr.
2010-Apr-29 14:49 UTC
Re: [Xen-users] pci device not owned by pciback.
James, Taking a look at mkinitrd manpage: --preload=moduleLoad the module module in the initial ramdisk image. The module gets loaded before any SCSI modules which are specified in /etc/modules.conf. This option may be used as many times as necessary. mkinitrd --preload=pciback IMAGENAME KERNELVERSION Specify in /etc/modules.conf your SCSI driver. Your pciback options should be in modprobe.conf as well. Maybe I''m missing some point, but you should try this. If it do not work, maybe you can think in recompile your kernel with pciback inside your kernel. Hope this helps. -- Sergio Roberto Charpinel Jr. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Thu, 2010-04-29 at 11:49 -0300, Sergio Charpinel Jr. wrote:> James, > Taking a look at mkinitrd manpage: > --preload=moduleLoad the module module in the initial ramdisk image. > The module gets loaded before any SCSI modules which are specified in > /etc/modules.conf. This option may be used as many times as necessary. > > mkinitrd --preload=pciback IMAGENAME KERNELVERSION > > Specify in /etc/modules.conf your SCSI driver. Your pciback options > should be in modprobe.conf as well. Maybe I''m missing some point, but > you should try this. > > If it do not work, maybe you can think in recompile your kernel with > pciback inside your kernel. > > Hope this helps.Sergio, Thanks for the response. I don''t think you''re really missing anything, except my lack of experience in this area. You say "you should try this". I''m not exactly sure what to try. So with the system booted up would I do the following steps? # cp initrd-2.6.27.19-5-xen initrd-2.6.27.19-5-xen.save # mkinitrd --preload=pciback initrd-2.6.27.19-5-xen 2.6.27.19-5-xen Then reboot after modules.conf is updated? Recompiling the kernel is not something I really want to get into. We''re looking at having a lot of servers that need to be able to do this. Thanks, James _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Sergio Charpinel Jr.
2010-May-03 11:47 UTC
Re: [Xen-users] pci device not owned by pciback.
Ok James, I think that you forgot some steps. Do this, renaming your pci device, initrd and kernel names: # echo "options pciback hide=(41:00.0)" >> /etc/modprobe.conf # cd /boot # cp initrd-2.6.27.19-5-xen initrd-2.6.27.19-5-xen.save # mkinitrd -f --with=pciback --preload=pciback initrd-2.6.27.19-5-xen 2.6.27.19-5-xen You could also try to specify your SCSI driver name in /etc/modules.conf, before making the image. Hope this helps. 2010/4/29 James Pifer <jep@obrien-pifer.com>:> On Thu, 2010-04-29 at 11:49 -0300, Sergio Charpinel Jr. wrote: >> James, >> Taking a look at mkinitrd manpage: >> --preload=moduleLoad the module module in the initial ramdisk image. >> The module gets loaded before any SCSI modules which are specified in >> /etc/modules.conf. This option may be used as many times as necessary. >> >> mkinitrd --preload=pciback IMAGENAME KERNELVERSION >> >> Specify in /etc/modules.conf your SCSI driver. Your pciback options >> should be in modprobe.conf as well. Maybe I''m missing some point, but >> you should try this. >> >> If it do not work, maybe you can think in recompile your kernel with >> pciback inside your kernel. >> >> Hope this helps. > > > Sergio, > > Thanks for the response. I don''t think you''re really missing anything, > except my lack of experience in this area. You say "you should try > this". I''m not exactly sure what to try. > > So with the system booted up would I do the following steps? > > # cp initrd-2.6.27.19-5-xen initrd-2.6.27.19-5-xen.save > # mkinitrd --preload=pciback initrd-2.6.27.19-5-xen 2.6.27.19-5-xen > > Then reboot after modules.conf is updated? > > Recompiling the kernel is not something I really want to get into. We''re > looking at having a lot of servers that need to be able to do this. > > Thanks, > James > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >-- Sergio Roberto Charpinel Jr. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Mon, 2010-05-03 at 08:47 -0300, Sergio Charpinel Jr. wrote:> Ok James, > > I think that you forgot some steps. Do this, renaming your pci device, > initrd and kernel names: > > # echo "options pciback hide=(41:00.0)" >> /etc/modprobe.conf > # cd /boot > # cp initrd-2.6.27.19-5-xen initrd-2.6.27.19-5-xen.save > # mkinitrd -f --with=pciback --preload=pciback initrd-2.6.27.19-5-xen > 2.6.27.19-5-xen > > You could also try to specify your SCSI driver name in > /etc/modules.conf, before making the image. > > Hope this helps.Sergio, I appreciate all your help. I was able to get this working this weekend and was going to update this thread this morning. The steps are very close to what you have above: Add the following to /etc/modprobe.conf.local (or modprobe.conf on non-SLES systems): options pciback hide=(0000:0e:04.0)(0000:0e:04.1) Edit /etc/sysconfig/kernel and add ''pciback'' to the INITRD_MODULES line. Rebuild the initrd using `mkinitrd` (just run mkinitrd as root) #find the device driver that might have the card right now find /sys/bus/pci/drivers -name 0000:0e:04.0 Based on results of find, add these lines to /etc/init.d/after.boot: echo -n "0000:0e:04.0" > "/sys/bus/pci/drivers/mptspi/unbind" echo -n "0000:0e:04.1" > "/sys/bus/pci/drivers/mptspi/unbind" echo -n "0000:0e:04.0" > /sys/bus/pci/drivers/pciback/new_slot echo -n "0000:0e:04.0" > /sys/bus/pci/drivers/pciback/bind echo -n "0000:0e:04.1" > /sys/bus/pci/drivers/pciback/new_slot echo -n "0000:0e:04.1" > /sys/bus/pci/drivers/pciback/bind When the system boots up I can now successfully start the virtual machine and it gets the PCI card. Woohoo!!! As a side note, I really don''t care much for SLES''s after.local. It doesn''t seem to work quite as well as Redhat''s rc.local, but thankfully these commands do work. I''m also not sure what will happen if xendomains starts my virtual machine, because I don''t know if after.local has run yet. Thanks again for all of your help!!!! James _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users