Hi, I’ve been trying to configure Xeon on my HP ML350 G4 server for the past two weeks and despite reading just about every word of the Xen wiki and numerous other posts and mails, I can’t find a solution to my problem. Any help on this would be much appreciated! Setup: HP ML350 G4, Dual xeon, 6Gb Ram, 6 disk scsi raid array, Digium TDM410P analogue PBX card on PCI. Hardware virtualization (Vt-d) is not an option with this machine. I followed these instructions (more or less) to set up Dom0 and DomU: http://www.howtoforge.com/virtualization-with-xen-on-centos-6.3-x86_64-paravirtualization-and-hardware-virtualization with the following changes: Host Dom0 (Centos 6.4): xen-4.2.2-4.el6.x86_64 kernel-xen-3.9.2-1.el6xen.x86_64 libvirt 1.0.3-1 (python-virtinstall causes libvirt to be upgraded to 1.0.3. From checking the source, the Xen patch appears to be there already, so no recompile needed – the Xen patch doesn’t work with this source anyway. XEND has been disabled from boot up because it causes problems with XL tools although the same address space collision occurs if I use the XM tool set. I have tried xen-pciback.hide(06:01.0) on the kernel module definitions in boot.conf but this doesn’t seem to do anything. Adding records to modprobe.conf and rc.local work better. The device I’m trying to passthrough is defined: 06:01.0 Ethernet controller: Digium, Inc. Wildcard TDM410 4-port analog card (rev 11) Subsystem: Digium, Inc. Wildcard TDM410 4-port analog card Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin A routed to IRQ 16 Region 0: I/O ports at 5000 [disabled] [size=256] Region 1: Memory at fdef0000 (32-bit, non-prefetchable) [disabled] [size=1K] [virtual] Expansion ROM at f0000000 [disabled] [size=128K] Capabilities: [c0] Power Management version 2 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=100mA PME(D0+,D1+,D2+,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Kernel driver in use: pciback Guest DomU (Centos 6.4): kernel-xen-3.9.2-1.el6xen.x86_64 Created using virt-install onto a 20G LVM with 1024Mb ram XML for DomU dumped and converted to native then the domain destroyed and undefined and recreated using XL create with the new cfg file. This is to allow inclusion of pci [‘06:01.0’] parameter in config. Using the static setup, I can get Dom0 to hide the PCI device. I can also achieve the same effect with the dynamic set up using pci-assignable-attach and pci-attach. Here is the dmesg relating to the device. Reg 30 is highlighted because this seems to be where the problem is. pci 0000:06:01.0: [d161:8005] type 00 class 0x020000 pci 0000:06:01.0: reg 10: [io 0x5000-0x50ff] pci 0000:06:01.0: reg 14: [mem 0xfdef0000-0xfdef03ff] pci 0000:06:01.0: reg 30: [mem 0x00000000-0x0001ffff pref] pci 0000:06:01.0: supports D1 D2 pci 0000:06:01.0: PME# supported from D0 D1 D2 D3hot D3cold pci 0000:06:01.0: BAR 6: assigned [mem 0xf0000000-0xf001ffff pref] pciback 0000:06:01.0: seizing device pciback 0000:06:01.0: PCI IRQ 48 -> rerouted to legacy IRQ 16 pciback 0000:06:01.0: PCI IRQ 48 -> rerouted to legacy IRQ 16 xen-pciback: vpci: 0000:06:01.0: assign to virtual slot 0 In the Dom0 I can define the device statically in the config file or dynamically as described above. Both scenarios result in the same error being displayed. pcifront pci-0: Installing PCI frontend pcifront pci-0: Creating PCI Frontend Bus 0000:00 pcifront pci-0: PCI host bridge to bus 0000:00 pci_bus 0000:00: root bus resource [io 0x0000-0xffff] pci_bus 0000:00: root bus resource [mem 0x00000000-0xfffffffff] pci_bus 0000:00: root bus resource [bus 00-ff] pci 0000:00:00.0: [d161:8005] type 00 class 0x020000 pci 0000:00:00.0: reg 10: [io 0x5000-0x50ff] pci 0000:00:00.0: reg 14: [mem 0xfdef0000-0xfdef03ff] pci 0000:00:00.0: reg 30: [mem 0xf0000000-0xffffffff pref] pci 0000:00:00.0: supports D1 D2 pcifront pci-0: claiming resource 0000:00:00.0/0 pcifront pci-0: claiming resource 0000:00:00.0/1 pcifront pci-0: claiming resource 0000:00:00.0/6 pci 0000:00:00.0: address space collision: [mem 0xf0000000-0xffffffff pref] conflicts with 0000:00:00.0 [mem 0xfdef0000-0xfdef03ff] pcifront pci-0: Could not claim resource 0000:00:00.0/6! Device offline. Try using e820_host=1 in the guest config. This appears to show that the PCI device is conflicting with itself (reg 14 with reg 30) because the address space for reg 30 is different in pciback to pcifront. I have tried setting up the domain with both XM and XL with the same result Adding passthrough and permissive settings with no change Adding iommu=soft to guest kernel command line. I’ve tried adding the e820_host flag to the config file but this doesn’t seem to solve anything. Different Xen enabled kernels. Wiping the server and rebuilding the whole thing from scratch (more than once) The Digium PCI card works fine on a normal Centos 6.3 setup with no Xen. I’m out of ideas now on how to solve this, so if anyone has made this card work by doing something different, I’d be grateful for any suggestions. I’ve looked at the source for pcifont.c and come to the conclusion that my c coding skills are not going to be good enough to debug/change this program. I can provide more dmesg outputs or other documentation if needed. Thanks in advance for any help Jon _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
Gordan Bobic
2013-May-21 06:35 UTC
Re: Problem with PCI Pass-through address space collision
I''m pretty sure I seem to recall that PCI passthrough will not work without VT-d, but by all means, feel free to try. Even if you did have working VT-d, though, you have to detach the device from dom0 before you can add it to domU, using something like: virsh nodedev-detach pci_0000_06_01_0 Given the EL6 CRC Xen packages you are using, they use pciback built as a module, so kernel boot parameters won''t help. What you need to do is add this to /etc/modprobe.d/: # cat xen-pciback.conf options xen-pciback permissive=1 hide=(06:01.0) Run depmod -a once you have done that. Then: # modprobe xen-pciback virsh nodedev-detach pci_0000_06_01_0 Also add the driver for the card to /etc/modprobe.d/blacklist.conf. After that you should be able to boot the domU with the device. You may also want to upgrade to the latest testing packages (4.2.2-5) since they include a PCI passthrough fix from a couple of days ago, although it doesn''t look like you are falling foul of it. Also, how much RAM are you passing to domU? Try giving it <= 2GB. There is a PCI memory map bug that can cause a nasty memory stomp that kept me chasing my tail for days. For most people it manifests at > 4GB, but on my system it manifested at > 2GB. HTH. Gordan On 05/20/2013 06:04 PM, Jon Skilling wrote:> Hi, > > I’ve been trying to configure Xeon on my HP ML350 G4 server for the past > two weeks and despite reading just about every word of the Xen wiki and > numerous other posts and mails, I can’t find a solution to my problem. > Any help on this would be much appreciated! > > Setup: > > HP ML350 G4, Dual xeon, 6Gb Ram, 6 disk scsi raid array, Digium TDM410P > analogue PBX card on PCI. Hardware virtualization (Vt-d) is not an > option with this machine. > > I followed these instructions (more or less) to set up Dom0 and DomU: > > http://www.howtoforge.com/virtualization-with-xen-on-centos-6.3-x86_64-paravirtualization-and-hardware-virtualization > > with the following changes: > > Host Dom0 (Centos 6.4): > > xen-4.2.2-4.el6.x86_64 > > kernel-xen-3.9.2-1.el6xen.x86_64 > > libvirt 1.0.3-1 (python-virtinstall causes libvirt to be upgraded to > 1.0.3. From checking the source, the Xen patch appears to be there > already, so no recompile needed – the Xen patch doesn’t work with this > source anyway. > > XEND has been disabled from boot up because it causes problems with XL > tools although the same address space collision occurs if I use the XM > tool set. > > I have tried xen-pciback.hide(06:01.0) on the kernel module definitions > in boot.conf but this doesn’t seem to do anything. Adding records to > modprobe.conf and rc.local work better. > > The device I’m trying to passthrough is defined: > > 06:01.0 Ethernet controller: Digium, Inc. Wildcard TDM410 4-port analog > card (rev 11) > > Subsystem: Digium, Inc. Wildcard TDM410 4-port analog card > > Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- > ParErr- Stepping- SERR- FastB2B- DisINTx- > > Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium > >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- > > Interrupt: pin A routed to IRQ 16 > > Region 0: I/O ports at 5000 [disabled] [size=256] > > Region 1: Memory at fdef0000 (32-bit, non-prefetchable) > [disabled] [size=1K] > > [virtual] Expansion ROM at f0000000 [disabled] [size=128K] > > Capabilities: [c0] Power Management version 2 > > Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=100mA > PME(D0+,D1+,D2+,D3hot+,D3cold+) > > Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- > > Kernel driver in use: pciback > > Guest DomU (Centos 6.4): > > kernel-xen-3.9.2-1.el6xen.x86_64 > > Created using virt-install onto a 20G LVM with 1024Mb ram > > XML for DomU dumped and converted to native then the domain destroyed > and undefined and recreated using XL create with the new cfg file. This > is to allow inclusion of pci [‘06:01.0’] parameter in config. > > Using the static setup, I can get Dom0 to hide the PCI device. I can > also achieve the same effect with the dynamic set up using > pci-assignable-attach and pci-attach. Here is the dmesg relating to the > device. Reg 30 is highlighted because this seems to be where the problem is. > > pci 0000:06:01.0: [d161:8005] type 00 class 0x020000 > > pci 0000:06:01.0: reg 10: [io 0x5000-0x50ff] > > pci 0000:06:01.0: reg 14: [mem 0xfdef0000-0xfdef03ff] > > pci 0000:06:01.0: *reg 30: [mem 0x00000000-0x0001ffff pref]* > > pci 0000:06:01.0: supports D1 D2 > > pci 0000:06:01.0: PME# supported from D0 D1 D2 D3hot D3cold > > pci 0000:06:01.0: BAR 6: assigned [mem 0xf0000000-0xf001ffff pref] > > pciback 0000:06:01.0: seizing device > > pciback 0000:06:01.0: PCI IRQ 48 -> rerouted to legacy IRQ 16 > > pciback 0000:06:01.0: PCI IRQ 48 -> rerouted to legacy IRQ 16 > > xen-pciback: vpci: 0000:06:01.0: assign to virtual slot 0 > > In the Dom0 I can define the device statically in the config file or > dynamically as described above. Both scenarios result in the same error > being displayed. > > pcifront pci-0: Installing PCI frontend > > pcifront pci-0: Creating PCI Frontend Bus 0000:00 > > pcifront pci-0: PCI host bridge to bus 0000:00 > > pci_bus 0000:00: root bus resource [io 0x0000-0xffff] > > pci_bus 0000:00: root bus resource [mem 0x00000000-0xfffffffff] > > pci_bus 0000:00: root bus resource [bus 00-ff] > > pci 0000:00:00.0: [d161:8005] type 00 class 0x020000 > > pci 0000:00:00.0: reg 10: [io 0x5000-0x50ff] > > pci 0000:00:00.0: reg 14: [mem 0xfdef0000-0xfdef03ff] > > pci 0000:00:00.0: *reg 30: [mem 0xf0000000-0xffffffff pref]* > > pci 0000:00:00.0: supports D1 D2 > > pcifront pci-0: claiming resource 0000:00:00.0/0 > > pcifront pci-0: claiming resource 0000:00:00.0/1 > > pcifront pci-0: claiming resource 0000:00:00.0/6 > > pci 0000:00:00.0: address space collision: [mem 0xf0000000-0xffffffff > pref] conflicts with 0000:00:00.0 [mem 0xfdef0000-0xfdef03ff] > > pcifront pci-0: Could not claim resource 0000:00:00.0/6! Device offline. > Try using e820_host=1 in the guest config. > > This appears to show that the PCI device is conflicting with itself (reg > 14 with reg 30) because the address space for reg 30 is different in > pciback to pcifront. > > I have tried setting up the domain with both XM and XL with the same result > > Adding passthrough and permissive settings with no change > > Adding iommu=soft to guest kernel command line. > > I’ve tried adding the e820_host flag to the config file but this doesn’t > seem to solve anything. > > Different Xen enabled kernels. > > Wiping the server and rebuilding the whole thing from scratch (more than > once) > > The Digium PCI card works fine on a normal Centos 6.3 setup with no Xen. > > I’m out of ideas now on how to solve this, so if anyone has made this > card work by doing something different, I’d be grateful for any > suggestions. I’ve looked at the source for pcifont.c and come to the > conclusion that my c coding skills are not going to be good enough to > debug/change this program. > > I can provide more dmesg outputs or other documentation if needed. > > Thanks in advance for any help > > Jon > > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xen.org > http://lists.xen.org/xen-users >
Jon Skilling
2013-May-21 15:33 UTC
Re: Problem with PCI Pass-through address space collision
Hi Gordan, Thanks for your reply. It is my understanding that PCI pass-through should work for PV guests, which is what I am creating here. One thing that I notice though, other than XL DMESG giving the message about Vt-d being disabled, is that other than the command line message, iommu=soft doesn''t seem to produce any confirmation messages anywhere so I can''t really tell if this is running or not. I have to confess that I have not seen any documentation on XL stating that the device must be detached from Dom0 before it can be attached to a guest. Certainly, XL pci-detach freePBX 06:01.0 produces an error saying that the domain is invalid. However, I tried your suggestion prior to starting the domain and it runs without complaint and says that the device has been detached. I reincorporated your suggestion into xen-pciback.conf and added modprobe xen-pciback into rc.local to get everything to start at boot up and restarted the machine. I then ran the virsh-nodev-detach command followed by xl create -c freePBX.cfg. Unfortunately, this produced the same result as before. I''ve also tried adding iommu=soft to the Dom0 kernel line, but as I said, this didn''t produce any discernible differences to the XL DMESG output. Instead of hotplugging, I tried adding the two last lines to the end of the freePBX.cfg file: name = "freePBX" uuid = "91cd5696-1451-c333-f6a4-1628a79976bd" maxmem = 1024 memory = 1024 vcpus = 1 bootloader = "pygrub" localtime = 0 on_poweroff = "destroy" on_reboot = "restart" on_crash = "restart" disk = [ "phy:/dev/VolGroup/freePBX,xvda,w" ] vif = [ "mac=00:16:3e:00:9a:9c,bridge=br0,script=vif-bridge,vifname=vif2.0" ] extra = "iommu=soft debug loglevel=10 earlyprintk=xenboot console=hvc0 ro xencons=tty" pci = [''06:01.0,permissive=1''] As you can see from above, I''m allocating 1024Mb to the guest. I haven''t got time now until the weekend to apply the 4.2.2-5 patches to check if they solve my problem, but I''ll post a message when I''ve done it to update on how it went. Suffice to say, I''m still getting the same error message :-( Thanks for your help and input. Jon -----Original Message----- From: Gordan Bobic [mailto:gordan@bobich.net] Sent: 21 May 2013 07:36 To: Jon Skilling Cc: xen-users@lists.xen.org Subject: Re: [Xen-users] Problem with PCI Pass-through address space collision I''m pretty sure I seem to recall that PCI passthrough will not work without VT-d, but by all means, feel free to try. Even if you did have working VT-d, though, you have to detach the device from dom0 before you can add it to domU, using something like: virsh nodedev-detach pci_0000_06_01_0 Given the EL6 CRC Xen packages you are using, they use pciback built as a module, so kernel boot parameters won''t help. What you need to do is add this to /etc/modprobe.d/: # cat xen-pciback.conf options xen-pciback permissive=1 hide=(06:01.0) Run depmod -a once you have done that. Then: # modprobe xen-pciback virsh nodedev-detach pci_0000_06_01_0 Also add the driver for the card to /etc/modprobe.d/blacklist.conf. After that you should be able to boot the domU with the device. You may also want to upgrade to the latest testing packages (4.2.2-5) since they include a PCI passthrough fix from a couple of days ago, although it doesn''t look like you are falling foul of it. Also, how much RAM are you passing to domU? Try giving it <= 2GB. There is a PCI memory map bug that can cause a nasty memory stomp that kept me chasing my tail for days. For most people it manifests at > 4GB, but on my system it manifested at > 2GB. HTH. Gordan On 05/20/2013 06:04 PM, Jon Skilling wrote:> Hi, > > I''ve been trying to configure Xeon on my HP ML350 G4 server for the > past two weeks and despite reading just about every word of the Xen > wiki and numerous other posts and mails, I can''t find a solution to myproblem.> Any help on this would be much appreciated! > > Setup: > > HP ML350 G4, Dual xeon, 6Gb Ram, 6 disk scsi raid array, Digium > TDM410P analogue PBX card on PCI. Hardware virtualization (Vt-d) is > not an option with this machine. > > I followed these instructions (more or less) to set up Dom0 and DomU: > > http://www.howtoforge.com/virtualization-with-xen-on-centos-6.3-x86_64 > -paravirtualization-and-hardware-virtualization > > with the following changes: > > Host Dom0 (Centos 6.4): > > xen-4.2.2-4.el6.x86_64 > > kernel-xen-3.9.2-1.el6xen.x86_64 > > libvirt 1.0.3-1 (python-virtinstall causes libvirt to be upgraded to > 1.0.3. From checking the source, the Xen patch appears to be there > already, so no recompile needed - the Xen patch doesn''t work with this > source anyway. > > XEND has been disabled from boot up because it causes problems with XL > tools although the same address space collision occurs if I use the XM > tool set. > > I have tried xen-pciback.hide(06:01.0) on the kernel module > definitions in boot.conf but this doesn''t seem to do anything. Adding > records to modprobe.conf and rc.local work better. > > The device I''m trying to passthrough is defined: > > 06:01.0 Ethernet controller: Digium, Inc. Wildcard TDM410 4-port > analog card (rev 11) > > Subsystem: Digium, Inc. Wildcard TDM410 4-port analog card > > Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- > ParErr- Stepping- SERR- FastB2B- DisINTx- > > Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium > >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- > > Interrupt: pin A routed to IRQ 16 > > Region 0: I/O ports at 5000 [disabled] [size=256] > > Region 1: Memory at fdef0000 (32-bit, non-prefetchable) > [disabled] [size=1K] > > [virtual] Expansion ROM at f0000000 [disabled] [size=128K] > > Capabilities: [c0] Power Management version 2 > > Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=100mA > PME(D0+,D1+,D2+,D3hot+,D3cold+) > > Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 > PME- > > Kernel driver in use: pciback > > Guest DomU (Centos 6.4): > > kernel-xen-3.9.2-1.el6xen.x86_64 > > Created using virt-install onto a 20G LVM with 1024Mb ram > > XML for DomU dumped and converted to native then the domain destroyed > and undefined and recreated using XL create with the new cfg file. > This is to allow inclusion of pci [''06:01.0''] parameter in config. > > Using the static setup, I can get Dom0 to hide the PCI device. I can > also achieve the same effect with the dynamic set up using > pci-assignable-attach and pci-attach. Here is the dmesg relating to > the device. Reg 30 is highlighted because this seems to be where theproblem is.> > pci 0000:06:01.0: [d161:8005] type 00 class 0x020000 > > pci 0000:06:01.0: reg 10: [io 0x5000-0x50ff] > > pci 0000:06:01.0: reg 14: [mem 0xfdef0000-0xfdef03ff] > > pci 0000:06:01.0: *reg 30: [mem 0x00000000-0x0001ffff pref]* > > pci 0000:06:01.0: supports D1 D2 > > pci 0000:06:01.0: PME# supported from D0 D1 D2 D3hot D3cold > > pci 0000:06:01.0: BAR 6: assigned [mem 0xf0000000-0xf001ffff pref] > > pciback 0000:06:01.0: seizing device > > pciback 0000:06:01.0: PCI IRQ 48 -> rerouted to legacy IRQ 16 > > pciback 0000:06:01.0: PCI IRQ 48 -> rerouted to legacy IRQ 16 > > xen-pciback: vpci: 0000:06:01.0: assign to virtual slot 0 > > In the Dom0 I can define the device statically in the config file or > dynamically as described above. Both scenarios result in the same > error being displayed. > > pcifront pci-0: Installing PCI frontend > > pcifront pci-0: Creating PCI Frontend Bus 0000:00 > > pcifront pci-0: PCI host bridge to bus 0000:00 > > pci_bus 0000:00: root bus resource [io 0x0000-0xffff] > > pci_bus 0000:00: root bus resource [mem 0x00000000-0xfffffffff] > > pci_bus 0000:00: root bus resource [bus 00-ff] > > pci 0000:00:00.0: [d161:8005] type 00 class 0x020000 > > pci 0000:00:00.0: reg 10: [io 0x5000-0x50ff] > > pci 0000:00:00.0: reg 14: [mem 0xfdef0000-0xfdef03ff] > > pci 0000:00:00.0: *reg 30: [mem 0xf0000000-0xffffffff pref]* > > pci 0000:00:00.0: supports D1 D2 > > pcifront pci-0: claiming resource 0000:00:00.0/0 > > pcifront pci-0: claiming resource 0000:00:00.0/1 > > pcifront pci-0: claiming resource 0000:00:00.0/6 > > pci 0000:00:00.0: address space collision: [mem 0xf0000000-0xffffffff > pref] conflicts with 0000:00:00.0 [mem 0xfdef0000-0xfdef03ff] > > pcifront pci-0: Could not claim resource 0000:00:00.0/6! Device offline. > Try using e820_host=1 in the guest config. > > This appears to show that the PCI device is conflicting with itself > (reg > 14 with reg 30) because the address space for reg 30 is different in > pciback to pcifront. > > I have tried setting up the domain with both XM and XL with the same > result > > Adding passthrough and permissive settings with no change > > Adding iommu=soft to guest kernel command line. > > I''ve tried adding the e820_host flag to the config file but this > doesn''t seem to solve anything. > > Different Xen enabled kernels. > > Wiping the server and rebuilding the whole thing from scratch (more > than > once) > > The Digium PCI card works fine on a normal Centos 6.3 setup with no Xen. > > I''m out of ideas now on how to solve this, so if anyone has made this > card work by doing something different, I''d be grateful for any > suggestions. I''ve looked at the source for pcifont.c and come to the > conclusion that my c coding skills are not going to be good enough to > debug/change this program. > > I can provide more dmesg outputs or other documentation if needed. > > Thanks in advance for any help > > Jon > > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xen.org > http://lists.xen.org/xen-users >
Gordan Bobic
2013-May-23 06:53 UTC
Re: Problem with PCI Pass-through address space collision
On 05/21/2013 04:33 PM, Jon Skilling wrote:> Hi Gordan, > > Thanks for your reply. It is my understanding that PCI pass-through should > work for PV guests, which is what I am creating here. One thing that I > notice though, other than XL DMESG giving the message about Vt-d being > disabled, is that other than the command line message, iommu=soft doesn''t > seem to produce any confirmation messages anywhere so I can''t really tell if > this is running or not. > > I have to confess that I have not seen any documentation on XL stating that > the device must be detached from Dom0 before it can be attached to a guest. > Certainly, XL pci-detach freePBX 06:01.0 produces an error saying that the > domain is invalid. However, I tried your suggestion prior to starting the > domain and it runs without complaint and says that the device has been > detached. > > I reincorporated your suggestion into xen-pciback.conf and added modprobe > xen-pciback into rc.local to get everything to start at boot up and > restarted the machine. > I then ran the virsh-nodev-detach command followed by xl create -c > freePBX.cfg. Unfortunately, this produced the same result as before. > > I''ve also tried adding iommu=soft to the Dom0 kernel line, but as I said, > this didn''t produce any discernible differences to the XL DMESG output. > > Instead of hotplugging, I tried adding the two last lines to the end of the > freePBX.cfg file: > > name = "freePBX" > uuid = "91cd5696-1451-c333-f6a4-1628a79976bd" > maxmem = 1024 > memory = 1024 > vcpus = 1 > bootloader = "pygrub" > localtime = 0 > on_poweroff = "destroy" > on_reboot = "restart" > on_crash = "restart" > disk = [ "phy:/dev/VolGroup/freePBX,xvda,w" ] > vif = [ "mac=00:16:3e:00:9a:9c,bridge=br0,script=vif-bridge,vifname=vif2.0" > ] > > extra = "iommu=soft debug loglevel=10 earlyprintk=xenboot console=hvc0 ro > xencons=tty" > pci = [''06:01.0,permissive=1''] > > As you can see from above, I''m allocating 1024Mb to the guest. I haven''t > got time now until the weekend to apply the 4.2.2-5 patches to check if they > solve my problem, but I''ll post a message when I''ve done it to update on how > it went. Suffice to say, I''m still getting the same error message :-(Have you tried using xm instead of xl? I''m finding that xl is still missing some vital features to make everything work. I''d also start with getting a standard HVM domU working with xm and then take it further from there. I''m pretty sure iommu=soft shouldn''t be on the domU kernel boot parameters. I suggest you start by removing the extra= line and remove the ,permissive=1 from the pci spec since I don''t think xm understands that (you have it in your module config anyway). Then see if you can get it working with xm/xend as a HVM. Gordan
Jon Skilling
2013-May-28 16:41 UTC
Re: Problem with PCI Pass-through address space collision
Hi Gordan, Sorry about the delay in replying. It was bank holiday w/end here and I''ve been away. To answer your question, I''ve already tried building this with XM instead of XL and it produced the same result. However, I did have something of an epiphany whilst I was away and decided to revisit the PCI specs of the motherboard and PBX card. The card is a 32bit 33Mhz card which claims to run correctly in any PCI 2.2 slot or above. I''ve already tested this so I know it''s correct. The slots in the mobo are 64bit, 2 x 133Mhz,1 x 100Mhz, and 1 x 66Mhz and the card was in the 100Mhz slot. It turns out however, that moving the card to the 66Mhz slot solves the problem! It seems a bit silly now that I originally put the card in the 100Mhz slot, but then, it worked fine like that until I introduced xen. Unfortunately, this wasn''t the end of my problems, although it looks like I''m about 99% there... For the sake of interest, Asterisk and freePBX seem to have installed ok, but the dahdi drivers haven''t. This is because I have to compile the drivers over the xen kernel (3.9.3-1.el6xen.x86_64) and each of these driver modules displays a warning about "no private gpg key". It looks like I may need to patch the install scripts to stop this breaking the install but this could be the last step before it starts working! Thanks for your help and suggestions, Jon -----Original Message----- From: Gordan Bobic [mailto:gordan@bobich.net] Sent: 23 May 2013 07:53 To: Jon Skilling Cc: xen-users@lists.xen.org Subject: Re: [Xen-users] Problem with PCI Pass-through address space collision On 05/21/2013 04:33 PM, Jon Skilling wrote:> Hi Gordan, > > Thanks for your reply. It is my understanding that PCI pass-through > should work for PV guests, which is what I am creating here. One > thing that I notice though, other than XL DMESG giving the message > about Vt-d being disabled, is that other than the command line > message, iommu=soft doesn''t seem to produce any confirmation messages > anywhere so I can''t really tell if this is running or not. > > I have to confess that I have not seen any documentation on XL stating > that the device must be detached from Dom0 before it can be attached to aguest.> Certainly, XL pci-detach freePBX 06:01.0 produces an error saying that > the domain is invalid. However, I tried your suggestion prior to > starting the domain and it runs without complaint and says that the > device has been detached. > > I reincorporated your suggestion into xen-pciback.conf and added > modprobe xen-pciback into rc.local to get everything to start at boot > up and restarted the machine. > I then ran the virsh-nodev-detach command followed by xl create -c > freePBX.cfg. Unfortunately, this produced the same result as before. > > I''ve also tried adding iommu=soft to the Dom0 kernel line, but as I > said, this didn''t produce any discernible differences to the XL DMESGoutput.> > Instead of hotplugging, I tried adding the two last lines to the end > of the freePBX.cfg file: > > name = "freePBX" > uuid = "91cd5696-1451-c333-f6a4-1628a79976bd" > maxmem = 1024 > memory = 1024 > vcpus = 1 > bootloader = "pygrub" > localtime = 0 > on_poweroff = "destroy" > on_reboot = "restart" > on_crash = "restart" > disk = [ "phy:/dev/VolGroup/freePBX,xvda,w" ] vif = [ > "mac=00:16:3e:00:9a:9c,bridge=br0,script=vif-bridge,vifname=vif2.0" > ] > > extra = "iommu=soft debug loglevel=10 earlyprintk=xenboot console=hvc0 > ro xencons=tty" > pci = [''06:01.0,permissive=1''] > > As you can see from above, I''m allocating 1024Mb to the guest. I > haven''t got time now until the weekend to apply the 4.2.2-5 patches to > check if they solve my problem, but I''ll post a message when I''ve done > it to update on how it went. Suffice to say, I''m still getting the > same error message :-(Have you tried using xm instead of xl? I''m finding that xl is still missing some vital features to make everything work. I''d also start with getting a standard HVM domU working with xm and then take it further from there. I''m pretty sure iommu=soft shouldn''t be on the domU kernel boot parameters. I suggest you start by removing the extra= line and remove the ,permissive=1 from the pci spec since I don''t think xm understands that (you have it in your module config anyway). Then see if you can get it working with xm/xend as a HVM. Gordan
Gordan Bobic
2013-May-28 17:28 UTC
Re: Problem with PCI Pass-through address space collision
On 05/28/2013 05:41 PM, Jon Skilling wrote:> Hi Gordan, > > Sorry about the delay in replying. It was bank holiday w/end here and I''ve > been away.No worries, I''ve had a bank holiday weekend, too. :)> To answer your question, I''ve already tried building this with > XM instead of XL and it produced the same result. However, I did have > something of an epiphany whilst I was away and decided to revisit the PCI > specs of the motherboard and PBX card. The card is a 32bit 33Mhz card which > claims to run correctly in any PCI 2.2 slot or above. I''ve already tested > this so I know it''s correct. The slots in the mobo are 64bit, 2 x 133Mhz,1 > x 100Mhz, and 1 x 66Mhz and the card was in the 100Mhz slot. > It turns out however, that moving the card to the 66Mhz slot solves the > problem! It seems a bit silly now that I originally put the card in the > 100Mhz slot, but then, it worked fine like that until I introduced xen.If I had to guess, putting it in a different slot solved the problem because of the way PCI bridges are wired up, not because it is a different speed slot. Can you check: lspci -vt and see if in the old slot you had more than just this card behind the same PCI bridge?> Unfortunately, this wasn''t the end of my problems, although it looks like > I''m about 99% there... For the sake of interest, Asterisk and freePBX seem > to have installed ok, but the dahdi drivers haven''t. This is because I have > to compile the drivers over the xen kernel (3.9.3-1.el6xen.x86_64) and each > of these driver modules displays a warning about "no private gpg key". It > looks like I may need to patch the install scripts to stop this breaking the > install but this could be the last step before it starts working!So, you are down to a mere Linux problem rather than a Xen one. :) Awesome stuff, should be easier from here. ;) Gordan