thr3ads.net - Xen users - [Xen-users] VTd not showing PCI device in VM [Dec 2008]

If this information is useful, please help other people find it:
Share via:

Rafael Ayala

2008-Dec-10 23:00 UTC

[Xen-users] VTd not showing PCI device in VM

I''m having trouble actually seeing a PCI device in my VMs. 
I''ve
resolved several of my issues using previous posts and using the
VTdHowTo wiki page.  I have both VT and VTd BIOS options enabled.  I
have pciback hiding the devices and xm can list and assign the devices
to VMs.  I don''t see the PCI devices in my VMs though.  I have tried a
windows and a Linux vm, without success. I also made sure I removed the
drivers for the devices to make sure dom0 was not using them.  

Grub command:
title Xen 3.3 (2.6.18.8)
    root (hd0,1)
    kernel /boot/xen-3.3.0.gz vtd=1 iommu=1
    module /boot/vmlinuz-2.6.18.8-xen
root=/dev/disk/by-id/scsi-SATA_ST3250310NS_9SF0LZAP-part2 vga=0x317
resume=/dev/sda1 splash=silent showopts
pciback.hide=(01:00.0)(01:00.1)(06:00.0)
    module /boot/initrd-2.6.18.8-xen

The PCI devices I''m trying to assign:
# lspci
01:00.0 Fibre Channel: Emulex Corporation Saturn-X: LightPulse Fibre
Channel Host Adapter (rev 03)
01:00.1 Fibre Channel: Emulex Corporation Saturn-X: LightPulse Fibre
Channel Host Adapter (rev 03)
06:00.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1068E
PCI-Express Fusion-MPT SAS (rev 08)

Xen says I can assign them
# xm pci-list-assignable-devices
0000:06:00.0
0000:01:00.0 0000:01:00.1

Create my VM:
# xm create w2k3.vxm pci=01:00.0 pci=01:00.1
Using config file "./w2k3.vxm".
Started domain w2k3

I can confirm Xen thinks I have assigned the devices
# xm pci-list w2k3
domain   bus   slot   func
0x0  0x01   0x00    0x0    
0x0  0x01   0x00    0x1    

# xm create rhel5.vxm.vxm pci=01:00.0 pci=01:00.1
Using config file "./rhel5.vxm".
Started domain rhel5

# xm pci-list rhel5
domain   bus   slot   func
0x0  0x06   0x00    0x0    

Again, neither windows (device manager) nor Linux (lspci) VMs can see
the assigned devices.  Below is "xm dmesg" showing VTd is enabled.
Another curious fact is that I can assign the same devices to 2 VMs.  I
would have expected an error, but perhaps it is allowed to share
devices.  Anyone have any thoughts on what I''m missing?


(XEN) Xen version 3.3.0 (root@xyratex.com) (gcc version 4.1.2 20070115
(SUSE Linux)) Wed Dec 10 10:59:53 EST 2008
(XEN) Latest ChangeSet: unavailable
(XEN) Command line: vtd=1 iommu=1
(XEN) Video information:
(XEN)  VGA is text mode 80x25, font 8x16
(XEN)  VBE/DDC methods: none; EDID transfer time: 2 seconds
(XEN)  EDID info not retrieved because no DDC retrieval method detected
(XEN) Disc information:
(XEN)  Found 4 MBR signatures
(XEN)  Found 4 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN)  0000000000000000 - 000000000009cc00 (usable)
(XEN)  000000000009cc00 - 00000000000a0000 (reserved)
(XEN)  00000000000e0000 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 00000000cfef0000 (usable)
(XEN)  00000000cfef0000 - 00000000cff03000 (ACPI data)
(XEN)  00000000cff03000 - 00000000cff04000 (ACPI NVS)
(XEN)  00000000cff04000 - 00000000d0000000 (reserved)
(XEN)  00000000e0000000 - 00000000f0000000 (reserved)
(XEN)  00000000fec00000 - 00000000fec10000 (reserved)
(XEN)  00000000fee00000 - 00000000fee01000 (reserved)
(XEN)  00000000ff000000 - 0000000100000000 (reserved)
(XEN)  0000000100000000 - 0000000230000000 (usable)
(XEN) System RAM: 8190MB (8387120kB)
(XEN) ACPI: RSDP 000F6530, 0024 (r2 PTLTD )
(XEN) ACPI: XSDT CFEFA758, 00CC (r1 SMCSLP SUPERMIC  6040000  LTP
0)
(XEN) ACPI: FACP CFF022FC, 00F4 (r3 INTEL  STOAKLEY  6040000 PTL
3)
(XEN) ACPI: DSDT CFEFD9BC, 48CC (r1  Intel SEABURG   6040000 MSFT
3000001)
(XEN) ACPI: FACS CFF03FC0, 0040
(XEN) ACPI: DMAR CFF023F0, 0108 (r1 Intel  OEMDMAR   6040000 LOHR
1)
(XEN) ACPI: TCPA CFF024F8, 0032 (r1 Intel  STOAKLEY  6040000 LOHR
5A)
(XEN) ACPI: APIC CFF0252A, 00D4 (r1 PTLTD        APIC    6040000  LTP
0)
(XEN) ACPI: MCFG CFF025FE, 003C (r1 PTLTD    MCFG    6040000  LTP
0)
(XEN) ACPI: BOOT CFF0263A, 0028 (r1 PTLTD  $SBFTBL$  6040000  LTP
1)
(XEN) ACPI: SPCR CFF02662, 0050 (r1 PTLTD  $UCRTBL$  6040000 PTL
1)
(XEN) ACPI: ERST CFF026B2, 0590 (r1 SMCI   ERSTTBL   6040000 SMCI
1)
(XEN) ACPI: HEST CFF02C42, 00A8 (r1 SMCI   HESTTBL   6040000 SMCI
1)
(XEN) ACPI: BERT CFF02CEA, 0030 (r1 SMCI   BERTTBL   6040000 SMCI
1)
(XEN) ACPI: EINJ CFF02D1A, 0170 (r1 SMCI   EINJTBL   6040000 SMCI
1)
(XEN) ACPI: SLIC CFF02E8A, 0176 (r1 SMCSLP SUPERMIC  6040000  LTP
0)
(XEN) ACPI: SSDT CFEFC0B3, 025F (r1  PmRef  Cpu0Tst     3000 INTL
20050228)
(XEN) ACPI: SSDT CFEFC00D, 00A6 (r1  PmRef  Cpu7Tst     3000 INTL
20050228)
(XEN) ACPI: SSDT CFEFBF67, 00A6 (r1  PmRef  Cpu6Tst     3000 INTL
20050228)
(XEN) ACPI: SSDT CFEFBEC1, 00A6 (r1  PmRef  Cpu5Tst     3000 INTL
20050228)
(XEN) ACPI: SSDT CFEFBE1B, 00A6 (r1  PmRef  Cpu4Tst     3000 INTL
20050228)
(XEN) ACPI: SSDT CFEFBD75, 00A6 (r1  PmRef  Cpu3Tst     3000 INTL
20050228)
(XEN) ACPI: SSDT CFEFBCCF, 00A6 (r1  PmRef  Cpu2Tst     3000 INTL
20050228)
(XEN) ACPI: SSDT CFEFBC29, 00A6 (r1  PmRef  Cpu1Tst     3000 INTL
20050228)
(XEN) ACPI: SSDT CFEFA824, 1405 (r1  PmRef    CpuPm     3000 INTL
20050228)
(XEN) Xen heap: 14MB (14500kB)
(XEN) Domain heap initialised
(XEN) Processor #0 7:7 APIC version 20
(XEN) Processor #4 7:7 APIC version 20
(XEN) Processor #1 7:7 APIC version 20
(XEN) Processor #5 7:7 APIC version 20
(XEN) Processor #2 7:7 APIC version 20
(XEN) Processor #6 7:7 APIC version 20
(XEN) Processor #3 7:7 APIC version 20
(XEN) Processor #7 7:7 APIC version 20
(XEN) IOAPIC[0]: apic_id 8, version 32, address 0xfec00000, GSI 0-23
(XEN) IOAPIC[1]: apic_id 9, version 32, address 0xfec86000, GSI 24-47
(XEN) IOAPIC[2]: apic_id 10, version 32, address 0xfec89000, GSI 48-71
(XEN) Enabling APIC mode:  Flat.  Using 3 I/O APICs
(XEN) Intel VT-d has been enabled
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Detected 2666.833 MHz processor.
(XEN) HVM: VMX enabled
(XEN) CPU0: Intel(R) Xeon(R) CPU           E5430  @ 2.66GHz stepping 06
(XEN) Booting processor 1/4 eip 8c000
(XEN) CPU1: Intel(R) Xeon(R) CPU           E5430  @ 2.66GHz stepping 06
(XEN) Booting processor 2/1 eip 8c000
(XEN) CPU2: Intel(R) Xeon(R) CPU           E5430  @ 2.66GHz stepping 06
(XEN) Booting processor 3/5 eip 8c000
(XEN) CPU3: Intel(R) Xeon(R) CPU           E5430  @ 2.66GHz stepping 06
(XEN) Booting processor 4/2 eip 8c000
(XEN) CPU4: Intel(R) Xeon(R) CPU           E5430  @ 2.66GHz stepping 06
(XEN) Booting processor 5/6 eip 8c000
(XEN) CPU5: Intel(R) Xeon(R) CPU           E5430  @ 2.66GHz stepping 06
(XEN) Booting processor 6/3 eip 8c000
(XEN) CPU6: Intel(R) Xeon(R) CPU           E5430  @ 2.66GHz stepping 06
(XEN) Booting processor 7/7 eip 8c000
(XEN) CPU7: Intel(R) Xeon(R) CPU           E5430  @ 2.66GHz stepping 06
(XEN) Total of 8 processors activated.
(XEN) ENABLING IO-APIC IRQs
(XEN)  -> Using new ACK method
(XEN) checking TSC synchronization across 8 CPUs: passed.
(XEN) Platform timer is 3.579MHz ACPI PM Timer
(XEN) Brought up 8 CPUs
(XEN) I/O virtualisation enabled
(XEN) I/O virtualisation for PV guests disabled
(XEN) *** LOADING DOMAIN 0 ***
(XEN)  Xen  kernel: 64-bit, lsb, compat32
(XEN)  Dom0 kernel: 64-bit, lsb, paddr 0xffffffff80200000 ->
0xffffffff805c148c
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN)  Dom0 alloc.:   0000000224000000->0000000226000000 (2017483 pages
to be allocated)
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN)  Loaded kernel: ffffffff80200000->ffffffff805c148c
(XEN)  Init. ramdisk: ffffffff805c2000->ffffffff80d07e00
(XEN)  Phys-Mach map: ffffffff80d08000->ffffffff81c7c658
(XEN)  Start info:    ffffffff81c7d000->ffffffff81c7d4a4
(XEN)  Page tables:   ffffffff81c7e000->ffffffff81c91000
(XEN)  Boot stack:    ffffffff81c91000->ffffffff81c92000
(XEN)  TOTAL:         ffffffff80000000->ffffffff82000000
(XEN)  ENTRY ADDRESS: ffffffff80200000
(XEN) Dom0 has maximum 8 VCPUs
(XEN) Scrubbing Free RAM: .done.
(XEN) Xen trace buffers: disabled
(XEN) Std. Loglevel: Errors and warnings
(XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
(XEN) Xen is relinquishing VGA console.
(XEN) *** Serial input -> DOM0 (type ''CTRL-a'' three times
to switch
input to Xen)
(XEN) Freed 108kB init memory.
(XEN) mtrr: type mismatch for d0000000,2000000 old: write-back new:
write-combining


-----------------------------------------
Rafael E. Ayala
Senior Software Engineer
Xyratex International, Inc.


______________________________________________________________________
This email may contain privileged or confidential information, which should only
be used for the purpose for which it was sent by Xyratex. No further rights or
licenses are granted to use such information. If you are not the intended
recipient of this message, please notify the sender by return and delete it. You
may not use, copy, disclose or rely on the information contained in it.

Internet email is susceptible to data corruption, interception and unauthorised
amendment for which Xyratex does not accept liability. While we have taken
reasonable precautions to ensure that this email is free of viruses, Xyratex
does not accept liability for the presence of any computer viruses in this
email, nor for any losses caused as a result of viruses.

Xyratex Technology Limited (03134912), Registered in England & Wales,
Registered Office, Langstone Road, Havant, Hampshire, PO9 1SA.

The Xyratex group of companies also includes, Xyratex Ltd, registered in
Bermuda, Xyratex International Inc, registered in California, Xyratex (Malaysia)
Sdn Bhd registered in Malaysia, Xyratex Technology (Wuxi) Co Ltd registered in
The People''s Republic of China and Xyratex Japan Limited registered in
Japan.
______________________________________________________________________

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Todd Deshane

2008-Dec-10 23:17 UTC

head link

Re: [Xen-users] VTd not showing PCI device in VM

On Wed, Dec 10, 2008 at 6:00 PM, Rafael Ayala
<rafael_ayala@us.xyratex.com> wrote:> I''m having trouble actually seeing a PCI device in my VMs. 
I''ve resolved
> several of my issues using previous posts and using the VTdHowTo wiki page.
> I have both VT and VTd BIOS options enabled.  I have pciback hiding the
> devices and xm can list and assign the devices to VMs.  I don''t
see the PCI
> devices in my VMs though.  I have tried a windows and a Linux vm, without
> success. I also made sure I removed the drivers for the devices to make
sure
> dom0 was not using them.
>
> Grub command:
> title Xen 3.3 (2.6.18.8)
>     root (hd0,1)
>     kernel /boot/xen-3.3.0.gz vtd=1 iommu=1
>     module /boot/vmlinuz-2.6.18.8-xen
> root=/dev/disk/by-id/scsi-SATA_ST3250310NS_9SF0LZAP-part2 vga=0x317
> resume=/dev/sda1 splash=silent showopts
> pciback.hide=(01:00.0)(01:00.1)(06:00.0)
>
>     module /boot/initrd-2.6.18.8-xen
>
> The PCI devices I''m trying to assign:
> # lspci
> 01:00.0 Fibre Channel: Emulex Corporation Saturn-X: LightPulse Fibre
Channel
> Host Adapter (rev 03)
> 01:00.1 Fibre Channel: Emulex Corporation Saturn-X: LightPulse Fibre
Channel
> Host Adapter (rev 03)
> 06:00.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1068E
> PCI-Express Fusion-MPT SAS (rev 08)
>
> Xen says I can assign them
> # xm pci-list-assignable-devices
> 0000:06:00.0
> 0000:01:00.0 0000:01:00.1
>
> Create my VM:
> # xm create w2k3.vxm pci=01:00.0 pci=01:00.1
> Using config file "./w2k3.vxm".
> Started domain w2k3
>
> I can confirm Xen thinks I have assigned the devices
> # xm pci-list w2k3
> domain   bus   slot   func
> 0x0  0x01   0x00    0x0
> 0x0  0x01   0x00    0x1
>
> # xm create rhel5.vxm.vxm pci=01:00.0 pci=01:00.1
> Using config file "./rhel5.vxm".
> Started domain rhel5
>
> # xm pci-list rhel5
> domain   bus   slot   func
> 0x0  0x06   0x00    0x0
>
> Again, neither windows (device manager) nor Linux (lspci) VMs can see the
> assigned devices.  Below is "xm dmesg" showing VTd is enabled. 
Another
> curious fact is that I can assign the same devices to 2 VMs.  I would have
> expected an error, but perhaps it is allowed to share devices.  Anyone have
> any thoughts on what I''m missing?
>
>From the looks of it, you have done your homework and it looks like it
*should*work.

I would recommend trying to hook up a serial console to the Xen box and getting
good debug info and sending it to xen-devel using the latest xen-unstable pull.

here is a serial console example:

kernel          /boot/xen-3.4-unstable.gz loglvl=all guest_loglvl=all
com1=115200,8n1 console=com1
module          /boot/vmlinuz-2.6.18.8-xen root=/dev/sda1 ro
console=hvc0 earlyprintk=xen xencons=ttyS0,115200

You will of course want to add in the iommu stuff too.

Hope that helps.

Cheers,
Todd

-- 
Todd Deshane
http://todddeshane.net
http://runningxen.com

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Rafael Ayala

2008-Dec-12 22:13 UTC

head link

RE: [Xen-users] VTd not showing PCI device in VM

For completeness sake, I figured out my problem.  I saw in the xend log
the following error message:

[2008-12-12 06:32:14 5487] ERROR (XendDomainInfo:547) Device model
didn\047t tell the vslots for PCI device

That lead me to Simon "Horms" Horman blog where he explains that error
is caused by an improperly built qemu-dm.  I did not have libpci-dev
(formerly pciutils-dev), so pci pass-through support was not built in.
A clean and a rebuild worked wonders.

----------------------------------------- 
Rafael E. Ayala 
Senior Software Engineer 
Xyratex International, Inc. 


______________________________________________________________________
This email may contain privileged or confidential information, which should only
be used for the purpose for which it was sent by Xyratex. No further rights or
licenses are granted to use such information. If you are not the intended
recipient of this message, please notify the sender by return and delete it. You
may not use, copy, disclose or rely on the information contained in it.

Internet email is susceptible to data corruption, interception and unauthorised
amendment for which Xyratex does not accept liability. While we have taken
reasonable precautions to ensure that this email is free of viruses, Xyratex
does not accept liability for the presence of any computer viruses in this
email, nor for any losses caused as a result of viruses.

Xyratex Technology Limited (03134912), Registered in England & Wales,
Registered Office, Langstone Road, Havant, Hampshire, PO9 1SA.

The Xyratex group of companies also includes, Xyratex Ltd, registered in
Bermuda, Xyratex International Inc, registered in California, Xyratex (Malaysia)
Sdn Bhd registered in Malaysia, Xyratex Technology (Wuxi) Co Ltd registered in
The People''s Republic of China and Xyratex Japan Limited registered in
Japan.
______________________________________________________________________

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Seemingly Similar Threads

Search for more maybe matching threads

Xen users - Dec 2008 - VTd not showing PCI device in VM

[Xen-users] VTd not showing PCI device in VM

Re: [Xen-users] VTd not showing PCI device in VM

RE: [Xen-users] VTd not showing PCI device in VM

Seemingly Similar Threads