Hello, I am trying to test out VT-d functionality in Xen, and failing. I have encountered two separate obstacles, one I can work around, but the other I would greatly appreciate some assistance with. Began working on Squeeze, but haven''t been able to successfully get Xen 4.1.2 functioning, so I moved to Wheezy (3.2 kernel). Problem #1: Xen-Wheezy with an EFI Boot Partition does not recognize the RAM, it sees 511MB per dmesg instead of 12GB. Booting Wheezy itself sees the full 12GB, and I don''t think it is a Dom0 issue since dmesg is where I see the limitation. The only solution I found involved recompiling from source, which I haven''t a clue where to start: http://serverfault.com/questions/342109/xen-only-sees-512mb-of-system-ram-should-be-8gb-uefi-boot If anyone knows how to fix this without a recompile, I would appreciate it. For now my solution is to use MBR instead for now. Problem #2: This problem is my priority, as I haven''t found a workaround or solution. Installed Xen from the package manager (aptitude) and it works fine. Modified grub to set Dom0 memory and vcpus, no problems. However, it ignores any pciback commands (both pciback.hide and xen-pciback.hide). When I try running "modprobe pciback" I get: "FATAL: Module pciback not found." My assumption is that it didn''t installed the pciback module, but I can''t find any instructions on how or where to get this. Any help would be appreciated. Below is a set of supporting information including the main system hardware, partition scheme & Xen related Kernel Configuration. Let me know if additional information is needed. System Hardware: Core i7 2600 ASRock Extreme7 Gen3 12GB Corsair XMS RAM (2x4G, 2x2G) 240GB OCZ Vertex 3 LG DVDRW Partitioned scheme: 256M Fat32 (Optional EFI) 15G Ext4 root 2G Swap 220G LVM (for DomU''s) Xen Wheezy 3.2 Kernel Config contains these: CONFIG_XEN=y CONFIG_XEN_DOM0=y CONFIG_XEN_PRIVILEGED_GUEST=y CONFIG_XEN_PVHVM=y CONFIG_XEN_MAX_DOMAIN_MEMORY=128 CONFIG_XEN_SAVE_RESTORE=y # CONFIG_XEN_DEBUG_FS is not set CONFIG_PCI_XEN=y CONFIG_XEN_PCIDEV_FRONTEND=m CONFIG_XEN_BLKDEV_FRONTEND=m CONFIG_XEN_BLKDEV_BACKEND=m CONFIG_NETXEN_NIC=m CONFIG_XEN_NETDEV_FRONTEND=m CONFIG_XEN_NETDEV_BACKEND=m CONFIG_INPUT_XEN_KBDDEV_FRONTEND=y CONFIG_HVC_XEN=y CONFIG_XEN_WDT=m CONFIG_XEN_FBDEV_FRONTEND=y CONFIG_XEN_BALLOON=y # CONFIG_XEN_BALLOON_MEMORY_HOTPLUG is not set CONFIG_XEN_SCRUB_PAGES=y CONFIG_XEN_DEV_EVTCHN=m CONFIG_XEN_BACKEND=y CONFIG_XENFS=m CONFIG_XEN_COMPAT_XENFS=y CONFIG_XEN_SYS_HYPERVISOR=y CONFIG_XEN_XENBUS_FRONTEND=y CONFIG_XEN_GNTDEV=m CONFIG_XEN_GRANT_DEV_ALLOC=m CONFIG_SWIOTLB_XEN=y CONFIG_XEN_PCIDEV_BACKEND=m _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
Hi Casey, problem #2 may be solved by: modprobe xen-pciback the name was changed some kernel-versions ago; if nothing else helps, see if find /lib/modules/$(uname -r) -name "*xen*" does find anything close to pciback ... IIRC its better to compile it into the kernel and provide the parameters on the kernel cmdline via grub/lilo. Am Montag, 5. März 2012, 21:13:07 schrieb Casey DeLorme:> CONFIG_XEN_PCIDEV_BACKEND=mGreetings Tobias
Hello Tobias, and thanks for the suggestions! I had been told that Debian Squeeze came with pciback "out of the box", but clearly that was a misnomer (given the "m" in my kernel config), and as mentioned in my initial email I tried adding lines to grub with no results. Using the find command I was able to find three records with pci: /lib/modules/3.2.0-1-amd64/kernel/drivers/pci/xen-pcifront.ko /lib/modules/3.2.0-1-amd64/kernel/drivers/xen/xen-pciback /lib/modules/3.2.0-1-amd64/kernel/drivers/xen/xen-pciback/xen-pciback.ko Also, I can type just "modprobe xen-pciback" without any errors. However, when I attempt to send a command to it, it fails: sudo modprobe xen-pciback ''hide=(03:00.0)'' ERROR: could not insert ''xen_pciback'': No such file or directory I am new enough to Xen that this really doesn''t give me enough to go on. It''s trying to insert... someplace, and it''s using xen_pciback (not xen-pciback?) Your other suggestion is much more appealing, but I want to ask a few questions before I give that a spin. Question #1: I would be compiling the whole kernel, correct? I just want to make sure I''m not missing some faster or easier method that would append functionality to an existing kernel. Question #2: With a custom Kernel will I be able to use the package manager to install Xen, or will I have to compile Xen from source to have it recognize my custom Linux Kernel? Question #3: If I wanted to compile Xen from source, or the package manager is not an option, how can I compile the latest Xen source with my custom kernel? I found instructions for 4.1.1 in a pdf, but they didn''t work verbatim with 4.1.2, and I ended up with 4x .gz files in my boot directory, and half a dozen new grub records, none of which booted. Compiled from source would be ideal as I could try fixing the efi-boot memory issues, but without some kind of guide for 4.1.2 I would be lost at accomplishing this. ~Casey _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
Hello again, and many thanks. I bring a status update!
I was able to compile the Linux Kernel 3.2.9 with pciback built-in instead
of as a module. I also installed Xen Hypervisor 4.1 via aptitude and was
able to adjust settings in grub.cfg after.
Xen boots, and the grub line appears to work:
xen-pciback.hide=(01:00.0)(02:00.0)(03:00.0)(03:00.1)
Now when I run xm pci-list-assignable-devices, I get:
0000:03:00.0
0000:03:00.1
0000:01:00.0
0000:02:00.0
0000:02:02.0
0000:02:03.0
My objective is to achieve USB & VGA passthrough, but my graphics card
appears to be sitting behind numerous devices.
My DomU Configuration:
name="windows7"
maxmem=6912
memory=6912
#maxvcpus=6
vcpus=4
on_poweroff="destroy"
on_reboot="restart"
on_crash="destroy"
localtime=0
builder="hvm"
device_model="/usr/lib/xen-4.1/bin/qemu-dm"
kernel="/usr/lib/xen-4.1/boot/hvmloader"
boot="dc"
disk=[
''phy:/dev/guests/win7,hda,w'',
''file:/home/xen/win7.iso,hdc:cdrom,r''
]
vif=[ ''bridge=xenbr0, model=e1000'' ]
stdvga=1
sdl=0
vnc=1
vncunused=1
viridian=1
pae=1
usbdevice=''tablet''
I can run xm create -c windows7 without any problems.
When I add this to the configuration:
pci=[
''03:00.0'',
''03:00.1''
]
I get this error:
Error: pci: to avoid potential security issue, 0000:03:00.0 is not allowed
to be assigned to guest since it is behind PCIe switch that does not
support or enable ACS.
So I tried this:
pci=[
''01:00.0''
''03:00.0'',
''03:00.1''
]
However, I then receive this error:
Error: pci: PCI Backend and pci-stub don''t own device 0000:01:00.0
Even though it''s listed as an assignable device, so I don''t
know if I
misread what the card is sitting behind, or if there is some override I
need to add to my configuration.
I can pass other devices, such as my USB and onboard Audio without any
problems, for example:
pci=[
''00:1a.0''
]
If I try to attach the device after starting the DomU I get:
xm pci-attach 3 0000:01:00.0
Error: pci: PCI Backend and pci-stub don''t own device 0000:01:00.0
Usage: xm pci-attach [-o|--options=<opt>] <Domain>
<domain:bus:slot.func>
[virtual slot]
Insert a new pass-through pci device.
So, I have two questions.
#1: Is there a way to override the first error, if it''s a
"potential
security" risk, surely there must be an option to ignore it?
#2: Any ideas as to why the 01:00.0 devices is listed as assignable yet
not available? Do I need to pass a step further up?
lspci -tv output:
-[0000:00]-+-00.0 Intel Corporation 2nd Generation Core Processor Family
DRAM Controller
+-01.0-[01-05]----00.0-[02-05]--+-00.0-[03]--+-00.0 nVidia
Corporation GF104 [GeForce GTX 460]
| | \-00.1 nVidia
Corporation GF104 High Definition Audio Controller
| +-02.0-[04]--
| \-03.0-[05]--
+-02.0 Intel Corporation 2nd Generation Core Processor Family
Integrated Graphics Controller
+-16.0 Intel Corporation 6 Series/C200 Series Chipset Family
MEI Controller #1
+-1a.0 Intel Corporation 6 Series/C200 Series Chipset Family
USB Enhanced Host Controller #2
+-1b.0 Intel Corporation 6 Series/C200 Series Chipset Family
High Definition Audio Controller
+-1c.0-[06]--
+-1c.5-[07]----00.0 ASMedia Technology Inc. ASM1062 Serial ATA
Controller
+-1c.6-[08]----00.0 ASMedia Technology Inc. ASM1042 SuperSpeed
USB Host Controller
+-1c.7-[09-12]----00.0-[0a-12]--+-01.0-[0b]----00.0 Broadcom
Corporation NetLink BCM57781 Gigabit Ethernet PCIe
| +-04.0-[0c]----00.0 ASMedia
Technology Inc. ASM1062 Serial ATA Controller
| +-05.0-[0d]----00.0 Broadcom
Corporation NetLink BCM57781 Gigabit Ethernet PCIe
| +-06.0-[0e]----00.0 VIA
Technologies, Inc. VT6315 Series Firewire Controller
| +-07.0-[0f-10]----00.0-[10]--
| +-08.0-[11]----00.0 ASMedia
Technology Inc. ASM1042 SuperSpeed USB Host Controller
| \-09.0-[12]----00.0 ASMedia
Technology Inc. ASM1042 SuperSpeed USB Host Controller
+-1d.0 Intel Corporation 6 Series/C200 Series Chipset Family
USB Enhanced Host Controller #1
+-1f.0 Intel Corporation Z68 Express Chipset Family LPC
Controller
+-1f.2 Intel Corporation 6 Series/C200 Series Chipset Family 6
port SATA AHCI Controller
\-1f.3 Intel Corporation 6 Series/C200 Series Chipset Family
SMBus Controller
lspci:
00:00.0 Host bridge: Intel Corporation 2nd Generation Core Processor Family
DRAM Controller (rev 09)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200/2nd Generation Core
Processor Family PCI Express Root Port (rev 09)
00:02.0 VGA compatible controller: Intel Corporation 2nd Generation Core
Processor Family Integrated Graphics Controller (rev 09)
00:16.0 Communication controller: Intel Corporation 6 Series/C200 Series
Chipset Family MEI Controller #1 (rev 04)
00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset
Family USB Enhanced Host Controller #2 (rev 05)
00:1b.0 Audio device: Intel Corporation 6 Series/C200 Series Chipset Family
High Definition Audio Controller (rev 05)
00:1c.0 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family
PCI Express Root Port 1 (rev b5)
00:1c.5 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family
PCI Express Root Port 6 (rev b5)
00:1c.6 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family
PCI Express Root Port 7 (rev b5)
00:1c.7 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family
PCI Express Root Port 8 (rev b5)
00:1d.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset
Family USB Enhanced Host Controller #1 (rev 05)
00:1f.0 ISA bridge: Intel Corporation Z68 Express Chipset Family LPC
Controller (rev 05)
00:1f.2 SATA controller: Intel Corporation 6 Series/C200 Series Chipset
Family 6 port SATA AHCI Controller (rev 05)
00:1f.3 SMBus: Intel Corporation 6 Series/C200 Series Chipset Family SMBus
Controller (rev 05)
01:00.0 PCI bridge: nVidia Corporation NF200 PCIe 2.0 switch for mainboards
(rev a3)
02:00.0 PCI bridge: nVidia Corporation NF200 PCIe 2.0 switch for mainboards
(rev a3)
02:02.0 PCI bridge: nVidia Corporation NF200 PCIe 2.0 switch for mainboards
(rev a3)
02:03.0 PCI bridge: nVidia Corporation NF200 PCIe 2.0 switch for mainboards
(rev a3)
03:00.0 VGA compatible controller: nVidia Corporation GF104 [GeForce GTX
460] (rev a1)
03:00.1 Audio device: nVidia Corporation GF104 High Definition Audio
Controller (rev a1)
07:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA
Controller (rev 01)
08:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host
Controller
09:00.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI
Express Gen 2 (5.0 GT/s) Switch (rev ba)
0a:01.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI
Express Gen 2 (5.0 GT/s) Switch (rev ba)
0a:04.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI
Express Gen 2 (5.0 GT/s) Switch (rev ba)
0a:05.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI
Express Gen 2 (5.0 GT/s) Switch (rev ba)
0a:06.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI
Express Gen 2 (5.0 GT/s) Switch (rev ba)
0a:07.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI
Express Gen 2 (5.0 GT/s) Switch (rev ba)
0a:08.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI
Express Gen 2 (5.0 GT/s) Switch (rev ba)
0a:09.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI
Express Gen 2 (5.0 GT/s) Switch (rev ba)
0b:00.0 Ethernet controller: Broadcom Corporation NetLink BCM57781 Gigabit
Ethernet PCIe (rev 10)
0c:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA
Controller (rev 01)
0d:00.0 Ethernet controller: Broadcom Corporation NetLink BCM57781 Gigabit
Ethernet PCIe (rev 10)
0e:00.0 FireWire (IEEE 1394): VIA Technologies, Inc. VT6315 Series Firewire
Controller (rev 01)
0f:00.0 PCI bridge: ASMedia Technology Inc. ASM108x PCIe to PCI Bridge
Controller (rev 01)
11:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host
Controller
12:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host
Controller
On Tue, Mar 6, 2012 at 1:29 PM, Casey DeLorme <cdelorme@gmail.com> wrote:
> Hello Tobias, and thanks for the suggestions!
>
> I had been told that Debian Squeeze came with pciback "out of the
box",
> but clearly that was a misnomer (given the "m" in my kernel
config), and as
> mentioned in my initial email I tried adding lines to grub with no results.
>
> Using the find command I was able to find three records with pci:
> /lib/modules/3.2.0-1-amd64/kernel/drivers/pci/xen-pcifront.ko
> /lib/modules/3.2.0-1-amd64/kernel/drivers/xen/xen-pciback
> /lib/modules/3.2.0-1-amd64/kernel/drivers/xen/xen-pciback/xen-pciback.ko
>
> Also, I can type just "modprobe xen-pciback" without any errors.
However,
> when I attempt to send a command to it, it fails:
> sudo modprobe xen-pciback ''hide=(03:00.0)''
> ERROR: could not insert ''xen_pciback'': No such file or
directory
>
> I am new enough to Xen that this really doesn''t give me enough to
go on.
> It''s trying to insert... someplace, and it''s using
xen_pciback (not
> xen-pciback?)
>
>
> Your other suggestion is much more appealing, but I want to ask a few
> questions before I give that a spin.
>
> Question #1:
> I would be compiling the whole kernel, correct? I just want to make sure
> I''m not missing some faster or easier method that would append
> functionality to an existing kernel.
>
> Question #2:
> With a custom Kernel will I be able to use the package manager to install
> Xen, or will I have to compile Xen from source to have it recognize my
> custom Linux Kernel?
>
> Question #3:
> If I wanted to compile Xen from source, or the package manager is not an
> option, how can I compile the latest Xen source with my custom kernel?
>
> I found instructions for 4.1.1 in a pdf, but they didn''t work
verbatim
> with 4.1.2, and I ended up with 4x .gz files in my boot directory, and half
> a dozen new grub records, none of which booted.
>
> Compiled from source would be ideal as I could try fixing the efi-boot
> memory issues, but without some kind of guide for 4.1.2 I would be lost at
> accomplishing this.
>
> ~Casey
>
_______________________________________________
Xen-users mailing list
Xen-users@lists.xen.org
http://lists.xen.org/xen-users
Hi Casey, looks like nvidia brings with it not only a GPU and a HDMI pci-device, but also "NF200 PCIe 2.0 switch" ; Maybe this is one of the many reasons why NVIDIA GPUs make so much problems when passing them through in xen. Sorry, my short answer is: get a ATI Card :) With ATI it''s - generally - no problem to pass it through in xen. no such PCI- Device-ID/BUS plethora, no xen code patches to make - it "just works", at least here with a Radeon 6970. You may test David Techer''s way of adapting xen to your individual nvidia setup, search (also in xen-devel) for his posts and urls to his blog. Greetings and good luck anyways! Tobias Am Mittwoch, 7. März 2012, 01:55:05 schrieb Casey DeLorme:> Hello again, and many thanks. I bring a status update! > > I was able to compile the Linux Kernel 3.2.9 with pciback built-in instead > of as a module. I also installed Xen Hypervisor 4.1 via aptitude and was > able to adjust settings in grub.cfg after. > > Xen boots, and the grub line appears to work: > xen-pciback.hide=(01:00.0)(02:00.0)(03:00.0)(03:00.1) > > Now when I run xm pci-list-assignable-devices, I get: > 0000:03:00.0 > 0000:03:00.1 > 0000:01:00.0 > 0000:02:00.0 > 0000:02:02.0 > 0000:02:03.0 > > My objective is to achieve USB & VGA passthrough, but my graphics card > appears to be sitting behind numerous devices. > > My DomU Configuration: > > name="windows7" > maxmem=6912 > memory=6912 > #maxvcpus=6 > vcpus=4 > on_poweroff="destroy" > on_reboot="restart" > on_crash="destroy" > localtime=0 > builder="hvm" > device_model="/usr/lib/xen-4.1/bin/qemu-dm" > kernel="/usr/lib/xen-4.1/boot/hvmloader" > boot="dc" > disk=[ > ''phy:/dev/guests/win7,hda,w'', > ''file:/home/xen/win7.iso,hdc:cdrom,r'' > ] > vif=[ ''bridge=xenbr0, model=e1000'' ] > stdvga=1 > sdl=0 > vnc=1 > vncunused=1 > viridian=1 > pae=1 > usbdevice=''tablet'' > > > I can run xm create -c windows7 without any problems. > > When I add this to the configuration: > > pci=[ > ''03:00.0'', > ''03:00.1'' > ] > > I get this error: > Error: pci: to avoid potential security issue, 0000:03:00.0 is not allowed > to be assigned to guest since it is behind PCIe switch that does not > support or enable ACS. > > So I tried this: > > pci=[ > ''01:00.0'' > ''03:00.0'', > ''03:00.1'' > ] > > However, I then receive this error: > Error: pci: PCI Backend and pci-stub don''t own device 0000:01:00.0 > > Even though it''s listed as an assignable device, so I don''t know if I > misread what the card is sitting behind, or if there is some override I > need to add to my configuration. > > I can pass other devices, such as my USB and onboard Audio without any > problems, for example: > > pci=[ > ''00:1a.0'' > ] > > If I try to attach the device after starting the DomU I get: > > xm pci-attach 3 0000:01:00.0 > Error: pci: PCI Backend and pci-stub don''t own device 0000:01:00.0 > Usage: xm pci-attach [-o|--options=<opt>] <Domain> <domain:bus:slot.func> > [virtual slot] > > Insert a new pass-through pci device. > > > So, I have two questions. > > #1: Is there a way to override the first error, if it''s a "potential > security" risk, surely there must be an option to ignore it? > > #2: Any ideas as to why the 01:00.0 devices is listed as assignable yet > not available? Do I need to pass a step further up? > > > lspci -tv output: > > -[0000:00]-+-00.0 Intel Corporation 2nd Generation Core Processor Family > DRAM Controller > +-01.0-[01-05]----00.0-[02-05]--+-00.0-[03]--+-00.0 nVidia > Corporation GF104 [GeForce GTX 460] > > | | \-00.1 nVidia > > Corporation GF104 High Definition Audio Controller > > | +-02.0-[04]-- > | \-03.0-[05]-- > > +-02.0 Intel Corporation 2nd Generation Core Processor Family > Integrated Graphics Controller > +-16.0 Intel Corporation 6 Series/C200 Series Chipset Family > MEI Controller #1 > +-1a.0 Intel Corporation 6 Series/C200 Series Chipset Family > USB Enhanced Host Controller #2 > +-1b.0 Intel Corporation 6 Series/C200 Series Chipset Family > High Definition Audio Controller > +-1c.0-[06]-- > +-1c.5-[07]----00.0 ASMedia Technology Inc. ASM1062 Serial ATA > Controller > +-1c.6-[08]----00.0 ASMedia Technology Inc. ASM1042 SuperSpeed > USB Host Controller > +-1c.7-[09-12]----00.0-[0a-12]--+-01.0-[0b]----00.0 Broadcom > Corporation NetLink BCM57781 Gigabit Ethernet PCIe > > | +-04.0-[0c]----00.0 ASMedia > > Technology Inc. ASM1062 Serial ATA Controller > > | +-05.0-[0d]----00.0 Broadcom > > Corporation NetLink BCM57781 Gigabit Ethernet PCIe > > | +-06.0-[0e]----00.0 VIA > > Technologies, Inc. VT6315 Series Firewire Controller > > | +-07.0-[0f-10]----00.0-[10]-- > | +-08.0-[11]----00.0 ASMedia > > Technology Inc. ASM1042 SuperSpeed USB Host Controller > > | \-09.0-[12]----00.0 ASMedia > > Technology Inc. ASM1042 SuperSpeed USB Host Controller > +-1d.0 Intel Corporation 6 Series/C200 Series Chipset Family > USB Enhanced Host Controller #1 > +-1f.0 Intel Corporation Z68 Express Chipset Family LPC > Controller > +-1f.2 Intel Corporation 6 Series/C200 Series Chipset Family 6 > port SATA AHCI Controller > \-1f.3 Intel Corporation 6 Series/C200 Series Chipset Family > SMBus Controller > > lspci: > > 00:00.0 Host bridge: Intel Corporation 2nd Generation Core Processor Family > DRAM Controller (rev 09) > 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200/2nd Generation Core > Processor Family PCI Express Root Port (rev 09) > 00:02.0 VGA compatible controller: Intel Corporation 2nd Generation Core > Processor Family Integrated Graphics Controller (rev 09) > 00:16.0 Communication controller: Intel Corporation 6 Series/C200 Series > Chipset Family MEI Controller #1 (rev 04) > 00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset > Family USB Enhanced Host Controller #2 (rev 05) > 00:1b.0 Audio device: Intel Corporation 6 Series/C200 Series Chipset Family > High Definition Audio Controller (rev 05) > 00:1c.0 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family > PCI Express Root Port 1 (rev b5) > 00:1c.5 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family > PCI Express Root Port 6 (rev b5) > 00:1c.6 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family > PCI Express Root Port 7 (rev b5) > 00:1c.7 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family > PCI Express Root Port 8 (rev b5) > 00:1d.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset > Family USB Enhanced Host Controller #1 (rev 05) > 00:1f.0 ISA bridge: Intel Corporation Z68 Express Chipset Family LPC > Controller (rev 05) > 00:1f.2 SATA controller: Intel Corporation 6 Series/C200 Series Chipset > Family 6 port SATA AHCI Controller (rev 05) > 00:1f.3 SMBus: Intel Corporation 6 Series/C200 Series Chipset Family SMBus > Controller (rev 05) > 01:00.0 PCI bridge: nVidia Corporation NF200 PCIe 2.0 switch for mainboards > (rev a3) > 02:00.0 PCI bridge: nVidia Corporation NF200 PCIe 2.0 switch for mainboards > (rev a3) > 02:02.0 PCI bridge: nVidia Corporation NF200 PCIe 2.0 switch for mainboards > (rev a3) > 02:03.0 PCI bridge: nVidia Corporation NF200 PCIe 2.0 switch for mainboards > (rev a3) > 03:00.0 VGA compatible controller: nVidia Corporation GF104 [GeForce GTX > 460] (rev a1) > 03:00.1 Audio device: nVidia Corporation GF104 High Definition Audio > Controller (rev a1) > 07:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA > Controller (rev 01) > 08:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host > Controller > 09:00.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI > Express Gen 2 (5.0 GT/s) Switch (rev ba) > 0a:01.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI > Express Gen 2 (5.0 GT/s) Switch (rev ba) > 0a:04.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI > Express Gen 2 (5.0 GT/s) Switch (rev ba) > 0a:05.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI > Express Gen 2 (5.0 GT/s) Switch (rev ba) > 0a:06.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI > Express Gen 2 (5.0 GT/s) Switch (rev ba) > 0a:07.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI > Express Gen 2 (5.0 GT/s) Switch (rev ba) > 0a:08.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI > Express Gen 2 (5.0 GT/s) Switch (rev ba) > 0a:09.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI > Express Gen 2 (5.0 GT/s) Switch (rev ba) > 0b:00.0 Ethernet controller: Broadcom Corporation NetLink BCM57781 Gigabit > Ethernet PCIe (rev 10) > 0c:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA > Controller (rev 01) > 0d:00.0 Ethernet controller: Broadcom Corporation NetLink BCM57781 Gigabit > Ethernet PCIe (rev 10) > 0e:00.0 FireWire (IEEE 1394): VIA Technologies, Inc. VT6315 Series Firewire > Controller (rev 01) > 0f:00.0 PCI bridge: ASMedia Technology Inc. ASM108x PCIe to PCI Bridge > Controller (rev 01) > 11:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host > Controller > 12:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host > Controller > > On Tue, Mar 6, 2012 at 1:29 PM, Casey DeLorme <cdelorme@gmail.com> wrote: > > Hello Tobias, and thanks for the suggestions! > > > > I had been told that Debian Squeeze came with pciback "out of the box", > > but clearly that was a misnomer (given the "m" in my kernel config), and > > as mentioned in my initial email I tried adding lines to grub with no > > results. > > > > Using the find command I was able to find three records with pci: > > /lib/modules/3.2.0-1-amd64/kernel/drivers/pci/xen-pcifront.ko > > /lib/modules/3.2.0-1-amd64/kernel/drivers/xen/xen-pciback > > /lib/modules/3.2.0-1-amd64/kernel/drivers/xen/xen-pciback/xen-pciback.ko > > > > Also, I can type just "modprobe xen-pciback" without any errors. > > However, when I attempt to send a command to it, it fails: > > sudo modprobe xen-pciback ''hide=(03:00.0)'' > > ERROR: could not insert ''xen_pciback'': No such file or directory > > > > I am new enough to Xen that this really doesn''t give me enough to go on. > > > > It''s trying to insert... someplace, and it''s using xen_pciback (not > > > > xen-pciback?) > > > > > > Your other suggestion is much more appealing, but I want to ask a few > > questions before I give that a spin. > > > > Question #1: > > I would be compiling the whole kernel, correct? I just want to make sure > > I''m not missing some faster or easier method that would append > > functionality to an existing kernel. > > > > Question #2: > > With a custom Kernel will I be able to use the package manager to install > > Xen, or will I have to compile Xen from source to have it recognize my > > custom Linux Kernel? > > > > Question #3: > > If I wanted to compile Xen from source, or the package manager is not an > > option, how can I compile the latest Xen source with my custom kernel? > > > > I found instructions for 4.1.1 in a pdf, but they didn''t work verbatim > > with 4.1.2, and I ended up with 4x .gz files in my boot directory, and > > half a dozen new grub records, none of which booted. > > > > Compiled from source would be ideal as I could try fixing the efi-boot > > memory issues, but without some kind of guide for 4.1.2 I would be lost > > at accomplishing this. > > > > ~Casey
Thanks for the follow-up Tobias, and for making today an incredible success! I owe many of todays accomplishments to your initial suggestion: - Compiled Linux Kernel 3.2.9 (this was a first) - Verified pciback functioning from Dom0 - Got a Remote VNC working - Installed a Windows 7 HVM DomU - Passed onboard HD Audio & USB 3.0 - Successfully tested the onboard Audio (100% functional) I began researching VT-d three months ago, and my mind was lost in a sea of innumerable possibilities. I spent a month verifying compatible components, including selecting a motherboard. ASUS and Gigabyte were ruled out due to there distinct lack of Socket 1155 boards with VT-d support. I settled on ASRock as their price to feature was pretty decent, but only two 1155 boards had VT-d & Dual LAN, the Extreme7 and Fatal1ty. I went with the Extreme7, because the Fatal1ty had useless (to me) components (IDE & PS/2). Both boards have a PLX PEX8608 chip, so each would have at least one layer between the PCI card and the board, so I don''t know if the Fatal1ty would solve my problem. To clarify, the NF200 chip is built into the motherboard, it is not a part of the nVidia GPU, but it is made by nVidia. Even if I had an ATI card it would still run through the same layer and be prone to the same problem. The NF200 chip is used to manage SLI and CrossFire, so it isn''t an nVidia feature exclusively. I confirmed via lspci -vvv, the NF200 chip does not feature ACS functionality, hence the error. I read that the security setting can be ignored by modifying the xend-config, but I won''t be testing this until tomorrow. Alternatively there is one PCIe slot not sitting behind the NF200 that I can also try. In all likelihood though, I may end up swapping motherboards again. I began working on this project 17 days ago, and since then I have read through roughly 3000~ pages of documentation. I am on my 47th fresh installation of Linux, with over 40 pages of detailed notes. Words cannot express how pleased I was to have audio today. That said, if anyone has had any experience working with layered pci passthrough, or overriding the Xen security setting I would be most grateful to hear about it. Signing off, ~Casey On Wed, Mar 7, 2012 at 3:31 AM, Tobias Geiger <tobias.geiger@vido.info>wrote:> Hi Casey, > > looks like nvidia brings with it not only a GPU and a HDMI pci-device, but > also "NF200 PCIe 2.0 switch" ; Maybe this is one of the many reasons why > NVIDIA GPUs make so much problems when passing them through in xen. > > Sorry, my short answer is: get a ATI Card :) > With ATI it''s - generally - no problem to pass it through in xen. no such > PCI- > Device-ID/BUS plethora, no xen code patches to make - it "just works", at > least here with a Radeon 6970. > > You may test David Techer''s way of adapting xen to your individual nvidia > setup, search (also in xen-devel) for his posts and urls to his blog. > > Greetings and good luck anyways! > Tobias > > Am Mittwoch, 7. März 2012, 01:55:05 schrieb Casey DeLorme: > > Hello again, and many thanks. I bring a status update! > > > > I was able to compile the Linux Kernel 3.2.9 with pciback built-in > instead > > of as a module. I also installed Xen Hypervisor 4.1 via aptitude and was > > able to adjust settings in grub.cfg after. > > > > Xen boots, and the grub line appears to work: > > xen-pciback.hide=(01:00.0)(02:00.0)(03:00.0)(03:00.1) > > > > Now when I run xm pci-list-assignable-devices, I get: > > 0000:03:00.0 > > 0000:03:00.1 > > 0000:01:00.0 > > 0000:02:00.0 > > 0000:02:02.0 > > 0000:02:03.0 > > > > My objective is to achieve USB & VGA passthrough, but my graphics card > > appears to be sitting behind numerous devices. > > > > My DomU Configuration: > > > > name="windows7" > > maxmem=6912 > > memory=6912 > > #maxvcpus=6 > > vcpus=4 > > on_poweroff="destroy" > > on_reboot="restart" > > on_crash="destroy" > > localtime=0 > > builder="hvm" > > device_model="/usr/lib/xen-4.1/bin/qemu-dm" > > kernel="/usr/lib/xen-4.1/boot/hvmloader" > > boot="dc" > > disk=[ > > ''phy:/dev/guests/win7,hda,w'', > > ''file:/home/xen/win7.iso,hdc:cdrom,r'' > > ] > > vif=[ ''bridge=xenbr0, model=e1000'' ] > > stdvga=1 > > sdl=0 > > vnc=1 > > vncunused=1 > > viridian=1 > > pae=1 > > usbdevice=''tablet'' > > > > > > I can run xm create -c windows7 without any problems. > > > > When I add this to the configuration: > > > > pci=[ > > ''03:00.0'', > > ''03:00.1'' > > ] > > > > I get this error: > > Error: pci: to avoid potential security issue, 0000:03:00.0 is not > allowed > > to be assigned to guest since it is behind PCIe switch that does not > > support or enable ACS. > > > > So I tried this: > > > > pci=[ > > ''01:00.0'' > > ''03:00.0'', > > ''03:00.1'' > > ] > > > > However, I then receive this error: > > Error: pci: PCI Backend and pci-stub don''t own device 0000:01:00.0 > > > > Even though it''s listed as an assignable device, so I don''t know if I > > misread what the card is sitting behind, or if there is some override I > > need to add to my configuration. > > > > I can pass other devices, such as my USB and onboard Audio without any > > problems, for example: > > > > pci=[ > > ''00:1a.0'' > > ] > > > > If I try to attach the device after starting the DomU I get: > > > > xm pci-attach 3 0000:01:00.0 > > Error: pci: PCI Backend and pci-stub don''t own device 0000:01:00.0 > > Usage: xm pci-attach [-o|--options=<opt>] <Domain> <domain:bus:slot.func> > > [virtual slot] > > > > Insert a new pass-through pci device. > > > > > > So, I have two questions. > > > > #1: Is there a way to override the first error, if it''s a "potential > > security" risk, surely there must be an option to ignore it? > > > > #2: Any ideas as to why the 01:00.0 devices is listed as assignable yet > > not available? Do I need to pass a step further up? > > > > > > lspci -tv output: > > > > -[0000:00]-+-00.0 Intel Corporation 2nd Generation Core Processor Family > > DRAM Controller > > +-01.0-[01-05]----00.0-[02-05]--+-00.0-[03]--+-00.0 nVidia > > Corporation GF104 [GeForce GTX 460] > > > > | | \-00.1 nVidia > > > > Corporation GF104 High Definition Audio Controller > > > > | +-02.0-[04]-- > > | \-03.0-[05]-- > > > > +-02.0 Intel Corporation 2nd Generation Core Processor Family > > Integrated Graphics Controller > > +-16.0 Intel Corporation 6 Series/C200 Series Chipset Family > > MEI Controller #1 > > +-1a.0 Intel Corporation 6 Series/C200 Series Chipset Family > > USB Enhanced Host Controller #2 > > +-1b.0 Intel Corporation 6 Series/C200 Series Chipset Family > > High Definition Audio Controller > > +-1c.0-[06]-- > > +-1c.5-[07]----00.0 ASMedia Technology Inc. ASM1062 Serial > ATA > > Controller > > +-1c.6-[08]----00.0 ASMedia Technology Inc. ASM1042 > SuperSpeed > > USB Host Controller > > +-1c.7-[09-12]----00.0-[0a-12]--+-01.0-[0b]----00.0 Broadcom > > Corporation NetLink BCM57781 Gigabit Ethernet PCIe > > > > | +-04.0-[0c]----00.0 ASMedia > > > > Technology Inc. ASM1062 Serial ATA Controller > > > > | +-05.0-[0d]----00.0 Broadcom > > > > Corporation NetLink BCM57781 Gigabit Ethernet PCIe > > > > | +-06.0-[0e]----00.0 VIA > > > > Technologies, Inc. VT6315 Series Firewire Controller > > > > | +-07.0-[0f-10]----00.0-[10]-- > > | +-08.0-[11]----00.0 ASMedia > > > > Technology Inc. ASM1042 SuperSpeed USB Host Controller > > > > | \-09.0-[12]----00.0 ASMedia > > > > Technology Inc. ASM1042 SuperSpeed USB Host Controller > > +-1d.0 Intel Corporation 6 Series/C200 Series Chipset Family > > USB Enhanced Host Controller #1 > > +-1f.0 Intel Corporation Z68 Express Chipset Family LPC > > Controller > > +-1f.2 Intel Corporation 6 Series/C200 Series Chipset Family > 6 > > port SATA AHCI Controller > > \-1f.3 Intel Corporation 6 Series/C200 Series Chipset Family > > SMBus Controller > > > > lspci: > > > > 00:00.0 Host bridge: Intel Corporation 2nd Generation Core Processor > Family > > DRAM Controller (rev 09) > > 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200/2nd Generation Core > > Processor Family PCI Express Root Port (rev 09) > > 00:02.0 VGA compatible controller: Intel Corporation 2nd Generation Core > > Processor Family Integrated Graphics Controller (rev 09) > > 00:16.0 Communication controller: Intel Corporation 6 Series/C200 Series > > Chipset Family MEI Controller #1 (rev 04) > > 00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset > > Family USB Enhanced Host Controller #2 (rev 05) > > 00:1b.0 Audio device: Intel Corporation 6 Series/C200 Series Chipset > Family > > High Definition Audio Controller (rev 05) > > 00:1c.0 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family > > PCI Express Root Port 1 (rev b5) > > 00:1c.5 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family > > PCI Express Root Port 6 (rev b5) > > 00:1c.6 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family > > PCI Express Root Port 7 (rev b5) > > 00:1c.7 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family > > PCI Express Root Port 8 (rev b5) > > 00:1d.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset > > Family USB Enhanced Host Controller #1 (rev 05) > > 00:1f.0 ISA bridge: Intel Corporation Z68 Express Chipset Family LPC > > Controller (rev 05) > > 00:1f.2 SATA controller: Intel Corporation 6 Series/C200 Series Chipset > > Family 6 port SATA AHCI Controller (rev 05) > > 00:1f.3 SMBus: Intel Corporation 6 Series/C200 Series Chipset Family > SMBus > > Controller (rev 05) > > 01:00.0 PCI bridge: nVidia Corporation NF200 PCIe 2.0 switch for > mainboards > > (rev a3) > > 02:00.0 PCI bridge: nVidia Corporation NF200 PCIe 2.0 switch for > mainboards > > (rev a3) > > 02:02.0 PCI bridge: nVidia Corporation NF200 PCIe 2.0 switch for > mainboards > > (rev a3) > > 02:03.0 PCI bridge: nVidia Corporation NF200 PCIe 2.0 switch for > mainboards > > (rev a3) > > 03:00.0 VGA compatible controller: nVidia Corporation GF104 [GeForce GTX > > 460] (rev a1) > > 03:00.1 Audio device: nVidia Corporation GF104 High Definition Audio > > Controller (rev a1) > > 07:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA > > Controller (rev 01) > > 08:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB > Host > > Controller > > 09:00.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI > > Express Gen 2 (5.0 GT/s) Switch (rev ba) > > 0a:01.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI > > Express Gen 2 (5.0 GT/s) Switch (rev ba) > > 0a:04.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI > > Express Gen 2 (5.0 GT/s) Switch (rev ba) > > 0a:05.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI > > Express Gen 2 (5.0 GT/s) Switch (rev ba) > > 0a:06.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI > > Express Gen 2 (5.0 GT/s) Switch (rev ba) > > 0a:07.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI > > Express Gen 2 (5.0 GT/s) Switch (rev ba) > > 0a:08.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI > > Express Gen 2 (5.0 GT/s) Switch (rev ba) > > 0a:09.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI > > Express Gen 2 (5.0 GT/s) Switch (rev ba) > > 0b:00.0 Ethernet controller: Broadcom Corporation NetLink BCM57781 > Gigabit > > Ethernet PCIe (rev 10) > > 0c:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA > > Controller (rev 01) > > 0d:00.0 Ethernet controller: Broadcom Corporation NetLink BCM57781 > Gigabit > > Ethernet PCIe (rev 10) > > 0e:00.0 FireWire (IEEE 1394): VIA Technologies, Inc. VT6315 Series > Firewire > > Controller (rev 01) > > 0f:00.0 PCI bridge: ASMedia Technology Inc. ASM108x PCIe to PCI Bridge > > Controller (rev 01) > > 11:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB > Host > > Controller > > 12:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB > Host > > Controller > > > > On Tue, Mar 6, 2012 at 1:29 PM, Casey DeLorme <cdelorme@gmail.com> > wrote: > > > Hello Tobias, and thanks for the suggestions! > > > > > > I had been told that Debian Squeeze came with pciback "out of the box", > > > but clearly that was a misnomer (given the "m" in my kernel config), > and > > > as mentioned in my initial email I tried adding lines to grub with no > > > results. > > > > > > Using the find command I was able to find three records with pci: > > > /lib/modules/3.2.0-1-amd64/kernel/drivers/pci/xen-pcifront.ko > > > /lib/modules/3.2.0-1-amd64/kernel/drivers/xen/xen-pciback > > > > /lib/modules/3.2.0-1-amd64/kernel/drivers/xen/xen-pciback/xen-pciback.ko > > > > > > Also, I can type just "modprobe xen-pciback" without any errors. > > > However, when I attempt to send a command to it, it fails: > > > sudo modprobe xen-pciback ''hide=(03:00.0)'' > > > ERROR: could not insert ''xen_pciback'': No such file or directory > > > > > > I am new enough to Xen that this really doesn''t give me enough to go > on. > > > > > > It''s trying to insert... someplace, and it''s using xen_pciback (not > > > > > > xen-pciback?) > > > > > > > > > Your other suggestion is much more appealing, but I want to ask a few > > > questions before I give that a spin. > > > > > > Question #1: > > > I would be compiling the whole kernel, correct? I just want to make > sure > > > I''m not missing some faster or easier method that would append > > > functionality to an existing kernel. > > > > > > Question #2: > > > With a custom Kernel will I be able to use the package manager to > install > > > Xen, or will I have to compile Xen from source to have it recognize my > > > custom Linux Kernel? > > > > > > Question #3: > > > If I wanted to compile Xen from source, or the package manager is not > an > > > option, how can I compile the latest Xen source with my custom kernel? > > > > > > I found instructions for 4.1.1 in a pdf, but they didn''t work verbatim > > > with 4.1.2, and I ended up with 4x .gz files in my boot directory, and > > > half a dozen new grub records, none of which booted. > > > > > > Compiled from source would be ideal as I could try fixing the efi-boot > > > memory issues, but without some kind of guide for 4.1.2 I would be lost > > > at accomplishing this. > > > > > > ~Casey > >_______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users