Hi list, I''m trying to run pvlinux (via pvgrub) using pci passthrough (and vga pass through), xen version is 4.2.1, dom0 & domU kernel are both 3.8.2, I checked the kernel log (DMAR) and the motherboard (H67) supports vt-d, processor is i7-2600 and it supports vt-d too. When pci passthrough is not enabled, everything is fine; but after I enabled pci pass through, domU can only see 512MB RAM (USB passed from pci-passthrough works, but video/intel didn''t work, likely due to i915 load error from dmesg, everything else is also good). I googled and found a similar issue is reported at: http://lists.xen.org/archives/html/xen-users/2012-12/msg00179.html But seems there is no follow up how the problem is fixed, how can I fix the issue? Below are some links from pastebin.com: 1) native dmesg without loading xen.gz: http://pastebin.com/BTVNTi8D 2) (kernel) dmesg of dom0: http://pastebin.com/DEZ7gfmG 3) (xl) dmesg of dom0: http://pastebin.com/46bR2ngv 4) domU dmesg without pci passthrough: http://pastebin.com/DGZR4NLP 5) domU dmesg with pci passthrough: http://pastebin.com/aujxwys2 I tried xen version 4.2.0 too but still no luck. My kernel have all xen feature enabled; and I can provide more info if necessary. Thanks baojun
ShadesOfGrey
2013-Mar-11 22:42 UTC
Re: domU only see 512MB RAM when enabling PCI passthrough
From Casey Delorme''s "Comprehensive Xen Debian Wheezy PCI Passthrough Tutorial", this may be the potential cause of your issue.> *EFI Source Modification* > > Given that I encountered no make errors, I was shocked to find that > they still hadn''t addressed the grub efi compatibility bug where it > fails to recognize available system memory. > > The fix for this is a source modification, though they have supposedly > added a build process for xen.efi, which would replace the debian.efi > generated by grub. I have yet to find adequate instructions to make > that work, and even if I did may break adding |xen-pciback.hide| to > the grub configuration, which we need if we are planning to do the > easy method of PCI Passthrough. > > So, let''s open up |xen/arch/x86/setup.c| with our favorite text > editor, and find the line containing |( e820_raw_nr != 0 )|, and make > that area look like this: > > |#if 0 > else if ( e820_raw_nr != 0 ) > { > memmap_type = "Xen-e820"; > } > else if ( bootsym(lowmem_kb) ) > { > memmap_type = "Xen-e801"; > e820_raw[0].addr = 0; > e820_raw[0].size = bootsym(lowmem_kb) << 10; > e820_raw[0].type = E820_RAM; > e820_raw[1].addr = 0x100000; > e820_raw[1].size = bootsym(highmem_kb) << 10; > e820_raw[1].type = E820_RAM; > e820_raw_nr = 2; > } > #endif > else if ( mbi->flags & MBI_MEMMAP ) > | > > Notice we are commenting out a section of code using "if 0", which > always evaluates to false, to eliminate the section of e801 mapping > which breaks our ram recognition. > > I have no idea what kind of adverse affects this has on other systems, > but it has worked for me since Xen 4.1.2. > > *End EFI Source Modification* >On 03/08/2013 07:44 PM, Baojun Wang wrote:> Hi list, > > I''m trying to run pvlinux (via pvgrub) using pci passthrough (and vga > pass through), xen version is 4.2.1, dom0 & domU kernel are both > 3.8.2, I checked the kernel log (DMAR) and the motherboard (H67) > supports vt-d, processor is i7-2600 and it supports vt-d too. > > When pci passthrough is not enabled, everything is fine; > but after I enabled pci pass through, domU can only see 512MB RAM (USB > passed from pci-passthrough works, but video/intel didn''t work, likely > due to i915 load error from dmesg, everything else is also good). > > I googled and found a similar issue is reported at: > http://lists.xen.org/archives/html/xen-users/2012-12/msg00179.html > > But seems there is no follow up how the problem is fixed, how can I > fix the issue? > > > Below are some links from pastebin.com: > > 1) native dmesg without loading xen.gz: http://pastebin.com/BTVNTi8D > 2) (kernel) dmesg of dom0: http://pastebin.com/DEZ7gfmG > 3) (xl) dmesg of dom0: http://pastebin.com/46bR2ngv > 4) domU dmesg without pci passthrough: http://pastebin.com/DGZR4NLP > 5) domU dmesg with pci passthrough: http://pastebin.com/aujxwys2 > > I tried xen version 4.2.0 too but still no luck. > My kernel have all xen feature enabled; and I can provide more info if > necessary. > > Thanks > baojun > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xen.org > http://lists.xen.org/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
Casey DeLorme
2013-Mar-12 00:15 UTC
Re: domU only see 512MB RAM when enabling PCI passthrough
That patch was for the EFI Dom0 memory recognition, and was patched sometime before 4.2 released (by reversing the if statement). Your xl dmesg shows e820 (and approx. 16G) so I am quite certain that is not the cause of your DomU memory bug. Can you provide your PV DomU configuration? I know I''ve heard about the bug before, usually in the context of over 3GB to a PV domain, but from what I recall it was fixed: http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1673 On Mon, Mar 11, 2013 at 6:42 PM, ShadesOfGrey <shades_of_grey@earthlink.net>wrote:> From Casey Delorme''s "Comprehensive Xen Debian Wheezy PCI Passthrough > Tutorial", this may be the potential cause of your issue. > > *EFI Source Modification* > > Given that I encountered no make errors, I was shocked to find that they > still hadn''t addressed the grub efi compatibility bug where it fails to > recognize available system memory. > > The fix for this is a source modification, though they have supposedly > added a build process for xen.efi, which would replace the debian.efi > generated by grub. I have yet to find adequate instructions to make that > work, and even if I did may break adding xen-pciback.hide to the grub > configuration, which we need if we are planning to do the easy method of > PCI Passthrough. > > So, let''s open up xen/arch/x86/setup.c with our favorite text editor, and > find the line containing ( e820_raw_nr != 0 ), and make that area look > like this: > > #if 0 > else if ( e820_raw_nr != 0 ) > { > memmap_type = "Xen-e820"; > } > else if ( bootsym(lowmem_kb) ) > { > memmap_type = "Xen-e801"; > e820_raw[0].addr = 0; > e820_raw[0].size = bootsym(lowmem_kb) << 10; > e820_raw[0].type = E820_RAM; > e820_raw[1].addr = 0x100000; > e820_raw[1].size = bootsym(highmem_kb) << 10; > e820_raw[1].type = E820_RAM; > e820_raw_nr = 2; > } > #endif > else if ( mbi->flags & MBI_MEMMAP ) > > > Notice we are commenting out a section of code using "if 0", which always > evaluates to false, to eliminate the section of e801 mapping which breaks > our ram recognition. > > I have no idea what kind of adverse affects this has on other systems, but > it has worked for me since Xen 4.1.2. > > *End EFI Source Modification* > > > > On 03/08/2013 07:44 PM, Baojun Wang wrote: > > Hi list, > > I''m trying to run pvlinux (via pvgrub) using pci passthrough (and vga > pass through), xen version is 4.2.1, dom0 & domU kernel are both > 3.8.2, I checked the kernel log (DMAR) and the motherboard (H67) > supports vt-d, processor is i7-2600 and it supports vt-d too. > > When pci passthrough is not enabled, everything is fine; > but after I enabled pci pass through, domU can only see 512MB RAM (USB > passed from pci-passthrough works, but video/intel didn''t work, likely > due to i915 load error from dmesg, everything else is also good). > > I googled and found a similar issue is reported at:http://lists.xen.org/archives/html/xen-users/2012-12/msg00179.html > > But seems there is no follow up how the problem is fixed, how can I > fix the issue? > > > Below are some links from pastebin.com: > > 1) native dmesg without loading xen.gz: http://pastebin.com/BTVNTi8D > 2) (kernel) dmesg of dom0: http://pastebin.com/DEZ7gfmG > 3) (xl) dmesg of dom0: http://pastebin.com/46bR2ngv > 4) domU dmesg without pci passthrough: http://pastebin.com/DGZR4NLP > 5) domU dmesg with pci passthrough: http://pastebin.com/aujxwys2 > > I tried xen version 4.2.0 too but still no luck. > My kernel have all xen feature enabled; and I can provide more info if > necessary. > > Thanks > baojun > > _______________________________________________ > Xen-users mailing listXen-users@lists.xen.orghttp://lists.xen.org/xen-users > > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xen.org > http://lists.xen.org/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
Casey DeLorme
2013-Mar-12 04:42 UTC
Re: domU only see 512MB RAM when enabling PCI passthrough
As mentioned, that patch is only for Dom0 with EFI and has nothing to do with DomU''s. I have very little PV experience myself, but I can suggest a couple of ideas. You might try dropping the RAM to maybe 3G (3072) and seeing if anything changes. Perhaps try increments of 1G and see where it breaks. If reducing the supplied RAM suddenly provides more memory then maybe the bug I referenced earlier wasn''t fixed after all. It would give you something more to work with anyways. Another possibility is manually setting `e820_host`, which according to the man pages is specifically for supporting PCI Passthrough to PV, however the same description claims that it should automatically be on when devices are passed. Can''t hurt to give it a try. Also, just a reminder to use "Reply All" to keep the list in the loop. Out of curiosity, is there any reason why you would choose PV over PVHVM? My understanding is that PVHVM performs nearly the same while remaining mostly encapsulated without all the confusion I see regarding PV configuration and bugs. If you don''t have a specific reason it might be worth investigating. In any event, I hope these help with troubleshooting. Sincerely, ~Casey PS - Sorry for top posting, just wanted the original message to remain in context when I included xen-users again. Also, another reference link: http://wiki.xen.org/wiki/Virtualization_Spectrum On Mon, Mar 11, 2013 at 10:47 PM, Baojun Wang <wangbj@gmail.com> wrote:> Yep, I tried comment the code by shadowofgray, but it still yields the > same result. > > Below is domU configuration: > > kernel = "/usr/lib/xen/boot/pv-grub-x86_64.gz" > extra = "(hd0)/boot/grub/pvgrub.lst" > > memory = ''8192'' > > vcpus = 4 > > disk = [ ''/dev/sda1,raw,xvda1,w'', ''/dev/vg00/gentoo,raw,xvda2,w'', > ''/dev/vg00/exports,raw,xvda3,w'', ''/dev/vg01/home,raw,xvda4,w'', > ''/dev/vg01/swap,raw,xvda5,w'' ] > > name = ''gentoo'' > > serial = ''pty'' > # > gfx_passthru=1 > > pci = [ ''00:02.0'', ''00:1a.0'', ''00:1b.0'', ''00:1d.0'', ''03:00.0'', ''04:00.0'' ] > > vif = [ ''mac=fe:00:36:79:00:01, bridge=br0, virtio'' ] > > > boot option for xen.gz already includes pci devices above. and pci > devices passed through are video/audio/usb: > Below is lspci output: > > 00:00.0 Host bridge: Intel Corporation 2nd Generation Core Processor > Family DRAM Controller (rev 09) > 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200/2nd Generation Core > Processor Family PCI Express Root Port (rev 09) > 00:02.0 VGA compatible controller: Intel Corporation 2nd Generation > Core Processor Family Integrated Graphics Controller (rev 09) > 00:16.0 Communication controller: Intel Corporation 6 Series/C200 > Series Chipset Family MEI Controller #1 (rev 04) > 00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset > Family USB Enhanced Host Controller #2 (rev 05) > 00:1b.0 Audio device: Intel Corporation 6 Series/C200 Series Chipset > Family High Definition Audio Controller (rev 05) > 00:1c.0 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset > Family PCI Express Root Port 1 (rev b5) > 00:1c.1 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset > Family PCI Express Root Port 2 (rev b5) > 00:1c.2 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset > Family PCI Express Root Port 3 (rev b5) > 00:1c.3 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset > Family PCI Express Root Port 4 (rev b5) > 00:1c.4 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset > Family PCI Express Root Port 5 (rev b5) > 00:1d.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset > Family USB Enhanced Host Controller #1 (rev 05) > 00:1f.0 ISA bridge: Intel Corporation H67 Express Chipset Family LPC > Controller (rev 05) > 00:1f.2 SATA controller: Intel Corporation 6 Series/C200 Series > Chipset Family SATA AHCI Controller (rev 05) > 00:1f.3 SMBus: Intel Corporation 6 Series/C200 Series Chipset Family > SMBus Controller (rev 05) > 03:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB > Host Controller > 04:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB > Host Controller > 05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. > RTL8111/8168 PCI Express Gigabit Ethernet controller (rev 06) > > > > Thanks > baojun > > On Mon, Mar 11, 2013 at 5:15 PM, Casey DeLorme <cdelorme@gmail.com> wrote: > > That patch was for the EFI Dom0 memory recognition, and was patched > sometime > > before 4.2 released (by reversing the if statement). Your xl dmesg shows > > e820 (and approx. 16G) so I am quite certain that is not the cause of > your > > DomU memory bug. > > > > Can you provide your PV DomU configuration? > > > > > > I know I''ve heard about the bug before, usually in the context of over > 3GB > > to a PV domain, but from what I recall it was fixed: > > http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1673 > > > > > > > > On Mon, Mar 11, 2013 at 6:42 PM, ShadesOfGrey < > shades_of_grey@earthlink.net> > > wrote: > >> > >> From Casey Delorme''s "Comprehensive Xen Debian Wheezy PCI Passthrough > >> Tutorial", this may be the potential cause of your issue. > >> > >> EFI Source Modification > >> > >> Given that I encountered no make errors, I was shocked to find that they > >> still hadn''t addressed the grub efi compatibility bug where it fails to > >> recognize available system memory. > >> > >> The fix for this is a source modification, though they have supposedly > >> added a build process for xen.efi, which would replace the debian.efi > >> generated by grub. I have yet to find adequate instructions to make that > >> work, and even if I did may break adding xen-pciback.hide to the grub > >> configuration, which we need if we are planning to do the easy method > of PCI > >> Passthrough. > >> > >> So, let''s open up xen/arch/x86/setup.c with our favorite text editor, > and > >> find the line containing ( e820_raw_nr != 0 ), and make that area look > like > >> this: > >> > >> #if 0 > >> else if ( e820_raw_nr != 0 ) > >> { > >> memmap_type = "Xen-e820"; > >> } > >> else if ( bootsym(lowmem_kb) ) > >> { > >> memmap_type = "Xen-e801"; > >> e820_raw[0].addr = 0; > >> e820_raw[0].size = bootsym(lowmem_kb) << 10; > >> e820_raw[0].type = E820_RAM; > >> e820_raw[1].addr = 0x100000; > >> e820_raw[1].size = bootsym(highmem_kb) << 10; > >> e820_raw[1].type = E820_RAM; > >> e820_raw_nr = 2; > >> } > >> #endif > >> else if ( mbi->flags & MBI_MEMMAP ) > >> > >> > >> Notice we are commenting out a section of code using "if 0", which > always > >> evaluates to false, to eliminate the section of e801 mapping which > breaks > >> our ram recognition. > >> > >> I have no idea what kind of adverse affects this has on other systems, > but > >> it has worked for me since Xen 4.1.2. > >> > >> End EFI Source Modification > >> > >> > >> > >> On 03/08/2013 07:44 PM, Baojun Wang wrote: > >> > >> Hi list, > >> > >> I''m trying to run pvlinux (via pvgrub) using pci passthrough (and vga > >> pass through), xen version is 4.2.1, dom0 & domU kernel are both > >> 3.8.2, I checked the kernel log (DMAR) and the motherboard (H67) > >> supports vt-d, processor is i7-2600 and it supports vt-d too. > >> > >> When pci passthrough is not enabled, everything is fine; > >> but after I enabled pci pass through, domU can only see 512MB RAM (USB > >> passed from pci-passthrough works, but video/intel didn''t work, likely > >> due to i915 load error from dmesg, everything else is also good). > >> > >> I googled and found a similar issue is reported at: > >> http://lists.xen.org/archives/html/xen-users/2012-12/msg00179.html > >> > >> But seems there is no follow up how the problem is fixed, how can I > >> fix the issue? > >> > >> > >> Below are some links from pastebin.com: > >> > >> 1) native dmesg without loading xen.gz: http://pastebin.com/BTVNTi8D > >> 2) (kernel) dmesg of dom0: http://pastebin.com/DEZ7gfmG > >> 3) (xl) dmesg of dom0: http://pastebin.com/46bR2ngv > >> 4) domU dmesg without pci passthrough: http://pastebin.com/DGZR4NLP > >> 5) domU dmesg with pci passthrough: http://pastebin.com/aujxwys2 > >> > >> I tried xen version 4.2.0 too but still no luck. > >> My kernel have all xen feature enabled; and I can provide more info if > >> necessary. > >> > >> Thanks > >> baojun > >> > >> _______________________________________________ > >> Xen-users mailing list > >> Xen-users@lists.xen.org > >> http://lists.xen.org/xen-users > >> > >> > >> > >> _______________________________________________ > >> Xen-users mailing list > >> Xen-users@lists.xen.org > >> http://lists.xen.org/xen-users > > > > > > > > _______________________________________________ > > Xen-users mailing list > > Xen-users@lists.xen.org > > http://lists.xen.org/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
Christian Holpert
2013-Mar-12 07:33 UTC
Re: domU only see 512MB RAM when enabling PCI passthrough
At 01:44 09.03.2013, you wrote:>I googled and found a similar issue is reported at: >http://lists.xen.org/archives/html/xen-users/2012-12/msg00179.html > >But seems there is no follow up how the problem is fixed, how can I >fix the issue?in my case from December: no fix found till now. Upgraded my Gentoo to app-emulation/xen-tools-4.2.1-r1, app-emulation/xen-4.2.1-r2. A workaround is to boot the domU without the PCI-Device in config and pass it later on. This way you can work with all the RAM you need _and_ the PCI-Device. Still hoping for a solution in one of the next updates. When I shutdown this domU for reboot I keep a ''ghost'' in xl list xen xen # xl list Name ID Mem VCPUs State Time(s) Domain-0 0 7529 4 r----- 85.2 colin 1 2048 3 -b---- 102.3 (null) 2 6 2 --ps-d 144.6 which can be removed by killing the corresponding qemu-system-i386 process. Regards, Christian -- Markt-Apotheke Christian Holpert e.K., Mömbris http://www.marktapotheke.org - info@marktapotheke.org http://www.blue-essentials-shop.de - info@blue-essentials-shop.de Apotheker Christian Holpert Im Markthof 5 63776 Mömbris Tel.: 06029-1379 Fax: 9986589 Amtsgericht Aschaffenburg, HR.A, Registernummer 2165