On a box with an amd64 X2, running the snv66 xen bits, and booted in 64-bit mode: Why does a hvm domU boot a 32-bit Solaris x86 kernel from hdd, instead of the 64-bit kernel? Is this because of bug 6471548 ? Synopsis: multiboot too restrictive about amd64 capable platforms. http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6471548 (Btw, qemu now contains a workaround for this; the latest qemu-system-x86_64 is able to boot S-x86 in 64 bit mode) This message posted from opensolaris.org
Hi Jurgen, Jürgen Keil wrote:> On a box with an amd64 X2, running the snv66 xen bits, and booted in 64-bit mode: > Why does a hvm domU boot a 32-bit Solaris x86 kernel from hdd, instead of the > 64-bit kernel? > > Is this because of bug 6471548 ? > Synopsis: multiboot too restrictive about amd64 capable platforms. > http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6471548It could be, but even if that is addressed there are more serious problems waiting down the line. I booted an HVM nevada domain and forced it to run the 64-bit kernel, but it doesn''t get very far in boot before hanging or spinning somewhere. Fortunately, kmdb loads and is able to begin stepping through mlsetup(). More investigation is needed here...I just filed 6586971 to track the issue. > (Btw, qemu now contains a workaround for this; the latest > qemu-system-x86_64 is able to boot S-x86 in 64 bit mode) Our xen tree has an older version of qemu, so if newer qemu bits are needed then this will most likely wait until we move to Xen 3.1. - Russ
Russ wrote,> Jürgen Keil wrote: > > On a box with an amd64 X2, running the snv66 xen bits, and booted in 64-bit mode: > > Why does a hvm domU boot a 32-bit Solaris x86 kernel from hdd, instead of the > > 64-bit kernel? > > > > Is this because of bug 6471548 ? > > Synopsis: multiboot too restrictive about amd64 capable platforms. > > http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6471548 > > It could be, but even if that is addressed there are more serious problems > waiting down the line.When I add a "verbose" grub command, I see that grub reports amd64: CPU does NOT support PSE This is from usr/src/grub/grub-0.95/stage2/builtins.c: 2911 if (!BITX(stdfeatures, 17, 17)) { 2912 grub_printf("amd64: CPU does NOT support PSE\n"); 2913 isamd64--; 2914 } The error message text isn't 100% correct, because it the PSE36 feature bit that isn't set in CPUID(1) %edx register, not the PSE bit. AFAIK, the S-x86 kernel doesn't need the PSE36 feature. Hmm, In Dom0, CPUID(1) returns a %edx with the PSE36 bit set, in a (HVM) DomU, the PSE36 bit isn't set.> I booted an HVM nevada domain and forced it to run the 64-bit kernel, but it > doesn't get very far in boot before hanging or spinning somewhere.Yep, I installed a modified grub stage2 that doesn't check for the CPUID(1) %EDX bit17 PSE36 any more. Not it starts to bootstrap the 64 bit kernel, but seems to hang in usr/src/uts/i86pc/os/pci_cfgspace.c, function pci_check_bios() at line 214: 204 static int 205 pci_check_bios(void) 206 { 207 struct bop_regs regs; 208 uint32_t carryflag; 209 uint16_t ax, dx; 210 211 bzero(®s, sizeof (regs)); 212 regs.eax.word.ax = (PCI_FUNCTION_ID << 8) | PCI_BIOS_PRESENT; 213 214 BOP_DOINT(bootops, 0x1a, ®s);> Fortunately, kmdb loads and is able to begin stepping through mlsetup(). > More investigation is needed here...I just filed 6586971 to track the issue. > > > (Btw, qemu now contains a workaround for this; the latest > > qemu-system-x86_64 is able to boot S-x86 in 64 bit mode) > > Our xen tree has an older version of qemu, so if newer qemu bits are needed > then this will most likely wait until we move to Xen 3.1.The qemu workaround is in target-i386/helper2.c, line 139; CPUID(1) returns the PSE36 feature: 134 env->cpuid_xlevel = 0x80000008; 135 136 /* these features are needed for Win64 and aren't fully implemented */ 137 env->cpuid_features |= CPUID_MTRR | CPUID_CLFLUSH | CPUID_MCA; 138 /* this feature is needed for Solaris and isn't fully implemented */ 139 env->cpuid_features |= CPUID_PSE36; This message posted from opensolaris.org _______________________________________________ xen-discuss mailing list xen-discuss@opensolaris.org
Russell wrote,> Jürgen Keil wrote: > > On a box with an amd64 X2, running the snv66 xen bits, and booted in 64-bit mode: > > Why does a hvm domU boot a 32-bit Solaris x86 kernel from hdd, instead of the > > 64-bit kernel? > > I booted an HVM nevada domain and forced it to run the 64-bit kernel, but it > doesn't get very far in boot before hanging or spinning somewhere. > Fortunately, kmdb loads and is able to begin stepping through mlsetup(). > More investigation is needed here...I just filed 6586971 to track the issue.On my amd64 X2 box, the 64bit hvm domU is hanging when doing a BIOS interrupt, while running the code in usr/src/uts/i86pc/ml/bios_call_src.s By adding some "outb $0x84" instructions to bios_call_src.s (and adding some debug messages to qemu-dm when i/o port 0x84 is written) I traced the hang to this piece of code in bios_call_src.s /* * disable long mode by: * - shutting down paging (bit 31 of cr0) * - flushing the TLB * - disabling LME (long made enable) in EFER (extended feature reg) */ movl %cr0, %eax btcl $31, %eax /* disable paging */ movl %eax, %cr0 <<<<<<<<<<<<<<< hangs here It seems to hang when %CR0 is written, after clearing the CR0.PG bit. When I boot the xen hypervisor with command line "kernel$ /boot/$ISADIR/xen.gz hvm_debug=0x20" and try to boot the solaris 64-bit kernel in a hvm domU, I see the following endless stream of messages is "xm dmesg": ... (XEN) [HVM:1.0] <svm_do_page_fault> svm_do_page_fault = 0x51c7, eip = 51c7, error_code = 10 (XEN) [HVM:1.0] <svm_vmexit_handler> eax=50011, ebx=0, ecx=0, edx=0, esi=0, edi=0 (XEN) [HVM:1.0] <svm_do_page_fault> svm_do_page_fault = 0x51c7, eip = 51c7, error_code = 10 (XEN) [HVM:1.0] <svm_vmexit_handler> eax=50011, ebx=0, ecx=0, edx=0, esi=0, edi=0 (XEN) [HVM:1.0] <svm_do_page_fault> svm_do_page_fault = 0x51c7, eip = 51c7, error_code = 10 (XEN) [HVM:1.0] <svm_vmexit_handler> eax=50011, ebx=0, ecx=0, edx=0, esi=0, edi=0 (XEN) [HVM:1.0] <svm_do_page_fault> svm_do_page_fault = 0x51c7, eip = 51c7, error_code = 10 (XEN) [HVM:1.0] <svm_vmexit_handler> eax=50011, ebx=0, ecx=0, edx=0, esi=0, edi=0 ... eip 0x51c7 is the address following the instruction where it writes to %CR0 (Where I've added an "outb(0x84, 0x52)" call ): ... 0x51bd: movl %cr0, %eax 0x51c0: btcl $31, %eax /* disable paging */ 0x51c4: movl %eax, %cr0 0x51c7: movb $0x52, %al ... It seems as if clearing CR0.PG has killed access to our protected mode code segment in the hvm domU? Btw can anyone with an Intel VT-x machine try to reproduce this? Does the same hang happen on Intel 64-bit? This message posted from opensolaris.org _______________________________________________ xen-discuss mailing list xen-discuss@opensolaris.org
Russ has been working on HVM support and just now has been concentrating on running Solaris 10 under HVM. It works under AMD-v, but not VT-x - due to all kinds of deficiencies in the Xen VT-x HVM real mode support. I'm sure with 64 bit issue you're running into similar issues which we just haven't gotten to yet. Jürgen Keil wrote:> Russell wrote, > >> Jürgen Keil wrote: >>> On a box with an amd64 X2, running the snv66 xen bits, and booted in 64-bit mode: >>> Why does a hvm domU boot a 32-bit Solaris x86 kernel from hdd, instead of the >>> 64-bit kernel? >> I booted an HVM nevada domain and forced it to run the 64-bit kernel, but it >> doesn't get very far in boot before hanging or spinning somewhere. >> Fortunately, kmdb loads and is able to begin stepping through mlsetup(). >> More investigation is needed here...I just filed 6586971 to track the issue. > > On my amd64 X2 box, the 64bit hvm domU is hanging when doing a BIOS > interrupt, while running the code in usr/src/uts/i86pc/ml/bios_call_src.s > > By adding some "outb $0x84" instructions to bios_call_src.s (and adding > some debug messages to qemu-dm when i/o port 0x84 is written) I traced the > hang to this piece of code in bios_call_src.s > > /* > * disable long mode by: > * - shutting down paging (bit 31 of cr0) > * - flushing the TLB > * - disabling LME (long made enable) in EFER (extended feature reg) > */ > movl %cr0, %eax > btcl $31, %eax /* disable paging */ > movl %eax, %cr0 <<<<<<<<<<<<<<< hangs here > > It seems to hang when %CR0 is written, after clearing the CR0.PG bit. > > When I boot the xen hypervisor with command line > "kernel$ /boot/$ISADIR/xen.gz hvm_debug=0x20" > and try to boot the solaris 64-bit kernel in a hvm domU, I see the > following endless stream of messages is "xm dmesg": > > ... > (XEN) [HVM:1.0] <svm_do_page_fault> svm_do_page_fault = 0x51c7, eip = 51c7, error_code = 10 > (XEN) [HVM:1.0] <svm_vmexit_handler> eax=50011, ebx=0, ecx=0, edx=0, esi=0, edi=0 > (XEN) [HVM:1.0] <svm_do_page_fault> svm_do_page_fault = 0x51c7, eip = 51c7, error_code = 10 > (XEN) [HVM:1.0] <svm_vmexit_handler> eax=50011, ebx=0, ecx=0, edx=0, esi=0, edi=0 > (XEN) [HVM:1.0] <svm_do_page_fault> svm_do_page_fault = 0x51c7, eip = 51c7, error_code = 10 > (XEN) [HVM:1.0] <svm_vmexit_handler> eax=50011, ebx=0, ecx=0, edx=0, esi=0, edi=0 > (XEN) [HVM:1.0] <svm_do_page_fault> svm_do_page_fault = 0x51c7, eip = 51c7, error_code = 10 > (XEN) [HVM:1.0] <svm_vmexit_handler> eax=50011, ebx=0, ecx=0, edx=0, esi=0, edi=0 > ... > > > eip 0x51c7 is the address following the instruction where it writes to %CR0 > (Where I've added an "outb(0x84, 0x52)" call ): > > ... > 0x51bd: movl %cr0, %eax > 0x51c0: btcl $31, %eax /* disable paging */ > 0x51c4: movl %eax, %cr0 > 0x51c7: movb $0x52, %al > ... > > > It seems as if clearing CR0.PG has killed access to > our protected mode code segment in the hvm domU? > > > > Btw can anyone with an Intel VT-x machine try to reproduce this? > Does the same hang happen on Intel 64-bit? > > > This message posted from opensolaris.org > _______________________________________________ > xen-discuss mailing list > xen-discuss@opensolaris.org_______________________________________________ xen-discuss mailing list xen-discuss@opensolaris.org
> Russ has been working on HVM support and just now has been concentrating on > running Solaris 10 under HVM. It works under AMD-v, but not VT-x - due to > all kinds of deficiencies in the Xen VT-x HVM real mode support.Interesting; I''ve tried to boot a HVM S10 GA (with the old boot.bin code) from an installation dvd iso image on an AMD X2, but it was extremely slow. After about an hour, the system had booted from cd and loaded a kernel and kmdb and waiting at the kmdb prompt. A qemu-dm process was busy 100% on one cpu core in dom0...
Old S10 GA 64 bit boot up is going to behave super horribly under Xen - if at all - due to how Solaris boot jumps back and forth to real mode for I/O and how often it does that. It does that in a way that no other OS does and hence little effort has (or probably ever will be) made in any hypervisor to do this quickly. S10 updates that use multiboot will be better, due to Grub and the boot-archive. This makes Solaris act much more like linux or bsd startup from a hypervisor''s view. Though it still jumps back and forth between 64 and 32 bit mode during boot quite a bit. It still drops all the way back to real mode a few times. This currently works farily well on Xen with AMD-v, but not VT-x. OpenSolaris builds using dboot will be even faster, as once it gets to 64 bit mode, it stays there. It does drop back to real mode a few times during boot to query the BIOS for information. Those few times are what probably don''t work yet when trying to run 64 bit OpenSolaris under HVM. As I said, Russ is actively working with the Xen HVM contributors to find the holes in the HVM, VT-x and AMD-v code.> > Interesting; I''ve tried to boot a HVM S10 GA (with > the old boot.bin code) > from an installation dvd iso image on an AMD X2, but > it was extremely slow. > After about an hour, the system had booted from cd > and loaded a kernel and > kmdb and waiting at the kmdb prompt. A qemu-dm > process was busy 100% on one > cpu core in dom0... > > _______________________________________________ > xen-discuss mailing list > xen-discuss@opensolaris.orgThis message posted from opensolaris.org
I''m trying to run the 66 build (domU) on a non-HVM amd64 linux kernel dom0. The installer is horribly slow, always in the "preparing java" phase. Does this problem still apply in the 66 build, and does it affect non-HVM cpu too? This message posted from opensolaris.org
Francois Saint-Jacques wrote:> I''m trying to run the 66 build (domU) on a non-HVM amd64> linux kernel dom0. The installer is horribly slow, always > in the "preparing java" phase. Does this problem still > apply in the 66 build, and does it affect non-HVM cpu too? Are you doing a PV or HVM install? If PV, try giving the domU 1G of memory during install, then you can use less memory once the install completes. If HVM, there are some known problem with the IDE CDROM driver in HVM in b66. Also, if your using a 64-bit solaris domU, you need to have a Xen 3.1 dom0. MRJ -- Mark Johnson <mark.johnson@sun.com> Sun Microsystems, Inc. (781) 442-0869
Well, setting it to 1024mb did the trick. I assume I''m doing a PV (not sure of what that mean tought) install since this is a socket 940 opteron and it doesn''t support HVM extension. Currently I''m running xen 3.0 on debian amd64. Is there any reason why I need 3.1 specificaly, the installer does boot with the 64 bit solaris kernel. Thank you for your help PS: If there is no mention of the 1G trick in the documentation, you should add it. If there is one, bold it ;) Francois This message posted from opensolaris.org
Francois Saint-Jacques wrote:> Well, setting it to 1024mb did the trick. I assume I''m doing a> PV (not sure of what that mean tought) install since this is a > socket 940 opteron and it doesn''t support HVM extension. > Currently I''m running xen 3.0 on debian amd64. Is there any > reason why I need 3.1 specificaly, the installer does > boot with the 64 bit solaris kernel. There are some bugs 64-bit Xen 3.0.3. They were fixed in 3.0.4-1... But most Linux distros either have 3.0.3 or 3.1..> Thank you for your help > > PS: If there is no mention of the 1G trick in the documentation, you should add it. If there is one, bold it ;)I''ll check it out... Thanks, MRJ> Francois > > > This message posted from opensolaris.org > _______________________________________________ > xen-discuss mailing list > xen-discuss@opensolaris.org-- Mark Johnson <mark.johnson@sun.com> Sun Microsystems, Inc. (781) 442-0869
Well, I have install xen-3.1 manualy. I wanted to give a shot to b75. I''m using this guide to install opensolaris domU: http://wiki.tardis.ed.ac.uk/index.php/OpenSolaris_Xen_domU except that instead of copying x86 binaries, I use the amd64 version. It seems that I can''t boot the provided kernel/ramdisk. Here''s the output of xend: xc_dom_allocate: cmdline="root=/dev/dsk/c0d0s0 /platform/i86xpv/kernel/amd64/unix -kd - nowin -B install_media=cdrom", featu res="" xc_dom_kernel_file: filename="/etc/xen/osol/unix" xc_dom_malloc_filemap : 1685 kB xc_dom_ramdisk_file: filename="/etc/xen/osol/x86.miniroot" xc_dom_malloc_filemap : 104 MB xc_dom_boot_xen_init: ver 3.1, caps xen-3.0-x86_64 xen-3.0-x86_32p xc_dom_parse_image: called xc_dom_find_loader: trying ELF-generic loader ... failed xc_dom_find_loader: trying multiboot-binary loader ... failed xc_dom_core.c:495: panic: xc_dom_find_loader: no loader found xc_dom_release: called I''ve succesfully installed opensolaris on this system using the same method but with the old ISO (b66). Any hint? Francois This message posted from opensolaris.org