Hello, I read in the announce of Xen 3.2.0 released that it has "preliminary support for a wider range of bootloaders in fully virtualised (HVM) guests, using full emulation of x86 ''real mode''" . I''d like to know what is the level of the emulation of x86 ''real mode''? I ask that because I tried to install OpenSuse 10.3 (I used the iso file openSUSE-10.3-GM-x86_64-mini.iso) with Xen 3.2-testing from mercurial and the boot failed. I got the following messages: "... Booting from CD-Rom... ISOLINUX 3.31 0x46f43c1e Copyright (C) 1994-2005 H. Peter Anvin Loading... " and it hangs after the "...". Logs don''t show any specifics problems. As OpenSuse 10.3 is known to have some ''real mode'' issues due to the presence of the gfx bootloader I asking myself what is the state of the art of ''real mode'' emulation into Xen? Do you have some tests to validate the real mode emulation? Thanks for the help Best Regards, Guillaume _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
You need to build Xen with ''vmxassist=n''. If you boot a debug build of Xen and look at the HVM guest output that gets written to the Xen console in that case as the HVM guest boots, if there is any reference to VMXASSIST then you do not have an appropriate build of Xen. If there is no reference to VMXASSIST, and you see the line "Invoking ROMBIOS ...", then you are good. If you pull latest xen-unstable (eventually-to-be-3.3) then vmxassist is disabled by default, and a few more instructions are correctly emulated. -- Keir On 23/1/08 14:32, "Guillaume Thouvenin" <guillaume.thouvenin@ext.bull.net> wrote:> Hello, > > I read in the announce of Xen 3.2.0 released that it has "preliminary > support for a wider range of bootloaders in fully virtualised (HVM) > guests, using full emulation of x86 ''real mode''" . > > I''d like to know what is the level of the emulation of x86 ''real > mode''? I ask that because I tried to install OpenSuse 10.3 (I used > the iso file openSUSE-10.3-GM-x86_64-mini.iso) with Xen 3.2-testing > from mercurial and the boot failed. I got the following messages: > > "... > Booting from CD-Rom... > > ISOLINUX 3.31 0x46f43c1e Copyright (C) 1994-2005 H. Peter Anvin > Loading... > " > > and it hangs after the "...". Logs don''t show any specifics problems. > As OpenSuse 10.3 is known to have some ''real mode'' issues due to the > presence of the gfx bootloader I asking myself what is the state of the > art of ''real mode'' emulation into Xen? > > Do you have some tests to validate the real mode emulation? > > > Thanks for the help > Best Regards, > > Guillaume_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hello, Guillaume Thouvenin, le Wed 23 Jan 2008 15:32:52 +0100, a écrit :> I''d like to know what is the level of the emulation of x86 ''real > mode''? I ask that because I tried to install OpenSuse 10.3 (I used > the iso file openSUSE-10.3-GM-x86_64-mini.iso) with Xen 3.2-testing > from mercurial and the boot failed. I got the following messages: > > "... > Booting from CD-Rom... > > ISOLINUX 3.31 0x46f43c1e Copyright (C) 1994-2005 H. Peter Anvin > Loading... > " > > and it hangs after the "...". Logs don''t show any specifics problems. > As OpenSuse 10.3 is known to have some ''real mode'' issues due to the > presence of the gfx bootloader I asking myself what is the state of the > art of ''real mode'' emulation into Xen?That''s the kind of bug that should get fixed by this real mode emulation indeed. Note that that precise bug (gfxboot) only appears on Intel machines with VMX. AMD''s SVM doesn''t have that issue. Samuel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Guillaume Thouvenin
2008-Jan-31 09:42 UTC
[Xen-devel] Re: Xen 3.2 and Big Real Mode support?
On Wed, 23 Jan 2008 14:35:44 +0000 Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote:> If you pull latest xen-unstable (eventually-to-be-3.3) then vmxassist is > disabled by default, and a few more instructions are correctly emulated.Thanks for your help. So I tried to install opensuse from openSUSE-10.3-GM-x86_64-mini.iso on my Intel Xeon and it seems that at least one instruction is not emulated in real-mode. I''m working on the problem. Regards, Guillaume (XEN) HVM1: HVM Loader (XEN) HVM1: Detected Xen v3.3-unstable (XEN) HVM1: Writing SMBIOS tables ... (XEN) HVM1: Loading ROMBIOS ... (XEN) HVM1: 9004 bytes of ROMBIOS high-memory extensions: (XEN) HVM1: Relocating to 0x3fff9c00-0x3fffbf2c ... done (XEN) irq.c:236: Dom1 PCI link 0 changed 0 -> 5 (XEN) HVM1: PCI-ISA link 0 routed to IRQ5 (XEN) irq.c:236: Dom1 PCI link 1 changed 0 -> 10 (XEN) HVM1: PCI-ISA link 1 routed to IRQ10 (XEN) irq.c:236: Dom1 PCI link 2 changed 0 -> 11 (XEN) HVM1: PCI-ISA link 2 routed to IRQ11 (XEN) irq.c:236: Dom1 PCI link 3 changed 0 -> 5 (XEN) HVM1: PCI-ISA link 3 routed to IRQ5 (XEN) HVM1: pci dev 01:2 INTA->IRQ10 (XEN) HVM1: pci dev 03:0 INTA->IRQ5 (XEN) HVM1: pci dev 04:0 INTA->IRQ5 (XEN) HVM1: pci dev 02:0 bar 10 size 02000000: f0000008 (XEN) HVM1: pci dev 03:0 bar 14 size 01000000: f2000008 (XEN) HVM1: pci dev 02:0 bar 14 size 00001000: f3000000 (XEN) HVM1: pci dev 03:0 bar 10 size 00000100: 0000c001 (XEN) HVM1: pci dev 04:0 bar 10 size 00000100: 0000c101 (XEN) HVM1: pci dev 04:0 bar 14 size 00000100: f3001000 (XEN) HVM1: pci dev 01:1 bar 20 size 00000010: 0000c201 (XEN) HVM1: Creating MP tables ... (XEN) HVM1: Loading Cirrus VGABIOS ... (XEN) HVM1: Loading ACPI ... (XEN) HVM1: BIOS map: (XEN) HVM1: c0000-c7fff: VGA BIOS (XEN) HVM1: e9000-e9143: SMBIOS tables (XEN) HVM1: ea000-eb1df: ACPI tables (XEN) HVM1: f0000-fffff: Main BIOS (XEN) HVM1: Invoking ROMBIOS ... (XEN) HVM1: rombios.c,v 1.138 2005/05/07 15:55:26 vruppert Exp $ (XEN) HVM1: VGABios $Id: vgabios.c,v 1.61 2005/05/24 16:50:50 vruppert Exp $ (XEN) HVM1: HVMAssist BIOS, 1 cpu, $Revision: 1.138 $ $Date: 2005/05/07 15:55:26 $ (XEN) HVM1: (XEN) HVM1: ata0-0: PCHS=16383/16/63 translation=lba LCHS=1024/255/63 (XEN) HVM1: ata0 master: QEMU HARDDISK ATA-7 Hard-Disk (10240 MBytes) (XEN) HVM1: ata0 slave: Unknown device (XEN) HVM1: ata1 master: QEMU CD-ROM ATAPI-4 CD-Rom/DVD-Rom (XEN) HVM1: ata1 slave: Unknown device (XEN) HVM1: (XEN) HVM1: (XEN) HVM1: (XEN) HVM1: Press F10 to select boot device. (XEN) HVM1: Booting from CD-Rom... (XEN) realmode.c:545:d1 Failed to emulate insn. (XEN) realmode.c:599:d1 Real-mode emulation failed @ 4004:0000b071: 0f 17 0f 01 17 0f (XEN) domain_crash_sync called from realmode.c:600 (XEN) Domain 1 (vcpu#0) crashed on cpu#6: (XEN) ----[ Xen-3.3-unstable x86_64 debug=y Not tainted ]---- (XEN) CPU: 6 (XEN) RIP: 0028:[<000000000000b071>] (XEN) RFLAGS: 0000000000000017 CONTEXT: hvm (XEN) rax: 0000000000004d0b rbx: 0000000000800000 rcx: 0000000000004f00 (XEN) rdx: 00000000000013ca rsi: 0000000000055e1c rdi: 0000000000055e1d (XEN) rbp: 000000000000200b rsp: 00000000ffff007a r8: 0000000000000000 (XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000000000010 cr4: 0000000000000000 (XEN) cr3: 0000000000000000 cr2: 0000000000000000 (XEN) ds: 0020 es: 0020 fs: 0020 gs: 0020 ss: 0020 cs: 0028 [ _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 31/1/08 09:42, "Guillaume Thouvenin" <guillaume.thouvenin@ext.bull.net> wrote:> (XEN) realmode.c:545:d1 Failed to emulate insn. > (XEN) realmode.c:599:d1 Real-mode emulation failed @ 4004:0000b071: 0f 17 0f > 01 17 0f > (XEN) domain_crash_sync called from realmode.c:600 > (XEN) Domain 1 (vcpu#0) crashed on cpu#6:The failing instruction is 0F 17, which is MOVHPS. That''s an SSE instruction, and it''s a bit unlikely to be the very first one we would see. I may try to repro this later. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 31/1/08 09:48, "Keir Fraser" <Keir.Fraser@cl.cam.ac.uk> wrote:> On 31/1/08 09:42, "Guillaume Thouvenin" <guillaume.thouvenin@ext.bull.net> > wrote: > >> (XEN) realmode.c:545:d1 Failed to emulate insn. >> (XEN) realmode.c:599:d1 Real-mode emulation failed @ 4004:0000b071: 0f 17 0f >> 01 17 0f >> (XEN) domain_crash_sync called from realmode.c:600 >> (XEN) Domain 1 (vcpu#0) crashed on cpu#6: > > The failing instruction is 0F 17, which is MOVHPS. That''s an SSE > instruction, and it''s a bit unlikely to be the very first one we would see. > I may try to repro this later.A reasonably simple test would be to modify the CPUID emulation intercept in realmode.c to mask out SSE feature bits. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Guillaume Thouvenin
2008-Jan-31 10:18 UTC
Re: [Xen-devel] Re: Xen 3.2 and Big Real Mode support?
On Thu, 31 Jan 2008 09:53:51 +0000 Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote:> On 31/1/08 09:48, "Keir Fraser" <Keir.Fraser@cl.cam.ac.uk> wrote: > > > On 31/1/08 09:42, "Guillaume Thouvenin" <guillaume.thouvenin@ext.bull.net> > > wrote: > > > >> (XEN) realmode.c:545:d1 Failed to emulate insn. > >> (XEN) realmode.c:599:d1 Real-mode emulation failed @ 4004:0000b071: 0f 17 0f > >> 01 17 0f > >> (XEN) domain_crash_sync called from realmode.c:600 > >> (XEN) Domain 1 (vcpu#0) crashed on cpu#6: > > > > The failing instruction is 0F 17, which is MOVHPS. That''s an SSE > > instruction, and it''s a bit unlikely to be the very first one we would see. > > I may try to repro this later. > > A reasonably simple test would be to modify the CPUID emulation intercept in > realmode.c to mask out SSE feature bits.Ok thanks for the hint. So I will mask bit 25 and 26 of edx (that indicate respectively SSE and SEE2 features) and bit 0 of ecx (that indicates SSE3 feature). I will see if it allows me to boot the installation CD. -- Guillaume Thouvenin _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 31/1/08 10:18, "Guillaume Thouvenin" <guillaume.thouvenin@ext.bull.net> wrote:> On Thu, 31 Jan 2008 09:53:51 +0000 > Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote: > >> On 31/1/08 09:48, "Keir Fraser" <Keir.Fraser@cl.cam.ac.uk> wrote: >> >>> On 31/1/08 09:42, "Guillaume Thouvenin" <guillaume.thouvenin@ext.bull.net> >>> wrote: > > Ok thanks for the hint. > > So I will mask bit 25 and 26 of edx (that indicate respectively SSE and > SEE2 features) and bit 0 of ecx (that indicates SSE3 feature). I will > see if it allows me to boot the installation CD.I reproduced the problem. CPUID is not executed before the crash. Also a disassembly of that area of memory does not look like sane 16-bit code. So I think something has gone wrong before we get to the (probably bogus) MOVHPS instruction. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Guillaume Thouvenin
2008-Jan-31 13:40 UTC
Re: [Xen-devel] Re: Xen 3.2 and Big Real Mode support?
On Thu, 31 Jan 2008 10:36:31 +0000 Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote:> I reproduced the problem. CPUID is not executed before the crash. Also a > disassembly of that area of memory does not look like sane 16-bit code. So I > think something has gone wrong before we get to the (probably bogus) MOVHPS > instruction.A question here. You reproduce the problem by starting a domain and booting it on the openSUSE-10.3-GM-x86_64-mini.iso CD-ROM? How do you disassemble the area of memory that produces the problem? Are you using "objdump" on the bootloader found on the iso file or are you using some other trick? Thanks for your help, -- Guillaume Thouvenin _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 31/1/08 13:40, "Guillaume Thouvenin" <guillaume.thouvenin@ext.bull.net> wrote:> On Thu, 31 Jan 2008 10:36:31 +0000 > Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote: > >> I reproduced the problem. CPUID is not executed before the crash. Also a >> disassembly of that area of memory does not look like sane 16-bit code. So I >> think something has gone wrong before we get to the (probably bogus) MOVHPS >> instruction. > > A question here. You reproduce the problem by starting a domain and > booting it on the openSUSE-10.3-GM-x86_64-mini.iso CD-ROM? How do you > disassemble the area of memory that produces the problem? Are you using > "objdump" on the bootloader found on the iso file or are you using some > other trick?I have a utility to scrape another domain''s memory to a file, and then I run ndisasm on that. The utility is attached. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 31/1/08 13:40, "Guillaume Thouvenin" <guillaume.thouvenin@ext.bull.net> wrote:>> I reproduced the problem. CPUID is not executed before the crash. Also a >> disassembly of that area of memory does not look like sane 16-bit code. So I >> think something has gone wrong before we get to the (probably bogus) MOVHPS >> instruction. > > A question here. You reproduce the problem by starting a domain and > booting it on the openSUSE-10.3-GM-x86_64-mini.iso CD-ROM? How do you > disassemble the area of memory that produces the problem? Are you using > "objdump" on the bootloader found on the iso file or are you using some > other trick?By the way, this is now fixed with tip of the xen-unstable tree (changeset 16980), obtainable from http://xenbits.xensource.com/staging/xen-unstable.hg -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Guillaume Thouvenin
2008-Feb-06 08:20 UTC
Re: [Xen-devel] Re: Xen 3.2 and Big Real Mode support?
On Tue, 05 Feb 2008 16:06:28 +0000 Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote:> On 31/1/08 13:40, "Guillaume Thouvenin" <guillaume.thouvenin@ext.bull.net> > wrote: > > >> I reproduced the problem. CPUID is not executed before the crash. Also a > >> disassembly of that area of memory does not look like sane 16-bit code. So I > >> think something has gone wrong before we get to the (probably bogus) MOVHPS > >> instruction. > > > > A question here. You reproduce the problem by starting a domain and > > booting it on the openSUSE-10.3-GM-x86_64-mini.iso CD-ROM? How do you > > disassemble the area of memory that produces the problem? Are you using > > "objdump" on the bootloader found on the iso file or are you using some > > other trick? > > By the way, this is now fixed with tip of the xen-unstable tree (changeset > 16980), obtainable from http://xenbits.xensource.com/staging/xen-unstable.hgWaow. I don''t understand everything (and especially how you find that the problem was here) but it works now. I have another question, why did you remove VMXASSIST? I saw that you replaced all invocation to VMXASSIST by only one call to vmx_goto_realmode() in arch/x86/hvm/svm/x86_64/exits.S. Are there some links to any discussion about this choice? Thanks for your help, Guillaume _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 6/2/08 08:20, "Guillaume Thouvenin" <guillaume.thouvenin@ext.bull.net> wrote:>> By the way, this is now fixed with tip of the xen-unstable tree (changeset >> 16980), obtainable from http://xenbits.xensource.com/staging/xen-unstable.hg > > Waow. I don''t understand everything (and especially how you find that > the problem was here) but it works now. > > I have another question, why did you remove VMXASSIST? I saw that you > replaced all invocation to VMXASSIST by only one call to > vmx_goto_realmode() in arch/x86/hvm/svm/x86_64/exits.S. Are there some > links to any discussion about this choice?The new emulation replaces vmxassist. There are fundamental limitations to teh vmxassist approach which mean that it could never execute, for example, the openSuSE install CDs because of their use of ''big real'' mode. This is talked about in a few threads on xen-devel, e.g.: http://lists.xensource.com/archives/html/xen-devel/2006-06/msg00005.html It''s bit hard to understand without more context though! Roughly speaking the main pro for emulation is more accurate execution leading to working with a wider range of bootloaders. It''s also less code, and I understand the code (since I wrote it) so I can maintain it better in future. The main con is that it may be a bit slower than vmxassist, since vmxassist did direct execution of most real-mode instructions rather than emulating all of them. But actually most time is spent emulating I/O in both cases, so I think the speed difference is not that important. Now that emulation seems to work reliably in xen-unstable (apart from openSuSE 10.1 CD, argh!) I will remove vmxassist entirely. Also I will backport the emulation fixes to 3.2-testing so that real-mode emulation will work properly in 3.2.1. I will keep VMXASSIST as the default in 3.2 branch though. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 6/2/08 08:20, "Guillaume Thouvenin" <guillaume.thouvenin@ext.bull.net> wrote:>> By the way, this is now fixed with tip of the xen-unstable tree (changeset >> 16980), obtainable from http://xenbits.xensource.com/staging/xen-unstable.hg > > Waow. I don''t understand everything (and especially how you find that > the problem was here) but it works now.I found the bug because I tracked down the real-mode -> protected-mode transition code in the SuSE bootloader and it did something like this at start of protected mode: mov %ss,%eax shl $4,%eax add %eax,%esp mov <protected mode flat segment>,%bx mov %bx,%ss ..... The problem was that the bottom bits of %ss got cleared on exit from real mode, to satisfy vmenter checks that the processor does. But this deliberate corruption of state can of course affect program execution and in this case we end up with a bad stack pointer! So the fix had to be to emulate far enough into protected mode that %cs and %ss both get reloaded with valid protected-mode segment data. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel