Xen devel - Mar 2007 - [PATCH][RFC] Emulating real mode with x86_emulate

Howdy,

Attached is a patch that begins to lay down the infrastructure for 
emulating real mode with x86_emulate().  With a little more refactoring, 
I think it could also replace the SVM emulator.

The patch introduces an HVMOP hypercall to set a flag in the hvm vcpu 
struct to signal that real mode should be emulated with x86_emulate 
instead of using vm86.  This is to make development a little bit easier 
since x86_emulate is not quite there yet wrt 16 bit emulation.  It can 
be enabled by passing -emulate-16bit to qemu-dm (I use a wrapper script 
similar to qemu-dm.debug).

The VT code keeps track of the whether it''s in the emulator and loops
on
the do_resume path in x86_emulate.  I think this code probably should be 
refactored into the common HVM code although this would require changing 
some of the HVM ops.  This would allow SVM to use the x86_emulate to 
handle individual instructions.

There are some issues to work out.  Namely, x86_emulate appears to want 
blocking PIO calls which isn''t conducive to the wait PIO works today in
HVM.  This is only a problem for instructions at the moment.  I''m also
a
bit confused about how to properly loop in the emulator.  schedule_tail 
is not meant to return so perhaps we should loop on emulating == 1 
instead of hypercall_preempt_check?  I didn''t think the hypervisor was 
preemptable though.

The current code doesn''t handle non-flat segments as I don''t
think
hvm_copy_from/to_guest handles it (which I assume it would need to).

However, it is enough to start running instructions in x86_emulate so 
it''s enough to start working on enhancing that.

Regards,

Anthony Liguori



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> There are some hard problems with migrating into QEMU.  Namely, the code 
> that QEMU generates isn''t SMP safe (atomic instructions lose their
> atomicity)
I see.

You''re worrying that real mode startup trampolines of OS rely 
on atomic instructions?  I can''t think of much other possible real mode
multi CPU code.

-Andi

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Anthony Liguori <aliguori@us.ibm.com> writes:
> Attached is a patch that begins to lay down the infrastructure for
> emulating real mode with x86_emulate().  With a little more
> refactoring, I think it could also replace the SVM emulator.
I thought the roadmap direction of this was to migrate the context
into qemu and let its JIT handle it? Or has that been dropped?

-Andi

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Fri, 2007-03-30 at 15:11 -0700, Anthony Liguori wrote:

> set_cr0 is returning 1 though which should increment eip to the next
> instruction.
> 
> I''m a bit perplexed about my eip now and also why your eip is
still
> 0. 
> It should be the instruction following the mov cr0.
> 
> Regards,
> 
> Anthony Liguori
> 

Hi Anthony,
    I don''t see any code doing context save/restore like
vmx_world_save , vmx_world_restore in the current code for the
hyperviser based emulator.
Also the function  arch_vmx_do_resume() is called at the time of vcpu
schedule, so it is not right to call the  vmx_do_emulate() from there.

The code would need vmx_world_save/restore in the code path before
returning to vmx_asm_vmexit_handler from vmx_vmexit_handler.
Without that I don''t see it can emulate any instructions.

Thanks & Regards,
Nitin 
Open Source Technology Center, Intel Corporation.
-------------------------------------------------------------------------
The mind is like a parachute; it works much better when it''s open. 



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Kamble, Nitin A wrote:
> On Fri, 2007-03-30 at 15:11 -0700, Anthony Liguori wrote:
>
>> set_cr0 is returning 1 though which should increment eip to the next
>> instruction.
>>
>> I''m a bit perplexed about my eip now and also why your eip is
still 0.
>> It should be the instruction following the mov cr0.
>>
>> Regards,
>>
>> Anthony Liguori
>>
>
> Hi Anthony,
>     I don''t see any code doing context save/restore like 
> vmx_world_save , vmx_world_restore in the current code for the 
> hyperviser based emulator.
Before calling x86_emulate, we use hvm_store_cpu_guest_regs() to copy 
the guest state into a regs struct (which happens to be the vcpu''s reg 
struct).  This reads directly from the VMCS via vmread() so it should be 
okay.  I don''t think a vmx_world_save/restore is actually needed 
although perhaps I''m missing something?
> Also the function  arch_vmx_do_resume() is called at the time of vcpu 
> schedule, so it is not right to call the  vmx_do_emulate() from there.
Right, the idea was to have x86_emulate() be called instead of 
vmentry().  I thought that being in the set_cr0 path would ensure that 
we go through do_resume() again.  Is this assumption incorrect?

I didn''t want to just stick it in the set_cr0 path because we ought to 
be able to pull the emulation loop into common code for SVM/VT and the 
do_resume path seems like the only place where there''s common place to 
hook right now.

Regards,

Anthony Liguori
> The code would need vmx_world_save/restore in the code path before 
> returning to vmx_asm_vmexit_handler from vmx_vmexit_handler.
> Without that I don''t see it can emulate any instructions.
>
> Thanks & Regards,
> Nitin
> Open Source Technology Center, Intel Corporation.
> -------------------------------------------------------------------------
> The mind is like a parachute; it works much better when it''s open.
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Mon, 2007-04-02 at 11:54 -0700, Anthony Liguori wrote:
> Before calling x86_emulate, we use hvm_store_cpu_guest_regs() to copy
> the guest state into a regs struct (which happens to be the vcpu''s
reg
> struct).  This reads directly from the VMCS via vmread() so it should
> be
> okay.  I don''t think a vmx_world_save/restore is actually needed
> although perhaps I''m missing something?
It may be ok to use hvm_store_cpu_guest_regs() for 1st few instructions,
but I think it is not complete enough for emulator use.
> 
> > Also the function  arch_vmx_do_resume() is called at the time of
> vcpu
> > schedule, so it is not right to call the  vmx_do_emulate() from
> there.
> 
> Right, the idea was to have x86_emulate() be called instead of
> vmentry().  I thought that being in the set_cr0 path would ensure that
> we go through do_resume() again.  Is this assumption incorrect?
Yes, This assumption is not right. arch_vmx_do_resume() is assigned to
schedule tail, so that the vcpu context is saved/restored when another
vcpu is scheduled on the physical cpu.
> 
> I didn''t want to just stick it in the set_cr0 path because we
ought to
> be able to pull the emulation loop into common code for SVM/VT and the
> do_resume path seems like the only place where there''s common
place to
> hook right now.
I thought the emulator will be needed only for VMX; why is it needed for
SVM?

Also calling the x86_emulate() to emulate multiple instructions from
vmx_do_resume() will block the physical cpu from other vcpus. 
I think we discussed the approach of using the non-root context for for
emulator within the Xen. Or did I misunderstanding it?

> Regards,
> 
> Anthony Liguori
> 

Thanks & Regards,
Nitin 
Open Source Technology Center, Intel Corporation.
-------------------------------------------------------------------------
The mind is like a parachute; it works much better when it''s open. 



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 3/4/07 00:52, "Kamble, Nitin A" <nitin.a.kamble@intel.com>
wrote:
>>  Right, the idea was to have x86_emulate() be called instead of
>>  vmentry().  I thought that being in the set_cr0 path would ensure that
>>  we go through do_resume() again.  Is this assumption incorrect?
> Yes, This assumption is not right. arch_vmx_do_resume() is assigned to
> schedule tail, so that the vcpu context is saved/restored when another vcpu
is
> scheduled on the physical cpu.
> 
If you want to keep the check off the Œfast¹ return path (which actually
isn¹t all that fast as yet, sadly) then you can kick the SCHEDULE_SOFTIRQ
from the set_cr0 path. This will take you the really slow path, through the
scheduler, but will ultimately guarantee that you go through
arch_vmx_do_resume() before returning to guest context.
>>  
>>  I didn''t want to just stick it in the set_cr0 path because we
ought to
>>  be able to pull the emulation loop into common code for SVM/VT and the
>>  do_resume path seems like the only place where there''s common
place to
>>  hook right now.
> I thought the emulator will be needed only for VMX; why is it needed for
SVM?
> 
It would be really nice to make as much of this infrastructure as possible
HVM-generic so that any HVM implementation can fall back to emulation to
handle corner cases. In SVM case this may just be a few tricky instructions
(SMSW springs to mind), but we¹d like to share code if
possible.
> 
> Also calling the x86_emulate() to emulate multiple instructions from
> vmx_do_resume() will block the physical cpu from other vcpus.
> I think we discussed the approach of using the non-root context for for
> emulator within the Xen. Or did I misunderstanding it?
I think you did. It¹s sufficient to check for softirq work at
emulated-instruction boundaries.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Kamble, Nitin A wrote:
> On Mon, 2007-04-02 at 11:54 -0700, Anthony Liguori wrote:
>> Before calling x86_emulate, we use hvm_store_cpu_guest_regs() to copy
>> the guest state into a regs struct (which happens to be the
vcpu''s reg
>> struct).  This reads directly from the VMCS via vmread() so it should
be
>> okay.  I don''t think a vmx_world_save/restore is actually
needed
>> although perhaps I''m missing something?
> It may be ok to use hvm_store_cpu_guest_regs() for 1st few 
> instructions, but I think it is not complete enough for emulator use.
What is missing?  x86_emulate() only uses info in the regs (it calls out 
to function pointers for special registers).  The GP registers should be 
kept up-to-date on vmexit and hvm_store_cpu_guest_regs() should sync the 
remainder of the register state.

Is there a specific item you think is missing?
>>
>> > Also the function  arch_vmx_do_resume() is called at the time of
vcpu
>> > schedule, so it is not right to call the  vmx_do_emulate() from
there.
>>
>> Right, the idea was to have x86_emulate() be called instead of
>> vmentry().  I thought that being in the set_cr0 path would ensure that
>> we go through do_resume() again.  Is this assumption incorrect?
> Yes, This assumption is not right. arch_vmx_do_resume() is assigned to 
> schedule tail, so that the vcpu context is saved/restored when another 
> vcpu is scheduled on the physical cpu.
Hrm, okay.  Manually invoking the scheduler is probably a reasonable 
place to start.  It would be nice to clean things up though so that 
wasn''t necessary.
>>
>> I didn''t want to just stick it in the set_cr0 path because we
ought to
>> be able to pull the emulation loop into common code for SVM/VT and the
>> do_resume path seems like the only place where there''s common
place to
>> hook right now.
> I thought the emulator will be needed only for VMX; why is it needed 
> for SVM?
As Keir mentioned, there are some corner cases where emulation is 
needed.  Also, there is some opportunity to simplify things by using the 
emulator.  For instance, instead of decoding a PIO instruction using the 
info in the VMCS/VMCB (none of which is actually common to VT/SVM), we 
may find that it is simpler to just call out to x86_emulate() and let it 
decode and dispatch the PIO operation.

In fact, a large number of the exits can be handled in this way.  I have 
no clue if this would impact performance in a significant way but it 
would definitely simplify things.
> Also calling the x86_emulate() to emulate multiple instructions from 
> vmx_do_resume() will block the physical cpu from other vcpus.
That''s what the hypercall_preempt_check() is for.
> I think we discussed the approach of using the non-root context for 
> for emulator within the Xen. Or did I misunderstanding it?
We discussed quite a few approaches :-)  I thought we settled on doing 
the emulation within Xen.  I''m not sure what would be gained from a 
non-root context other than better security assurances.

Regards,

Anthony Liguori
>> Regards,
>>
>> Anthony Liguori
>>
>
> Thanks & Regards,
> Nitin
> Open Source Technology Center, Intel Corporation.
> -------------------------------------------------------------------------
> The mind is like a parachute; it works much better when it''s open.
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Anthony, Keir,
  As per your suggestion I added the SCHEDULE_SOFTIRQ in the set_cr0
path. And now I see the code path going to vmx_do_emulate() for
emulation. I added the emulation code for ljmp and it is able to go
forward 1 instruction now.

(XEN) HVM2: Loading Cirrus
VGABIOS ...                                          
(XEN) HVM2: Loading
ACPI ...                                                    
(XEN) HVM2: Loading VMXAssist ...
deadbeef                                      
(XEN) HVM2:
foo                                                                 
(XEN)
hvmop_emulate_realmode                                                    
(XEN) guest requests real mode
emulation                                        
(XEN) foo
221                                                                   
(XEN) HVM2: Invoking
ROMBIOS ...                                                
(XEN) vmx_vmexit_handler called. eip 0x0                                      
(XEN) vmx_cr_access called
eip=0x0                                              
(XEN) mov_to_cr 0 called
eip=0x0                                                
(XEN) vmx_set_cr0 called
eip=0x0                                                
(XEN) Transfering -- control to x86_emulate eip
0x0                             
(XEN) cr access done: updating eip from eip 0x0                               
(XEN) updated
eip=0x3                                                           
(XEN) arch_vmx_do_resume: starting
emulation                                    
(XEN) vmx_do_emulate:
hvm_store_cpu_guest_regs                                  
(XEN) vmx_do_emulate: hypercall_preempt_check() 0                             
(XEN) vmx_do_emulate: cs:eip 0008:0003                                        
(XEN) x86_emulate realmode:
0008:0003                                           
(XEN) jmp far seg:offset (Xea)
handled                                          
(XEN) vmx_do_emulate: cs:eip 0000:0008                                        
(XEN) x86_emulate realmode:
0000:0008                                           
(XEN) x86_emulate.c:2441:d2 Instr:
8e                                           
(XEN) failed to emulate instruction at eip 0x8                                
(XEN) domain_crash_sync called from
vmcs.c:625                                  
(XEN) Domain 2 (vcpu#0) crashed on
cpu#0:                                       
(XEN) ----[ Xen-3.0-unstable  x86_32p  debug=n  Not
tainted ]----               
(XEN) CPU:
0                                                                 
(XEN) EIP:
0008:[<00000003>]                                                 
(XEN) EFLAGS: 00000046   CONTEXT:
hvm                                           
(XEN) eax: 00000000   ebx: 00000000   ecx: 00000000   edx:
00000000             
(XEN) esi: 00000000   edi: 00000000   ebp: 00000000   esp:
00000000             
(XEN) cr0: 00000000   cr4: 00000650   cr3: 00000000   cr2:
00000000             
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0010   cs: 0008  

I think getting more instruction emulated will be easy now. I will keep
posted about the progress.

Thanks & Regards,
Nitin 
Open Source Technology Center, Intel Corporation.
-------------------------------------------------------------------------
The mind is like a parachute; it works much better when it''s open. 

On Tue, 2007-04-03 at 07:03 -0700, Anthony Liguori wrote:
> Kamble, Nitin A wrote:
> > On Mon, 2007-04-02 at 11:54 -0700, Anthony Liguori wrote:
> >> Before calling x86_emulate, we use hvm_store_cpu_guest_regs() to
> copy
> >> the guest state into a regs struct (which happens to be the
vcpu''s
> reg
> >> struct).  This reads directly from the VMCS via vmread() so it
> should be
> >> okay.  I don''t think a vmx_world_save/restore is actually
needed
> >> although perhaps I''m missing something?
> > It may be ok to use hvm_store_cpu_guest_regs() for 1st few
> > instructions, but I think it is not complete enough for emulator
> use.
> 
> What is missing?  x86_emulate() only uses info in the regs (it calls
> out
> to function pointers for special registers).  The GP registers should
> be
> kept up-to-date on vmexit and hvm_store_cpu_guest_regs() should sync
> the
> remainder of the register state.
> 
> Is there a specific item you think is missing?
> 
> >>
> >> > Also the function  arch_vmx_do_resume() is called at the time
of
> vcpu
> >> > schedule, so it is not right to call the  vmx_do_emulate()
from
> there.
> >>
> >> Right, the idea was to have x86_emulate() be called instead of
> >> vmentry().  I thought that being in the set_cr0 path would ensure
> that
> >> we go through do_resume() again.  Is this assumption incorrect?
> > Yes, This assumption is not right. arch_vmx_do_resume() is assigned
> to
> > schedule tail, so that the vcpu context is saved/restored when
> another
> > vcpu is scheduled on the physical cpu.
> 
> Hrm, okay.  Manually invoking the scheduler is probably a reasonable
> place to start.  It would be nice to clean things up though so that
> wasn''t necessary.
> 
> >>
> >> I didn''t want to just stick it in the set_cr0 path
because we ought
> to
> >> be able to pull the emulation loop into common code for SVM/VT and
> the
> >> do_resume path seems like the only place where there''s
common place
> to
> >> hook right now.
> > I thought the emulator will be needed only for VMX; why is it needed
> > for SVM?
> 
> As Keir mentioned, there are some corner cases where emulation is
> needed.  Also, there is some opportunity to simplify things by using
> the
> emulator.  For instance, instead of decoding a PIO instruction using
> the
> info in the VMCS/VMCB (none of which is actually common to VT/SVM), we
> may find that it is simpler to just call out to x86_emulate() and let
> it
> decode and dispatch the PIO operation.
> 
> In fact, a large number of the exits can be handled in this way.  I
> have
> no clue if this would impact performance in a significant way but it
> would definitely simplify things.
> 
> > Also calling the x86_emulate() to emulate multiple instructions from
> > vmx_do_resume() will block the physical cpu from other vcpus.
> 
> That''s what the hypercall_preempt_check() is for.
> 
> > I think we discussed the approach of using the non-root context for
> > for emulator within the Xen. Or did I misunderstanding it?
> 
> We discussed quite a few approaches :-)  I thought we settled on doing
> the emulation within Xen.  I''m not sure what would be gained from
a
> non-root context other than better security assurances.
> 
> Regards,
> 
> Anthony Liguori
> 
> >> Regards,
> >>
> >> Anthony Liguori
> >>




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel