Xen devel - May 2006 - Re-using the x86_emulate_memop() to perform MMIO for HVM.

As stated earlier, I started looking at a using the x86_emulate_memop()
to support HVM''s memory-mapped IO emulation - since that would remove
the need to have two x86 instruction decode paths, and a whole heap of
other semi-duplicated code. Since these are fairly major chunks of code
in each place (not to mention that they are not entirely trivial to
understand in either place), it would be a "Good Thing(tm)" to combine
those bits of code. 

I got as far as I can clear the screen with the BIOS, but I ran into a
bit of a problem: The mmio-request that goes to QEMU needs to be ONE
"atomic" operation, as when we send the request to QEMU, Xen
schedules,
and eventually comes back to xxx_resume(), which is not where we need to
be to continue a read-modify-write operation. 

In xen/x86/hvm/platform.c, this is solved by figuring out the entire
operation, and based on that, does a RMW operation to QEMU (such as
IOREQ_TYPE_AND) and we therefore don''t have to wait for the read
operation to finish before continuing the write phase. 

As I see it, there''s several possibilities to solve this, but none of
them are particularly trivial to implement. 

The easiest one would be to supply a bigger set of function pointers to
x86_mem_emulator, such as and_emulated, or_emulated, xor_emulated, etc.
We could make those optional, and choose the "old" or "new"
method based
on the pointer being set to something or not. 

Another possibility would be to split the x86_emulate_memop() up, so
that we can point schedule_tail to the second half of it if necessary -
but I definitely don''t like this idea [I am not sure that it would even
work to do this - I haven''t actually looked at it]. 

A third, easier, but less pleasing way to solve it would be retain the
current two decode/emulate code-paths, and just add everythign twice
when new scenarios are needed to be decoded - I don''t quite like this
idea, but it certainly is the least amount of effort to implement!

Thoughts and comments (aside from the obvious "You should have thought
about this earlier!" ;-) would be welcome... 

--
Mats


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 3 May 2006, at 15:50, Petersson, Mats wrote:
> A third, easier, but less pleasing way to solve it would be retain the
> current two decode/emulate code-paths, and just add everythign twice
> when new scenarios are needed to be decoded - I don''t quite like
this
> idea, but it certainly is the least amount of effort to implement!
>
> Thoughts and comments (aside from the obvious "You should have thought
> about this earlier!" ;-) would be welcome...
We need an emulator both in Xen and in the device model. The current 
split decode-emulate is pretty barking. My plan for now would be to 
copy x86_emulate.c and plumb it into qemu-dm: so we do duplicate the 
code but it''s actually only one source file to maintain.

So, Xen would take the page fault and look up the corresponding mmio 
area. If it''s an area corresponding to a device emulated by qemu-dm 
then Xen hands off the entire problem. It does not bother to decode the 
instruction at all. Instead it stuffs register state into the shared 
memory area and hands off to qemu-dm in dom0.

There are plans afoot to try using the full qemu cpu emulator. Done 
well, that would provide fuller and faster emulation, but there is a 
bigger up-front cost to getting that plumbed in correctly and even if 
someone runs with this plan I don''t see it being completed to a high 
degree of robustness in the very near future.

  -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> -----Original Message-----
> From: Keir Fraser [mailto:Keir.Fraser@cl.cam.ac.uk] 
> Sent: 04 May 2006 10:10
> To: Petersson, Mats
> Cc: xen-devel; Khoa Huynh
> Subject: Re: Re-using the x86_emulate_memop() to perform MMIO for HVM.
> 
> 
> On 3 May 2006, at 15:50, Petersson, Mats wrote:
> 
> > A third, easier, but less pleasing way to solve it would be 
> retain the 
> > current two decode/emulate code-paths, and just add 
> everythign twice 
> > when new scenarios are needed to be decoded - I don''t quite 
> like this 
> > idea, but it certainly is the least amount of effort to implement!
> >
> > Thoughts and comments (aside from the obvious "You should 
> have thought 
> > about this earlier!" ;-) would be welcome...
> 
> We need an emulator both in Xen and in the device model. The 
> current split decode-emulate is pretty barking. My plan for 
> now would be to copy x86_emulate.c and plumb it into qemu-dm: 
> so we do duplicate the code but it''s actually only one source 
> file to maintain.
That does indeed sound like a good plan. 

And it sounds like it would work. But isn''t the "emulate within
Xen"
going to have a problem with that? Or do we use the Xen version of
x86_emulate for the Xen devices (as we can obviously read/write those
without switching to another context, and thus don''t have the problems
I''ve been hitting). 

So what you''re essentially saying is make a soft link from current
x86_emulate.[ch] into the tools/ioemu/, and set up suitable read/write
functions, and then use that in helper2.c, yes? [Sorry if I''m asking
obvious questions, but it''s usually better to ask first than to have to
do things twice because you didn''t ask...]
> 
> So, Xen would take the page fault and look up the 
> corresponding mmio area. If it''s an area corresponding to a 
> device emulated by qemu-dm then Xen hands off the entire 
> problem. It does not bother to decode the instruction at all. 
> Instead it stuffs register state into the shared memory area 
> and hands off to qemu-dm in dom0.
Makes life a whole lot simpler, I should think - but for the MMX/SSE
support, that would mean that we need to FXSAVE/FXRSTOR those around
this code, right? Will that not cost unnecessarily much?

I guess we should also supply the segment base (and limit?) information
in this memory block, so that x86_emulate can do things like
big-real-mode and other segmented operations that may happen at some
point. 

Anything else we should think to pass along as well "while we''re
at it"?
> 
> There are plans afoot to try using the full qemu cpu 
> emulator. Done well, that would provide fuller and faster 
> emulation, but there is a bigger up-front cost to getting 
> that plumbed in correctly and even if someone runs with this 
> plan I don''t see it being completed to a high degree of 
> robustness in the very near future.
Yes, I saw that discussion. I''m not sure if it''s much help to
do that
(for AMD at least, Intel has a problem because they don''t support
paged-real-mode, which of course is a bit of a nuisance to them...)

--
Mats
> 
>   -- Keir

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>> We need an emulator both in Xen and in the device model. The
>> current split decode-emulate is pretty barking. My plan for
>> now would be to copy x86_emulate.c and plumb it into qemu-dm:
>> so we do duplicate the code but it''s actually only one source
>> file to maintain.
>
> That does indeed sound like a good plan.
>
> And it sounds like it would work. But isn''t the "emulate
within Xen"
> going to have a problem with that? Or do we use the Xen version of
> x86_emulate for the Xen devices (as we can obviously read/write those
> without switching to another context, and thus don''t have the
problems
> I''ve been hitting).
Yes, you''d only invoke qemu-dm for the non-Xen devices.
> So what you''re essentially saying is make a soft link from current
> x86_emulate.[ch] into the tools/ioemu/, and set up suitable read/write
> functions, and then use that in helper2.c, yes? [Sorry if I''m
asking
> obvious questions, but it''s usually better to ask first than to
have to
> do things twice because you didn''t ask...
Pretty much. As you say below, we also need to provide access to 
segment bases. But actually most instructions are okay because we know 
the faulting linear address of one of the memory operands (and usually 
there is only one).
> Makes life a whole lot simpler, I should think - but for the MMX/SSE
> support, that would mean that we need to FXSAVE/FXRSTOR those around
> this code, right? Will that not cost unnecessarily much?
I think it''s in the noise compared with the context-switch cost. 
However we could save the FPU state on demand, only if it turns out 
that qemu-dm needs it. FXSAVE will only be needed if the guest has 
actually been using the FPU. Otherwise we already have the up-to-date 
FPU state saved in memory.
> I guess we should also supply the segment base (and limit?) information
> in this memory block, so that x86_emulate can do things like
> big-real-mode and other segmented operations that may happen at some
> point.
Yes, definitely, and this will require some modifications to the 
emulator itself. Either an extra block of state passed in to the 
emulator, or adding a call-out function hook from the emulator to 
obtain segment bases on demand. I think the former is probably simpler.
> Anything else we should think to pass along as well "while
we''re at
> it"?
No, I think that''s it.
> Yes, I saw that discussion. I''m not sure if it''s much
help to do that
> (for AMD at least, Intel has a problem because they don''t support
> paged-real-mode, which of course is a bit of a nuisance to them...)
Well, sometimes device register accesses happen in clusters, and it 
would be nice to amortise the cost of switching to qemu-dm across 
emulation of an entire cluster of accesses. It might well be a win in 
many cases, but it''s also fair amount of work I think.

  -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote on 05/04/2006 04:10:06 AM:
>
> On 3 May 2006, at 15:50, Petersson, Mats wrote:
>
> > A third, easier, but less pleasing way to solve it would be retain the
> > current two decode/emulate code-paths, and just add everythign twice
> > when new scenarios are needed to be decoded - I don''t quite
like this
> > idea, but it certainly is the least amount of effort to implement!
> >
> > Thoughts and comments (aside from the obvious "You should have
thought
> > about this earlier!" ;-) would be welcome...
>
> We need an emulator both in Xen and in the device model. The current
> split decode-emulate is pretty barking. My plan for now would be to
> copy x86_emulate.c and plumb it into qemu-dm: so we do duplicate the
> code but it''s actually only one source file to maintain.
Would this be sufficient to support real mode in qemu-dm with
x86_emulate in it (at least for Intel) ?
>
> So, Xen would take the page fault and look up the corresponding mmio
> area. If it''s an area corresponding to a device emulated by
qemu-dm
> then Xen hands off the entire problem. It does not bother to decode the
> instruction at all. Instead it stuffs register state into the shared
> memory area and hands off to qemu-dm in dom0.
Sounds like a good thing.  The path length for MMIO would be more or
less the same as it is now since most of it would be the cost of the
upcall into qemu-dm, which is there anyway.

This would place additional burden on dom0 until all of this
is moved into a mini guest.

I guess the *PIC MMIO and Xen-emulated device I/O would remain
in the hypervisor and use the x86-emulate copy in the hypervisor,
there''s no change for those...

Regards,
Khoa


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote on 05/04/2006 04:10:06
AM:
> 
> >
> > On 3 May 2006, at 15:50, Petersson, Mats wrote:
> >
> > > A third, easier, but less pleasing way to solve it would be
retain the
> > > current two decode/emulate code-paths, and just add everythign
twice
> > > when new scenarios are needed to be decoded - I don''t
quite like this
> > > idea, but it certainly is the least amount of effort to
implement!
> > >
> > > Thoughts and comments (aside from the obvious "You should
have thought
> > > about this earlier!" ;-) would be welcome...
> >
> > We need an emulator both in Xen and in the device model. The current
> > split decode-emulate is pretty barking. My plan for now would be to
> > copy x86_emulate.c and plumb it into qemu-dm: so we do duplicate the
> > code but it''s actually only one source file to maintain.
> 
> Would this be sufficient to support real mode in qemu-dm with
> x86_emulate in it (at least for Intel) ?
If we were to run all real mode in x86_emulate then it would need
extending for quite a few more instructions. Currently it handles only
instructions with at least one memory operand. I don''t think
it''s that
bad though, but it depends how complete we want to be (e.g., FPU
instructions). Are we just trying to get a good range of bootloaders
running robustly, or do we want to run MS-DOS and Win95 with decent
performance? The latter would in any case probably require something
more powerful (like the qemu cpu emulation engine) as running entirely
on x86_emulate is not going to have very good performance.

 -- Keir

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel