Xen devel - Jun 2006 - [PATCH][HVM] Removing 1:1 mapping from qemu-dm

The patch removes the 1:1 mapping against HVM guests from qemu-dm,
allowing to have bigger guests than qemu-dm. This is also the first step
for enabling balloning with HVM guests.

It sets up mapping dynamically, and thus it can cause some latency for
I/O operations for HVM guests. I don''t see visible latency yet, but I
think we need to tune the hashing size.

Signed-off-by: Jun Nakajima <jun.nakajima@intel.com>

 tools/ioemu/cpu-defs.h       |    1 
 tools/ioemu/cpu.h            |    4 -
 tools/ioemu/exec.c           |   32 ++++++++++-
 tools/ioemu/vl.c             |  115
++++++++++++++++++++++++++++++++++++++++---
 tools/ioemu/vl.h             |   23 ++++++++
 xen/arch/x86/mm.c            |    7 ++
 xen/arch/x86/shadow.c        |   33 ++++++------
 xen/arch/x86/shadow32.c      |   25 ++++-----
 xen/arch/x86/shadow_public.c |    2 
 xen/include/asm-x86/shadow.h |  115
++++++++++++++++++++++++++++---------------
 10 files changed, 273 insertions(+), 84 deletions(-)

Jun
---
Intel Open Source Technology Center

 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 28 Jun 2006, at 08:13, Nakajima, Jun wrote:
> The patch removes the 1:1 mapping against HVM guests from qemu-dm,
> allowing to have bigger guests than qemu-dm. This is also the first 
> step
> for enabling balloning with HVM guests.
>
> It sets up mapping dynamically, and thus it can cause some latency for
> I/O operations for HVM guests. I don''t see visible latency yet,
but I
> think we need to tune the hashing size.
What do all the shadow-mode changes do? I see you add an extra 
writable-mapping refcount to tlbflush_timestamp (only 4 bits) -- why is 
that needed?

  -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser wrote:
> On 28 Jun 2006, at 08:13, Nakajima, Jun wrote:
> 
>> The patch removes the 1:1 mapping against HVM guests from qemu-dm,
>> allowing to have bigger guests than qemu-dm. This is also the first
>> step for enabling balloning with HVM guests.
>> 
>> It sets up mapping dynamically, and thus it can cause some latency
>> for I/O operations for HVM guests. I don''t see visible latency
yet,
>> but I think we need to tune the hashing size.
> 
> What do all the shadow-mode changes do? I see you add an extra
> writable-mapping refcount to tlbflush_timestamp (only 4 bits) -- why
> is that needed?
Since the write refcount in type_info can change at any time now because
of map/unmap by qemu-dm at runtime, I stopped using it to avoid a hack
in mm.c. The extra code is a scaled-down version of write refcount
dedicated for guest page table pages, and I think 4 bits would be
sufficient in normal cases because normal guests don''t establish that
many translations (i.e. using different virtual addresses) against page
table pages. 

Since the number cannot exceed the length of the shadow hash chains
anyway, I can add an extra logic that detects overflow and scans the
entire chains if detected. With this we can just have a 2-bit ref count,
0 - no, 1 (most cases), and 2 - more than one.  

Jun
---
Intel Open Source Technology Center

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 28 Jun 2006, at 16:15, Nakajima, Jun wrote:
> Since the write refcount in type_info can change at any time now 
> because
> of map/unmap by qemu-dm at runtime, I stopped using it to avoid a hack
> in mm.c. The extra code is a scaled-down version of write refcount
> dedicated for guest page table pages, and I think 4 bits would be
> sufficient in normal cases because normal guests don''t establish
that
> many translations (i.e. using different virtual addresses) against page
> table pages.
>
> Since the number cannot exceed the length of the shadow hash chains
> anyway, I can add an extra logic that detects overflow and scans the
> entire chains if detected. With this we can just have a 2-bit ref 
> count,
> 0 - no, 1 (most cases), and 2 - more than one.
I''d prefer the hack in mm.c I think. It''s certainly much
smaller, and
it''d be okay with a comment. Is there any way for get/put_page_type to 
know if they''ve been called on a shadow page (as opposed to a guest 
page) and to avoid changing the type count based on that, rather than 
whether the page is local or foreign? That''d be a neater and less hacky
thing to use for the decision imo.

  -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser wrote:
> On 28 Jun 2006, at 16:15, Nakajima, Jun wrote:
> 
>> Since the write refcount in type_info can change at any time now
>> because of map/unmap by qemu-dm at runtime, I stopped using it to
>> avoid a hack in mm.c. The extra code is a scaled-down version of
>> write refcount dedicated for guest page table pages, and I think 4
>> bits would be sufficient in normal cases because normal guests
don''t
>> establish that many translations (i.e. using different virtual
>> addresses) against page table pages. 
>> 
>> Since the number cannot exceed the length of the shadow hash chains
>> anyway, I can add an extra logic that detects overflow and scans the
>> entire chains if detected. With this we can just have a 2-bit ref
>> count, 0 - no, 1 (most cases), and 2 - more than one.
> 
> I''d prefer the hack in mm.c I think. It''s certainly much
smaller, and
> it''d be okay with a comment. Is there any way for
get/put_page_type to
> know if they''ve been called on a shadow page (as opposed to a
guest
> page) and to avoid changing the type count based on that, rather than
> whether the page is local or foreign? That''d be a neater and less
> hacky thing to use for the decision imo.
> 
Basically we need to tell the parent L1 (or top) page that has a
reference to it, for example. If it''s a shadow page, the guest page is
being used in the shadow tree. It would be doable for get_page_type, but
almost impossible for put_page_type given it gets only one argument
(page_info) today.

And actually local vs. foreign does not work well because we need to
terminate and clean up guests from foreign as well. 

What we need is:
1. Don''t change type_info in get/put_page_type if the owner domain is
in
shadow translate mode,
2. Make special get/put_page_type that can change type_info, and they
are called only in shadow page table context. 

Jun
---
Intel Open Source Technology Center

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 28 Jun 2006, at 18:33, Nakajima, Jun wrote:
>> I''d prefer the hack in mm.c I think. It''s certainly
much smaller, and
>> it''d be okay with a comment. Is there any way for
get/put_page_type to
>> know if they''ve been called on a shadow page (as opposed to a
guest
>> page) and to avoid changing the type count based on that, rather than
>> whether the page is local or foreign? That''d be a neater and
less
>> hacky thing to use for the decision imo.
>>
>
> Basically we need to tell the parent L1 (or top) page that has a
> reference to it, for example. If it''s a shadow page, the guest
page is
> being used in the shadow tree. It would be doable for get_page_type, 
> but
> almost impossible for put_page_type given it gets only one argument
> (page_info) today.
Sorry, I actually meant get/put_page_from_l1e(). Does that make the 
problem easier? I''d be prepared to add an extra parameter to those 
functions if really necessary.

  -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser wrote:
> On 28 Jun 2006, at 18:33, Nakajima, Jun wrote:
> 
>>> I''d prefer the hack in mm.c I think. It''s
certainly much smaller,
>>> and it''d be okay with a comment. Is there any way for
>>> get/put_page_type to know if they''ve been called on a
shadow page
>>> (as opposed to a guest page) and to avoid changing the type count
>>> based on that, rather than whether the page is local or foreign?
>>> That''d be a neater and less hacky thing to use for the
decision imo.
>>> 
>> 
>> Basically we need to tell the parent L1 (or top) page that has a
>> reference to it, for example. If it''s a shadow page, the guest
page
>> is being used in the shadow tree. It would be doable for
>> get_page_type, but almost impossible for put_page_type given it gets
>> only one argument (page_info) today.
> 
> Sorry, I actually meant get/put_page_from_l1e(). Does that make the
> problem easier? I''d be prepared to add an extra parameter to those
> functions if really necessary.
> 
That would be sensible ;-) I think we need to add an extra parameter to
those to specify the context (guest or shadow, for example) rather than
who''s calling.

Jun
---
Intel Open Source Technology Center

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 28 Jun 2006, at 19:02, Nakajima, Jun wrote:
>> Sorry, I actually meant get/put_page_from_l1e(). Does that make the
>> problem easier? I''d be prepared to add an extra parameter to
those
>> functions if really necessary.
>>
>
> That would be sensible ;-) I think we need to add an extra parameter to
> those to specify the context (guest or shadow, for example) rather than
> who''s calling.
Since it''s just get/put_page_from_l1e I think we should just duplicate 
the required bits in the shadow code. Could the required parts be 
merged into shadow_get/put_page_from_l1e()? For example, non of the 
I/O-page mapping stuff is needed at all yet. We could for now BUG_ON 
!valid_mfn() and flags in L1_DISALLOW_MASK.

Could you split your original patch into the updated changes to shadow 
mode, and a separate patch for the actual mapcache logic?

  -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser wrote:
> On 28 Jun 2006, at 19:02, Nakajima, Jun wrote:
> 
>>> Sorry, I actually meant get/put_page_from_l1e(). Does that make the
>>> problem easier? I''d be prepared to add an extra parameter
to those
>>> functions if really necessary.
>>> 
>> 
>> That would be sensible ;-) I think we need to add an extra parameter
>> to those to specify the context (guest or shadow, for example)
>> rather than who''s calling.
> 
> Since it''s just get/put_page_from_l1e I think we should just
duplicate
> the required bits in the shadow code. Could the required parts be
> merged into shadow_get/put_page_from_l1e()? For example, non of the
> I/O-page mapping stuff is needed at all yet. We could for now BUG_ON
> !valid_mfn() and flags in L1_DISALLOW_MASK.
Yes, we can merge the required parts into
shadow_get/put_page_from_l1e().
We still need to change get/put_page_from_l1e to prevent qemu-dm from
changing type_info, but we don''t need to add an extra parameter that
way.
> 
> Could you split your original patch into the updated changes to shadow
> mode, and a separate patch for the actual mapcache logic?
> 
I''ll do that. 

Jun
---
Intel Open Source Technology Center

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 28 Jun 2006, at 20:51, Nakajima, Jun wrote:
>> Since it''s just get/put_page_from_l1e I think we should just
duplicate
>> the required bits in the shadow code. Could the required parts be
>> merged into shadow_get/put_page_from_l1e()? For example, non of the
>> I/O-page mapping stuff is needed at all yet. We could for now BUG_ON
>> !valid_mfn() and flags in L1_DISALLOW_MASK.
>
> Yes, we can merge the required parts into
> shadow_get/put_page_from_l1e().
> We still need to change get/put_page_from_l1e to prevent qemu-dm from
> changing type_info, but we don''t need to add an extra parameter
that
> way.
Ah yes. Testing for whether the page owner is shadow_mode_refcount 
should do? We can add a comment to explain why we do the test.
>>
>> Could you split your original patch into the updated changes to shadow
>> mode, and a separate patch for the actual mapcache logic?
>>
>
> I''ll do that.
Thanks!

  -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel