Xen devel - Aug 2006 - [PATCH] support HVM guests with more than 3.75G memory

support HVM guests with more than 3.75G memory.
Changes are:
1) M2P table and e820 table are changed to skip address space from
HVM_RAM_LIMIT_BELOW_4G to 4G.
2) shared io page location, when less than HVM_RAM_LIMIT_BELOW_4G
memory, it''s the last page of RAM as today, or it''s the last
page of
HVM_RAM_LIMIT_BELOW_4G RAM.
3) in qemu-dm address space from HVM_RAM_LIMIT_BELOW_4G to 4G are
stuffed with mfns starting from 4G, so the 1:1 mapping can still works.
This is ugly, but another limit check patch as changeset 10757 will
prevent qemu-dm to access this range.  This ugly stuffing will be
removed when the patch to remove 1:1 mapping from qemu-dm gets accepted
in the future.

Signed-off-by: Xin Li <xin.b.li@intel.com>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Xin,

This patch seems to mess with the e820 logic quite a bit, in particular
adding some extra entries. What''s going on there -- is it all just a
''clean
up'' or are there real changes going on in that section of the patch?

 -- Keir

On 10/8/06 10:51 am, "Li, Xin B" <xin.b.li@intel.com> wrote:
> support HVM guests with more than 3.75G memory.
> Changes are:
> 1) M2P table and e820 table are changed to skip address space from
> HVM_RAM_LIMIT_BELOW_4G to 4G.
> 2) shared io page location, when less than HVM_RAM_LIMIT_BELOW_4G
> memory, it''s the last page of RAM as today, or it''s the
last page of
> HVM_RAM_LIMIT_BELOW_4G RAM.
> 3) in qemu-dm address space from HVM_RAM_LIMIT_BELOW_4G to 4G are
> stuffed with mfns starting from 4G, so the 1:1 mapping can still works.
> This is ugly, but another limit check patch as changeset 10757 will
> prevent qemu-dm to access this range.  This ugly stuffing will be
> removed when the patch to remove 1:1 mapping from qemu-dm gets accepted
> in the future.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Keir, just rebased it to 10199, can you pls take a look?
Thanks
-Xin 

>
>support HVM guests with more than 3.75G memory.
>Changes are:
>1) M2P table and e820 table are changed to skip address space from
>HVM_RAM_LIMIT_BELOW_4G to 4G.
>2) shared io page location, when less than HVM_RAM_LIMIT_BELOW_4G
>memory, it''s the last page of RAM as today, or it''s the
last page of
>HVM_RAM_LIMIT_BELOW_4G RAM.
>3) in qemu-dm address space from HVM_RAM_LIMIT_BELOW_4G to 4G are
>stuffed with mfns starting from 4G, so the 1:1 mapping can still works.
>This is ugly, but another limit check patch as changeset 10757 will
>prevent qemu-dm to access this range.  This ugly stuffing will be
>removed when the patch to remove 1:1 mapping from qemu-dm gets accepted
>in the future.
>
>Signed-off-by: Xin Li <xin.b.li@intel.com>
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 15/8/06 4:32 pm, "Li, Xin B" <xin.b.li@intel.com> wrote:
> Hi Keir, just rebased it to 10199, can you pls take a look?
> Thanks
> -Xin 
My previous response:

Hi Xin,

This patch seems to mess with the e820 logic quite a bit, in particular
adding some extra entries. What''s going on there -- is it all just a
''clean
up'' or are there real changes going on in that section of the patch?

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>My previous response:
Oops, I missed your previous response :-(
>
>Hi Xin,
>
>This patch seems to mess with the e820 logic quite a bit, in particular
>adding some extra entries. What''s going on there -- is it all 
>just a ''clean
>up'' or are there real changes going on in that section of the
patch?
>
It''s "real changes" and the issue here is, in current HVM
guest memory
layout, we don''t reserve any guest physical address space for PCI MMIO
uses. On the other hand, qemu-dm PCI BIOS allocates PCI MMIO from
0xF0000000, that''s meaning the last 256M physical address space (below
4G) can not be regarded as normal RAM address space.  But current e820
table is blind to this :-(, and we already encountered a bug that
windows 2003 VMX guest only saw 1G RAM while we gave 5G RAM to it.

On a real platform with 8G RAM, it''s e820 table is like this:
(XEN)  0000000000000000 - 000000000009dc00 (usable) 
(XEN)  000000000009dc00 - 00000000000a0000 (reserved) 
(XEN)  00000000000e4000 - 0000000000100000 (reserved) 
(XEN)  0000000000100000 - 00000000bff60000 (usable) 
(XEN)  00000000bff60000 - 00000000bff69000 (ACPI data) 
(XEN)  00000000bff69000 - 00000000bff80000 (ACPI NVS) 
(XEN)  00000000bff80000 - 00000000c0000000 (reserved) 
(XEN)  00000000e0000000 - 00000000f0000000 (reserved)
(XEN)  00000000fec00000 - 00000000fec10000 (reserved) 
(XEN)  00000000fee00000 - 00000000fee01000 (reserved)
(XEN)  00000000ff000000 - 0000000100000000 (reserved) 
(XEN)  0000000100000000 - 0000000240000000 (usable)
 
8G is split into 2 parts: 3G and 5G:
1) 0 - 3G
2) 4G - 9G

So our HVM e820 table should also comply to this, and the 256M RAM that
falls into that range should be moved to 4G above, that''s the
"real
changes" in my patch.

-Xin

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>So our HVM e820 table should also comply to this, and the 256M RAM that
>falls into that range should be moved to 4G above, that''s the
"real
>changes" in my patch.
Thanks, I saw it''s checked in as changeset 11186.

But it can not work with shadow2, when I assigned 4096M memory, I got:

(XEN) ----[ Xen-3.0-unstable    Not tainted ]----
(XEN) CPU:    3
(XEN) RIP:    e010:[<ffff83000013c267>]
sh2_guess_wrmap__shadow_3_guest_3+0x67/0xd0
(XEN) RFLAGS: 0000000000010202   CONTEXT: hypervisor
(XEN) rax: ffff8140c0000000   rbx: 00000000001aed0d   rcx:
00000000e8000001
(XEN) rdx: 0000000000000aa2   rsi: 000000015448d000   rdi:
ffff8300001a0080
(XEN) rbp: ffff8300001a0080   rsp: ffff83000023fc98   r8:
00000000001aed0d
(XEN) r9:  0000000000000000   r10: 0000000000000000   r11:
0000000000000000
(XEN) r12: ffff828404350a08   r13: 0000000000000001   r14:
00000000c0881008
(XEN) r15: 0000000004350a08   cr0: 000000008005003b   cr3:
0000000080b7f000
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: e010
(XEN) Xen stack trace from rsp=ffff83000023fc98:
(XEN)    0000000000000000 ffff830000134533 00000001aed0d663
0000000000000000
(XEN)    ffff8300001a0080 ffff8300001ad080 00000000c0881008
0000000000000000
(XEN)    0000000000000000 ffff83000013f60b ffff83000023ff28
00000000c0881008
(XEN)    ffff818000603020 0000000000000003 00000000002234f3
0000000000223444
(XEN)    0000000000001e00 ffff8300001ad080 00000001001ad080
0000000080000000
(XEN)    0000000000000000 0000000080000000 0000000000000000
0000000000000000
(XEN)    0000000000000000 0000000000000000 00000000ba63eb40
00000000c05d3740
(XEN)    0000000000000000 0000000000000000 0000000000000000
0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000001000
0000000000000000
(XEN)    00000000c05d3748 000000f000000002 0000000000100000
0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000
0000000000000000
(XEN)    0000000000000000 ffff8300001a0080 00000000001531de
ffff83000013cdff
(XEN)    00000000c0881008 ffff8301531de178 ffff8301aec8e020
0000000000000000
(XEN)    0000000000000000 00000000001531de 00000000001aec8e
00000000001aed0d
(XEN)    ffff8300001d7744 ffff83000015bd60 ffff830000000004
ffff8300001a0080
(XEN)    00000001aed0d067 0000000000003000 ffff83000011d1d4
ffff83000023ff28
(XEN)    ffff8300001906c0 ffff8300001a0080 00000000c0881008
0000000000000000
(XEN)    0000000000000000 ffff83000015d003 ffff8300001baf80
0000000000000000
(XEN)    ffff8300001a0080 ffff830000156e38 ffff830000115545
ffff8300001ad080
(XEN)    0000000000000000 00000000ba63eb48 0000000000000000
0000000000000000
(XEN) Xen call trace:
(XEN)    [<ffff83000013c267>]
sh2_guess_wrmap__shadow_3_guest_3+0x67/0xd0
(XEN)    [<ffff830000134533>] shadow2_remove_write_access+0x173/0x2f0
(XEN)    [<ffff83000013f60b>]
sh2_page_fault__shadow_3_guest_3+0x33b/0x10c0
(XEN)    [<ffff83000013cdff>]
sh2_update_cr3__shadow_3_guest_3+0x39f/0x4a0
(XEN)    [<ffff83000015bd60>] vmx_cr_access+0x680/0xfa0
(XEN)    [<ffff83000011d1d4>] context_switch+0x624/0x640
(XEN)    [<ffff83000015d003>] vmx_vmexit_handler+0x4b3/0xe20
(XEN)    [<ffff830000156e38>] vmx_intr_assist+0xf8/0x400
(XEN)    [<ffff830000115545>] do_softirq+0x55/0x70
(XEN)    [<ffff83000015d995>] vmx_asm_vmexit_handler+0x25/0x30
(XEN)    
(XEN) Pagetable walk from ffff8140c0005510:
(XEN)  L4[0x102] = 0000000080b7f063 000000000000011b
(XEN)   L3[0x103] = 0000000080b7f063 000000000000011b
(XEN)    L2[0x0] = 0000000080b7e063 000000000000011c 
(XEN)     L1[0x5] = 0000000000000000 ffffffffffffffff
(XEN) 
(XEN) ****************************************
(XEN) Panic on CPU 3:
(XEN) CPU3 FATAL PAGE FAULT
(XEN) [error_code=0000]

I''m not familiar with new shadow yet, can anyone help to take a look?
-Xin

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>>So our HVM e820 table should also comply to this, and the 256M RAM that
>>falls into that range should be moved to 4G above, that''s the
"real
>>changes" in my patch.
>
>Thanks, I saw it''s checked in as changeset 11186.
>
>But it can not work with shadow2, when I assigned 4096M memory, I got:
>
>(XEN) ----[ Xen-3.0-unstable    Not tainted ]----
>(XEN) CPU:    3
>(XEN) RIP:    e010:[<ffff83000013c267>]
>sh2_guess_wrmap__shadow_3_guest_3+0x67/0xd0
>(XEN) RFLAGS: 0000000000010202   CONTEXT: hypervisor
>(XEN) rax: ffff8140c0000000   rbx: 00000000001aed0d   rcx:
>00000000e8000001
>(XEN) rdx: 0000000000000aa2   rsi: 000000015448d000   rdi:
>ffff8300001a0080
>(XEN) rbp: ffff8300001a0080   rsp: ffff83000023fc98   r8:
>00000000001aed0d
>(XEN) r9:  0000000000000000   r10: 0000000000000000   r11:
>0000000000000000
>(XEN) r12: ffff828404350a08   r13: 0000000000000001   r14:
>00000000c0881008
>(XEN) r15: 0000000004350a08   cr0: 000000008005003b   cr3:
>0000000080b7f000
>(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: e010
Interesting - the crash is in the [optional] ''guess_wrmap''
function
which is trying to fast path the lookup of a particular va.  Can you 
try temporarily disabling this optimization and retrying? Just edit 
xen/include/asm-x86/shadow2.c to replace 

#define SHADOW2_OPTIMZATIONS 0x03 

with 

#define SHADOW2_OPTIMZATIONS 0x02 


and recompile. 

It''d also be good to see if you can reproduce the above error on a 
guest with < 4G memory. 


I''ll try to take a more detailed look myself when I get a test box 
sorted out. 


cheers,


S.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>Interesting - the crash is in the [optional] ''guess_wrmap''
function
>which is trying to fast path the lookup of a particular va.  Can you 
>try temporarily disabling this optimization and retrying? Just edit 
>xen/include/asm-x86/shadow2.c to replace 
>
>#define SHADOW2_OPTIMZATIONS 0x03 
>
>with 
>
>#define SHADOW2_OPTIMZATIONS 0x02 
>
Your suggestion works for me, and windows 2003 server can boot with 5G
memory on 64bit xen.
But after I played with it for some time, qemu dm window disappeared,
and xen console reported:

(XEN) sh2 error: sh2_remove_shadows(): can''t find all shadows of mfn
1b2ddd (shadow2_flags=00000100)
(XEN) domain_crash called from shadow2-common.c:2251
(XEN) Domain 8 (vcpu#0) crashed on cpu#3:
(XEN) ----[ Xen-3.0-unstable    Not tainted ]----
(XEN) CPU:    3
(XEN) RIP:    0008:[<0000000080862ca2>]
(XEN) RFLAGS: 0000000000010256   CONTEXT: hvm
(XEN) rax: 0000000067e97001   rbx: 0000000000000000   rcx:
0000000000000010
(XEN) rdx: 00000000ba482b68   rsi: 0000000000000000   rdi:
00000000f7aa82a0
(XEN) rbp: 00000000ba482b94   rsp: 00000000ba482b54   r8:
0000000000000000
(XEN) r9:  0000000000000000   r10: 0000000000000000   r11:
0000000000000000
(XEN) r12: 0000000000000000   r13: 0000000000000000   r14:
0000000000000000
(XEN) r15: 0000000000000000   cr0: 0000000080010031   cr3:
00000000001a17a0
(XEN) ds: 0023   es: 0023   fs: 0030   gs: 0000   ss: 0010   cs: 0008
(XEN) sh2_update_paging_modes: postponing determination of shadow2 mode
(XEN) vmx_do_launch(): GUEST_CR3<=001a17a0, HOST_CR3<=222b8b000

>
>and recompile. 
>
>It''d also be good to see if you can reproduce the above error on a 
>guest with < 4G memory. 
>
With less than 4G RAM, I didn''t see this error.

Thanks
-Xin

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>
>>Interesting - the crash is in the [optional]
''guess_wrmap'' function
>>which is trying to fast path the lookup of a particular va.  Can you 
>>try temporarily disabling this optimization and retrying? Just edit 
>>xen/include/asm-x86/shadow2.c to replace 
>>
>>#define SHADOW2_OPTIMZATIONS 0x03 
>>
>>with 
>>
>>#define SHADOW2_OPTIMZATIONS 0x02 
>>
>
>Your suggestion works for me, and windows 2003 server can boot with 5G
>memory on 64bit xen.
>But after I played with it for some time, qemu dm window disappeared,
>and xen console reported:
>
>(XEN) sh2 error: sh2_remove_shadows(): can''t find all shadows of
mfn
>1b2ddd (shadow2_flags=00000100)
>(XEN) domain_crash called from shadow2-common.c:2251
>(XEN) Domain 8 (vcpu#0) crashed on cpu#3:
>(XEN) ----[ Xen-3.0-unstable    Not tainted ]----
>(XEN) CPU:    3
>(XEN) RIP:    0008:[<0000000080862ca2>]
>(XEN) RFLAGS: 0000000000010256   CONTEXT: hvm
>(XEN) rax: 0000000067e97001   rbx: 0000000000000000   rcx:
>0000000000000010
>(XEN) rdx: 00000000ba482b68   rsi: 0000000000000000   rdi:
>00000000f7aa82a0
>(XEN) rbp: 00000000ba482b94   rsp: 00000000ba482b54   r8:
>0000000000000000
>(XEN) r9:  0000000000000000   r10: 0000000000000000   r11:
>0000000000000000
>(XEN) r12: 0000000000000000   r13: 0000000000000000   r14:
>0000000000000000
>(XEN) r15: 0000000000000000   cr0: 0000000080010031   cr3:
>00000000001a17a0
>(XEN) ds: 0023   es: 0023   fs: 0030   gs: 0000   ss: 0010   cs: 0008
>(XEN) sh2_update_paging_modes: postponing determination of shadow2 mode
>(XEN) vmx_do_launch(): GUEST_CR3<=001a17a0, HOST_CR3<=222b8b000
Hi Steven, I still saw this bug, I think it should be fixed for xen
3.0.3.
thanks
-Xin


>With less than 4G RAM, I didn''t see this error.
>
>Thanks
>-Xin
>
>_______________________________________________
>Xen-devel mailing list
>Xen-devel@lists.xensource.com
>http://lists.xensource.com/xen-devel
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel