Xen devel - Oct 2007 - Problem with nr_nodes on large memory NUMA machine

We''ve run into an issue with an 8 node x3950 where xm info is showing 
only 6 nodes. I''ve traced the problem to the clip_to_limit function in 
arch/x86/e820.c.

#ifdef __x86_64__
   clip_to_limit((uint64_t)(MACH2PHYS_COMPAT_VIRT_END -
                            __HYPERVISOR_COMPAT_VIRT_START) << 10,
                 "Only the first %u GB of the physical memory map "
                 "can be accessed by 32-on-64 guests.");
#endif

Boot messages....
(XEN) WARNING: Only the first 166 GB of the physical memory map can be 
accessed by 32-on-64 guests. (XEN) Truncating memory map to 174063616kB

After the memory is clipped, acpi_scan_nodes runs cutoff_node, which 
limits the memory associated with each node according to the cutoff 
values. Then, acpi_scan_nodes calls unparse_node to "remove" nodes
that
don''t have the minimum amount of memory, due to the clipping of the 
memory range.

Can someone explain what this is all about and why it might be necessary?

-- 
Elizabeth Kon (Beth)
IBM Linux Technology Center
Open Hypervisor Team
email: eak@us.ibm.com


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

32-bit guests on 64-bit hypervisor can only address the bottom 166GB of the
memory map, due to constraints on how much of the machine-to-phys array they
can reasonably map into their limited address space.

 -- Keir

On 19/10/07 16:02, "beth kon" <eak@us.ibm.com> wrote:
> We''ve run into an issue with an 8 node x3950 where xm info is
showing
> only 6 nodes. I''ve traced the problem to the clip_to_limit
function in
> arch/x86/e820.c.
> 
> #ifdef __x86_64__
>    clip_to_limit((uint64_t)(MACH2PHYS_COMPAT_VIRT_END -
>                             __HYPERVISOR_COMPAT_VIRT_START) << 10,
>                  "Only the first %u GB of the physical memory map
"
>                  "can be accessed by 32-on-64 guests.");
> #endif
> 
> Boot messages....
> (XEN) WARNING: Only the first 166 GB of the physical memory map can be
> accessed by 32-on-64 guests. (XEN) Truncating memory map to 174063616kB
> 
> After the memory is clipped, acpi_scan_nodes runs cutoff_node, which
> limits the memory associated with each node according to the cutoff
> values. Then, acpi_scan_nodes calls unparse_node to "remove"
nodes that
> don''t have the minimum amount of memory, due to the clipping of
the
> memory range.
> 
> Can someone explain what this is all about and why it might be necessary?


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Beth,
We saw this issue on our boxes too. 
http://lists.xensource.com/archives/html/xen-devel/2007-08/msg00479.html
I am trying to figure out how to write the copy-to-low-memory path. 
Keir, could you give me some suggestions?
Thanks
Raj
>32-bit guests on 64-bit hypervisor can only address the bottom 
>166GB of the memory map, due to constraints on how much of the 
>machine-to-phys array they can reasonably map into their 
>limited address space.
>
> -- Keir
>
>On 19/10/07 16:02, "beth kon" <eak@us.ibm.com> wrote:
>
>> We''ve run into an issue with an 8 node x3950 where xm info 
>is showing 
>> only 6 nodes. I''ve traced the problem to the clip_to_limit 
>function in 
>> arch/x86/e820.c.
>> 
>> #ifdef __x86_64__
>>    clip_to_limit((uint64_t)(MACH2PHYS_COMPAT_VIRT_END -
>>                             __HYPERVISOR_COMPAT_VIRT_START) <<
10,
>>                  "Only the first %u GB of the physical memory map
"
>>                  "can be accessed by 32-on-64 guests.");
#endif
>> 
>> Boot messages....
>> (XEN) WARNING: Only the first 166 GB of the physical memory 
>map can be 
>> accessed by 32-on-64 guests. (XEN) Truncating memory map to 
>> 174063616kB
>> 
>> After the memory is clipped, acpi_scan_nodes runs cutoff_node, which 
>> limits the memory associated with each node according to the cutoff 
>> values. Then, acpi_scan_nodes calls unparse_node to "remove"
nodes
>> that don''t have the minimum amount of memory, due to the
clipping of
>> the memory range.
>> 
>> Can someone explain what this is all about and why it might 
>be necessary?
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 19/10/07 16:21, "Subrahmanian, Raj"
<raj.subrahmanian@unisys.com> wrote:
> Beth,
> We saw this issue on our boxes too.
> http://lists.xensource.com/archives/html/xen-devel/2007-08/msg00479.html
> I am trying to figure out how to write the copy-to-low-memory path.
> Keir, could you give me some suggestions?
In gnttab_transfer(), if the foreign domain (e) is 32-on-64 and the page
being stolen from the local domain (d) is above 166GB then allocate anothr
domheap page for e, copy the stolen page contents to it. Then free the
stolen page and the new page takes its place.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel