Xen devel - Feb 2007 - [PATCH]: Allow Xen to boot/run on large memory (>64G) machines

All,
     I''ve been tracking down a problem where dom0 refuses to boot on
very large
memory x86_64 machines.  Here''s what happens:

The hypervisor starts up with 1GB in the DMA zone.  Two large allocations come
out of the DMA zone; the frame table (in init_frametable()), and the memory for
dom0 (in construct_dom0()).  With a lot of memory in the box, most of the DMA
zone gets allocated during init_frametable; so much so, in fact, that there is
no room to make the allocation in construct_dom0, and the dom0 fails to boot
with:

(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) Not enough RAM for domain 0 allocation.
(XEN) ****************************************
(XEN)
(XEN) Reboot in five seconds...

The solution (suggested by Keir), is to make the frametable allocated out of
high memory instead of the DMA zone.  The attached patch (against 3.0.3, but the
problem is the same in unstable), does this.  I tested this out on a 96GB
machine; without the patch, the machine would reboot as described above; with
the patch, I was able to boot dom0 and create a PV guest with 92GB of memory.
     I only compile tested this on ia64, but I don''t see anything in it
that
should cause problems there.
     Note that this is not the end of the story, however.  For even larger
machines, it can *still* be the case that the allocation in construct_dom0()
fails; in particular, if the order goes above 17, it will fail in the same way.
 One way to fix it would be to just allocate that memory out of the normal zone
for x86_64, as well; however, I''m not sure if this will break anything
else.
Any comments?

Signed-off-by: Chris Lalancette <clalance@redhat.com>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 22/2/07 00:38, "Chris Lalancette" <clalance@redhat.com>
wrote:
>      Note that this is not the end of the story, however.  For even larger
> machines, it can *still* be the case that the allocation in
construct_dom0()
> fails; in particular, if the order goes above 17, it will fail in the same
> way.
>  One way to fix it would be to just allocate that memory out of the normal
> zone
> for x86_64, as well; however, I''m not sure if this will break
anything else.
> Any comments?
If there are no users of alloc_boot_pages() expecting low memory to be
returned then we can adjust the implementation of that existing function
rather than introduce a new one.

As for domain_build() there are two considerations: firstly that the
allocation is contiguous and secondly that it is from the DMA pool. The
builder makes simplifying assumptions based on contiguity. The allocation
from DMA pool I think I''ve tried to get rid of before -- I think I was
scuppered by something as simple as the PAE pgdir needing to be allocated
from low memory. I think we can stop allocating from the DMA pool, at least
for non-PAE host.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>>> Keir Fraser <Keir.Fraser@cl.cam.ac.uk> 22.02.07 08:50
>>>
>On 22/2/07 00:38, "Chris Lalancette" <clalance@redhat.com>
wrote:
>
>>      Note that this is not the end of the story, however.  For even
larger
>> machines, it can *still* be the case that the allocation in
construct_dom0()
>> fails; in particular, if the order goes above 17, it will fail in the
same
>> way.
>>  One way to fix it would be to just allocate that memory out of the
normal
>> zone
>> for x86_64, as well; however, I''m not sure if this will break
anything else.
>> Any comments?
>
>If there are no users of alloc_boot_pages() expecting low memory to be
>returned then we can adjust the implementation of that existing function
>rather than introduce a new one.
>
>As for domain_build() there are two considerations: firstly that the
>allocation is contiguous and secondly that it is from the DMA pool. The
>builder makes simplifying assumptions based on contiguity. The allocation
>from DMA pool I think I''ve tried to get rid of before -- I think I
was
>scuppered by something as simple as the PAE pgdir needing to be allocated
>from low memory. I think we can stop allocating from the DMA pool, at least
>for non-PAE host.
The page allocator changes that I posted a while back probably haven''t
been looked at so fat, given the above comments. The patch that kills the
DMA pool changes x86-64 to allocate the dom0 memory without restriction
(i386 has to remain restricted, yet not because of DMA address issues, but
in order to be able to see the memory in the 1:1 mapping).

Likewise, I''m not certain the changes presented here make a lot of
sense
in the context of the elimination of the DMA pool and the resulting desire
to unify xen heap and domain heap on x86-64.

Jan

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>Likewise, I''m not certain the changes presented here make a lot of
sense
>in the context of the elimination of the DMA pool and the resulting desire
>to unify xen heap and domain heap on x86-64.
Sorry, I wrote this based on the description, without having looked at the
patch. The changes certainly make sense (including doing this perhaps also
for 32-bits). You didn''t touch the dom0 allocation at all...

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 22/2/07 10:33, "Jan Beulich" <jbeulich@novell.com> wrote:
> The page allocator changes that I posted a while back probably
haven''t
> been looked at so fat, given the above comments. The patch that kills the
> DMA pool changes x86-64 to allocate the dom0 memory without restriction
> (i386 has to remain restricted, yet not because of DMA address issues, but
> in order to be able to see the memory in the 1:1 mapping).
Oh, good point. :-) And it sounds like it makes sense to leave this alone
for now and leave it to your patches.
> Likewise, I''m not certain the changes presented here make a lot of
sense
> in the context of the elimination of the DMA pool and the resulting desire
> to unify xen heap and domain heap on x86-64.
It makes sense for the boot allocator to prefer to allocate from high memory
if it can, rather then using what is currently the DMA pool (and, after your
patches are applied, will be from relatively-narrow-address-width pools). So
I think this patch is good and narrow enough in scope to go straight in
(although I think the behaviour of alloc_boot_pages() should be changed
rather than adding a new allocator function).

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Keir Fraser wrote:
> On 22/2/07 10:33, "Jan Beulich" <jbeulich@novell.com>
wrote:
> 
> 
>>The page allocator changes that I posted a while back probably
haven''t
>>been looked at so fat, given the above comments. The patch that kills
the
>>DMA pool changes x86-64 to allocate the dom0 memory without restriction
>>(i386 has to remain restricted, yet not because of DMA address issues,
but
>>in order to be able to see the memory in the 1:1 mapping).
> 
> 
> Oh, good point. :-) And it sounds like it makes sense to leave this alone
> for now and leave it to your patches.
> 
Ah, OK.  That will address the second concern.  Jan''s right, I
didn''t actually
look at those patches.  Thanks for pointing them out.
> 
> It makes sense for the boot allocator to prefer to allocate from high
memory
> if it can, rather then using what is currently the DMA pool (and, after
your
> patches are applied, will be from relatively-narrow-address-width pools).
So
> I think this patch is good and narrow enough in scope to go straight in
> (although I think the behaviour of alloc_boot_pages() should be changed
> rather than adding a new allocator function).
Yeah, I wasn''t quite sure how far to go with this.  The frame table was
the
worst offender, so I just went after that.  I can whip up a quick patch and test
it out here, changing the alloc_boot_pages() to always allocate from the top.

By the way, I assume we only want to do this for x86_64, yes?

Chris Lalancette

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 22/2/07 14:57, "Chris Lalancette" <clalance@redhat.com>
wrote:
>> It makes sense for the boot allocator to prefer to allocate from high
memory
>> if it can, rather then using what is currently the DMA pool (and, after
your
>> patches are applied, will be from relatively-narrow-address-width
pools). So
>> I think this patch is good and narrow enough in scope to go straight in
>> (although I think the behaviour of alloc_boot_pages() should be changed
>> rather than adding a new allocator function).
> 
> Yeah, I wasn''t quite sure how far to go with this.  The frame
table was the
> worst offender, so I just went after that.  I can whip up a quick patch and
> test
> it out here, changing the alloc_boot_pages() to always allocate from the
top.
Turns out there is one place that wants to allocate from the bottom (the
kdump path in arch/x86/setup.c). So I renamed alloc_boot_pages() to
alloc_lowmem_boot_pages() and called the new function alloc_boot_pages(). If
you care you use the more specific (lowmem) one. I''m about to check in
the
revised patch.
> By the way, I assume we only want to do this for x86_64, yes?
No, it makes sense to conserve ''narrower'' addresses.

 -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel