Hello, I''ve recently ran into a problem on a physical machine with a large amount of memory where 32-bit PV guests would refuse to boot with this error message: xm create: Error: (4, ''Out of memory'', "panic: xc_dom_boot.c:159: xc_dom_boot_mem_init: can''t allocate low memory for domain") We discovered that there''s a boundary where 32-bit PV guests can''t be allocated memory above 168GB, along with the work-around config option to reserve memory for these guests (total_available_memory in xend-config.sxp). Could someone clarify why this is a restriction? Is this some sort of hardware limitation? Thanks, Nick
>>> On 15.04.13 at 16:21, Nick Pegg <npegg@linode.com> wrote: > I''ve recently ran into a problem on a physical machine with a large > amount of memory where 32-bit PV guests would refuse to boot with this > error message: > > xm create: Error: (4, ''Out of memory'', "panic: xc_dom_boot.c:159: > xc_dom_boot_mem_init: can''t allocate low memory for domain") > > We discovered that there''s a boundary where 32-bit PV guests can''t be > allocated memory above 168GB, along with the work-around config option > to reserve memory for these guests (total_available_memory in > xend-config.sxp). > > Could someone clarify why this is a restriction? Is this some sort of > hardware limitation?No, this is an interface limitation: 32-bit PV guests can''t be expected to be able to deal with an address space hole of more than 168Mb at the top of the address space, yet a bigger hole would be needed to make the full (read-only) M2P table for the guest. If you wanted to lift that limit, you''d have to make the guest capable of falling back to using hypercalls for doing the M2P lookups for MFNs outside the visible range - no one cared enough about this to implement something like that. Jan
On 15/04/13 15:21, Nick Pegg wrote:> Hello, > > I''ve recently ran into a problem on a physical machine with a large > amount of memory where 32-bit PV guests would refuse to boot with this > error message: > > xm create: Error: (4, ''Out of memory'', "panic: xc_dom_boot.c:159: > xc_dom_boot_mem_init: can''t allocate low memory for domain") > > We discovered that there''s a boundary where 32-bit PV guests can''t be > allocated memory above 168GB, along with the work-around config option > to reserve memory for these guests (total_available_memory in > xend-config.sxp). > > Could someone clarify why this is a restriction? Is this some sort of > hardware limitation? > > > Thanks, > NickThis is because of the amount of the Xen m2p table mapped into 32bit kernel space, between 0xf5800000 and 0xffffffff. PV guests require the m2p table to perform paravirtualised memory management. This is a fundamental limit of 32bit PV guests. ~Andrew> > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel
On Mon, 2013-04-15 at 15:21 +0100, Nick Pegg wrote:> Hello, > > I''ve recently ran into a problem on a physical machine with a large > amount of memory where 32-bit PV guests would refuse to boot with this > error message: > > xm create: Error: (4, ''Out of memory'', "panic: xc_dom_boot.c:159: > xc_dom_boot_mem_init: can''t allocate low memory for domain") > > We discovered that there''s a boundary where 32-bit PV guests can''t be > allocated memory above 168GB, along with the work-around config option > to reserve memory for these guests (total_available_memory in > xend-config.sxp). > > Could someone clarify why this is a restriction? Is this some sort of > hardware limitation?It''s a 32-bit PV guest limitation which arises due to the need to map the m2p table into each PV guest, there is simply only enough room for 168GB worth of m2p in the 32-bit kernel address space before it starts to impinge on other stuff. In theory the host admin could be given the flexibility to make it a little bigger (not sure if any toolstack implements this), but not by much. There are various hacky things which could in theory be done, such as giving 32-bit guests views onto different 168GB windows of physical memory from which all their RAM must come, but none of those have been implemented AFAIK. Hopefully PVH will eventually support 32-bit and this restriction will go away. Ian.