Xen devel - May 2013 - Memory reservation for 32bit guests on high RAM systems

We''ve been looking in to support of 32bit domains on machines with
large amounts of RAM. Working from the ballooning and reservation support added
a few years ago:

http://lists.xen.org/archives/html/xen-devel/2009-11/msg01277.html

32 bit domains eventually fail to boot with the following Xen complaint:

xm create: Error: (4, ''Out of memory'', "panic:
xc_dom_boot.c:159: xc_dom_boot_mem_init: can''t allocate low memory for
domain")

Booting with dom0_mem=128G, with enable-dom0-ballooning=yes and
total_available_memory=128 set in xend''s configuration works as
expected.  128GB of RAM is avalable 32bit guests and 128GB for 64 bit guests. 
I''m able to account for the RAM in the box by totaling 32bit domains +
64bit domains + free_memory as reported by xm info + memory remaining in the
dom0 balloon.

However this config doesn''t allow use of all of the RAM available in
the physical machine, and increasing dom0_mem beyond 128G decreases the amount
of 32bit domains I am able to get to boot!  It appears that for every 1G over
128G for dom0_mem reduces the ability to boot 32bit domains by 512M.

The balloon appears to be starting too low in RAM. The output of "xm
debug-key h" indicates that a ''D'' will dump the p2m table
which sounds like it would be really useful in determining where the various
domains are actually mapped in physical RAM, however "xm debug-key D"
doesn''t produce any output on the console or xm dmesg.

Settings and RAM-worth of different bitness guests that were bootable on a box
with 384G of RAM:

dom0_mem         total_available_memory    32bit     64bit
min:3G,max:128G           128               125G      127G
-128G                     128                65G      128G
min:3G,max:-128G          240                65G      239G
min:3G,max:160G           150               115G      149G

We''re either missing something fundamental about how to configure this,
or there''s a bug in ballooning or reservation that is preventing us
from using the full 128G towards 32bit domains while being able to use all of
the remaining RAM for 64 bit guests.

Anyone have any insight on using ballooning and memory reservations in machines
with more than 256GB of RAM? Alternaively any information on getting a dump of
p2m map or otherwise getting a clear picture of where in physical RAM each
domain resides would be helpful in troubleshooting this further.

Regards,
Peter Sandin