beth kon
2007-Nov-20 19:00 UTC
[Xen-devel] Running out of Xen heap space with large memory
Hi. I have been debugging a hang when booting Xen with 512G memory. I am testing on an x3950, 8 nodes, 128 way, with xen 3.0.3. In order to get past the 166G limit, IETH added the no-pv-compat flag to the xen boot line. The boots have been hanging when memory is greater than about 432G. I found that, as memory size increases, the size of the xen heap decreases, because additional memory is being used for the memory housekeeping (seen in init_boot_allocator). Eventually (around 432G) the xen heap is reported as 0MB: (XEN) Command line: /xen.gz-2.6.18-53.el5 numa=on dom0_mem=512m com2=19200,8n1 console=com2 no-pv-compat (XEN) 0000000000000000 - 0000000000098000 (usable) (XEN) 0000000000098c00 - 00000000000a0000 (reserved) (XEN) 00000000000e0000 - 0000000000100000 (reserved) (XEN) 0000000000100000 - 000000007fea6000 (usable) (XEN) 000000007fea64c0 - 000000007fef6380 (ACPI data) (XEN) 000000007fef6380 - 0000000090000000 (reserved) (XEN) 00000000fec00000 - 0000000100000000 (reserved) (XEN) 0000000100000000 - 0000007e00000000 (usable) (XEN) System RAM: 514046MB (526383352kB) (XEN) ACPI: [SRAT:0x00] ignored 32 entries of 64 found (XEN) BETH xenheap_phys_start = 18677760, xenheap_phys_end=16777216 (XEN) BETH s = 4294967296, e=16777216 (XEN) Xen heap: 0MB (0kB) (XEN) Cannot handle page request order 2! (XEN) Cannot handle page request order 0! (XEN) Unknown interrupt The boot hangs at this point (no surprise). Note that the "BETH" debug statements show xenheap_phys_start>xenheap_phys_end. I realize that interest in this may be low since the no-pv-compat flag has been removed in xen-unstable. The resulting 166G limit was discussed here: http://lists.xensource.com/archives/html/xen-devel/2007-08/msg00493.html If pages being passed among domains is the issue, wouldn''t the no-pv-compat flag address that by not allowing any 32 bit guests on the machine? I assume page stealing is restricted to domains running on the same hypervisor, right? Why was the no-pv-compat flag removed? And as Raj discussed: http://lists.xensource.com/archives/html/xen-devel/2007-10/msg00550.html Even if Raj provides these changes, I believe the above issue with running out of heap space would still exist. I see that init_boot_allocator is unchanged between 3.0.3 and unstable. Any suggestions on how this issue could be corrected? I will continue to look at this code, but any suggestions from people who are more familiar with it would be greatly appreciated. -- Elizabeth Kon (Beth) IBM Linux Technology Center Open Hypervisor Team email: eak@us.ibm.com _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2007-Nov-20 19:32 UTC
Re: [Xen-devel] Running out of Xen heap space with large memory
The size of the Xen heap is adjustable. Try ''xenheap_megabytes=64'' on the Xen command line. We plan to get rid of this static partitioning between Xen heap and domain heap entirely. -- Keir On 20/11/07 19:00, "beth kon" <eak@us.ibm.com> wrote:> Hi. I have been debugging a hang when booting Xen with 512G memory. I am > testing on an x3950, 8 nodes, 128 way, with xen 3.0.3. In order to get > past the 166G limit, IETH added the no-pv-compat flag to the xen boot line. > > The boots have been hanging when memory is greater than about 432G. I > found that, as memory size increases, the size of the xen heap > decreases, because additional memory is being used for the memory > housekeeping (seen in init_boot_allocator). Eventually (around 432G) the > xen heap is reported as 0MB: > > (XEN) Command line: /xen.gz-2.6.18-53.el5 numa=on dom0_mem=512m com2=19200,8n1 > console=com2 no-pv-compat > (XEN) 0000000000000000 - 0000000000098000 (usable) > (XEN) 0000000000098c00 - 00000000000a0000 (reserved) > (XEN) 00000000000e0000 - 0000000000100000 (reserved) > (XEN) 0000000000100000 - 000000007fea6000 (usable) > (XEN) 000000007fea64c0 - 000000007fef6380 (ACPI data) > (XEN) 000000007fef6380 - 0000000090000000 (reserved) > (XEN) 00000000fec00000 - 0000000100000000 (reserved) > (XEN) 0000000100000000 - 0000007e00000000 (usable) > (XEN) System RAM: 514046MB (526383352kB) > (XEN) ACPI: [SRAT:0x00] ignored 32 entries of 64 found > (XEN) BETH xenheap_phys_start = 18677760, xenheap_phys_end=16777216 > (XEN) BETH s = 4294967296, e=16777216 > (XEN) Xen heap: 0MB (0kB) > (XEN) Cannot handle page request order 2! > (XEN) Cannot handle page request order 0! > (XEN) Unknown interrupt > > The boot hangs at this point (no surprise). Note that the "BETH" debug > statements show xenheap_phys_start>xenheap_phys_end. > > I realize that interest in this may be low since the no-pv-compat flag > has been removed in xen-unstable. The resulting 166G limit was discussed > here: > http://lists.xensource.com/archives/html/xen-devel/2007-08/msg00493.html > > If pages being passed among domains is the issue, wouldn''t the > no-pv-compat flag address that by not allowing any 32 bit guests on the > machine? I assume page stealing is restricted to domains running on the > same hypervisor, right? Why was the no-pv-compat flag removed? > > And as Raj discussed: > http://lists.xensource.com/archives/html/xen-devel/2007-10/msg00550.html > Even if Raj provides these changes, I believe the above issue with > running out of heap space would still exist. I see that > init_boot_allocator is unchanged between 3.0.3 and unstable. Any > suggestions on how this issue could be corrected? > > I will continue to look at this code, but any suggestions from people > who are more familiar with it would be greatly appreciated._______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel