Krysan, Susan
2007-Aug-22 22:33 UTC
[Xen-devel] xen does not see more than 173800500k of memory
Our test server (Unisys ES7000) has recently been upgraded with more
processors and more memory, so I am testing x86_64 xen with 64 cpus and
192gb. I compiled xen with max_phys_cpus=64 and debug=y. xen sees the
64 cpus, but only sees 173800500k of the 192gb. I booted dom0 with
2048M and numa=on. I get the same result when I configure the server to
have 176gb (xen still sees only 173800500k). Booting without numa=on
makes no difference.
xentop - 14:48:42 Xen 3.0-unstable
1 domains: 1 running, 0 blocked, 0 paused, 0 crashed, 0 dying, 0
shutdown
Mem: 173800500k total, 4324748k used, 169475752k free CPUs: 64 @
3400MHz
NAME STATE CPU(sec) CPU(%) MEM(k) MEM(%) MAXMEM(k)
MAXMEM(%) VCPUS
NETS NETTX(k) NETRX(k) VBDS VBD_OO VBD_RD VBD_WR SSID
Domain-0 -----r 95 0.0 2097152 1.2 no limit
n/a 32
0 0 0 0 0 0 0 0
When I configure the server with 160gb or less, xen sees all of the
memory.
Also, when I boot with the SLES kernel, SLES can see the all of the
memory.
I assume there is not a memory limit on xen. I would greatly appreciate
any help to resolve this problem.
Thanks,
Sue Krysan
Linux Systems Group
Unisys Corporation
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Keir Fraser
2007-Aug-22 22:38 UTC
Re: [Xen-devel] xen does not see more than 173800500k of memory
xm dmesg¹ output? -- Keir On 22/8/07 23:33, "Krysan, Susan" <KRYSANS@unisys.com> wrote:> > When I configure the server with 160gb or less, xen sees all of the memory. > > Also, when I boot with the SLES kernel, SLES can see the all of the memory. > > I assume there is not a memory limit on xen. I would greatly appreciate any > help to resolve this problem._______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jan Beulich
2007-Aug-23 07:34 UTC
Re: [Xen-devel] xen does not see more than 173800500k of memory
This seems to correspond to the truncation at 166Gb for 32on64 support (see xen/arch/x86/e820.c:machine_specific_memory_setup()). There used to be a command line options to disallow creation of compat mode guests (thereby allowing all memory to be used), but that option went away with fairly recent changes. Jan>>> "Krysan, Susan" <KRYSANS@unisys.com> 23.08.07 00:33 >>>Our test server (Unisys ES7000) has recently been upgraded with more processors and more memory, so I am testing x86_64 xen with 64 cpus and 192gb. I compiled xen with max_phys_cpus=64 and debug=y. xen sees the 64 cpus, but only sees 173800500k of the 192gb. I booted dom0 with 2048M and numa=on. I get the same result when I configure the server to have 176gb (xen still sees only 173800500k). Booting without numa=on makes no difference. xentop - 14:48:42 Xen 3.0-unstable 1 domains: 1 running, 0 blocked, 0 paused, 0 crashed, 0 dying, 0 shutdown Mem: 173800500k total, 4324748k used, 169475752k free CPUs: 64 @ 3400MHz NAME STATE CPU(sec) CPU(%) MEM(k) MEM(%) MAXMEM(k) MAXMEM(%) VCPUS NETS NETTX(k) NETRX(k) VBDS VBD_OO VBD_RD VBD_WR SSID Domain-0 -----r 95 0.0 2097152 1.2 no limit n/a 32 0 0 0 0 0 0 0 0 When I configure the server with 160gb or less, xen sees all of the memory. Also, when I boot with the SLES kernel, SLES can see the all of the memory. I assume there is not a memory limit on xen. I would greatly appreciate any help to resolve this problem. Thanks, Sue Krysan Linux Systems Group Unisys Corporation _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2007-Aug-23 14:27 UTC
Re: [Xen-devel] xen does not see more than 173800500k of memory
This should be easily fixed by properly applying domain_clamp_alloc_bitsize() in __alloc_domheap_pages(). Why is it only applied when the bitsize is explicitly specified by the caller? I think that''s the only thing to fix to allow the 166GB boot-time restriction to be lifted, but am I missing something, Jan? -- Keir On 23/8/07 08:34, "Jan Beulich" <jbeulich@novell.com> wrote:> This seems to correspond to the truncation at 166Gb for 32on64 support (see > xen/arch/x86/e820.c:machine_specific_memory_setup()). There used to be > a command line options to disallow creation of compat mode guests (thereby > allowing all memory to be used), but that option went away with fairly recent > changes. > > Jan > >>>> "Krysan, Susan" <KRYSANS@unisys.com> 23.08.07 00:33 >>> > Our test server (Unisys ES7000) has recently been upgraded with more > processors and more memory, so I am testing x86_64 xen with 64 cpus and > 192gb. I compiled xen with max_phys_cpus=64 and debug=y. xen sees the > 64 cpus, but only sees 173800500k of the 192gb. I booted dom0 with > 2048M and numa=on. I get the same result when I configure the server to > have 176gb (xen still sees only 173800500k). Booting without numa=on > makes no difference. > > > > xentop - 14:48:42 Xen 3.0-unstable > > 1 domains: 1 running, 0 blocked, 0 paused, 0 crashed, 0 dying, 0 > shutdown > > Mem: 173800500k total, 4324748k used, 169475752k free CPUs: 64 @ > 3400MHz > > NAME STATE CPU(sec) CPU(%) MEM(k) MEM(%) MAXMEM(k) > MAXMEM(%) VCPUS > > NETS NETTX(k) NETRX(k) VBDS VBD_OO VBD_RD VBD_WR SSID > > Domain-0 -----r 95 0.0 2097152 1.2 no limit > n/a 32 > > 0 0 0 0 0 0 0 0 > > > > When I configure the server with 160gb or less, xen sees all of the > memory. > > > > Also, when I boot with the SLES kernel, SLES can see the all of the > memory. > > > > I assume there is not a memory limit on xen. I would greatly appreciate > any help to resolve this problem. > > > > Thanks, > > Sue Krysan > > Linux Systems Group > > Unisys Corporation > > > > > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jan Beulich
2007-Aug-23 14:51 UTC
Re: [Xen-devel] xen does not see more than 173800500k of memory
>>> Keir Fraser <keir@xensource.com> 23.08.07 16:27 >>> >This should be easily fixed by properly applying >domain_clamp_alloc_bitsize() in __alloc_domheap_pages(). Why is it only >applied when the bitsize is explicitly specified by the caller? > >I think that''s the only thing to fix to allow the 166GB boot-time >restriction to be lifted, but am I missing something, Jan?We had this discussion before - the problem is not restricting the allocations a domain does, but pages getting passed to it from other domains, which (if they happen to lie outside the 166Gb range) the domain then can''t control. And yes, you said page flipping is basically dead, but this isn''t being enforced (and probably can''t as long as you want to support older guests potentially using it). Jan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2007-Aug-23 15:53 UTC
Re: [Xen-devel] xen does not see more than 173800500k of memory
It''s not a Xen security risk though. If you happen to use a compat guest with page flipping then it just won''t work. I think it''s fair to say at this point that that is just ''too bad''. If anyone really cares then they will need to add a copy-to-low-memory path in Xen''s page transfer code. The 166GB restriction has to go. -- Keir On 23/8/07 15:51, "Jan Beulich" <jbeulich@novell.com> wrote:>>>> Keir Fraser <keir@xensource.com> 23.08.07 16:27 >>> >> This should be easily fixed by properly applying >> domain_clamp_alloc_bitsize() in __alloc_domheap_pages(). Why is it only >> applied when the bitsize is explicitly specified by the caller? >> >> I think that''s the only thing to fix to allow the 166GB boot-time >> restriction to be lifted, but am I missing something, Jan? > > We had this discussion before - the problem is not restricting the allocations > a domain does, but pages getting passed to it from other domains, which (if > they happen to lie outside the 166Gb range) the domain then can''t control. > And yes, you said page flipping is basically dead, but this isn''t being > enforced (and probably can''t as long as you want to support older guests > potentially using it). > > Jan > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Subrahmanian, Raj
2007-Sep-05 16:45 UTC
RE: [Xen-devel] xen does not see more than 173800500k of memory
>Re-sent. The mail I sent out Monday is apparently not on the list.Thanks Raj All,>From looking at the code, it seems that the proximate cause for thislimit is the truncation at 166Gb for 32on64 support in e820.c, and not the domain_clamp_alloc_bitsize. If I understand the issue correctly, if this limit were removed from e820.c, this would work, except for the side-effect on 32-on-64 guests? Is this correct? Also, where should I look to find Xen''s page transfer code? Thanks Raj>It''s not a Xen security risk though. If you happen to use a compat >guest with page flipping then it just won''t work. I think it''s fair to >say at this point that that is just ''too bad''. If anyone really cares >then they will need to add a copy-to-low-memory path in Xen''s page >transfer code. The 166GB restriction has to go. > > -- Keir > >On 23/8/07 15:51, "Jan Beulich" <jbeulich@novell.com> wrote: > >>>>> Keir Fraser <keir@xensource.com> 23.08.07 16:27 >>> >>> This should be easily fixed by properly applying >>> domain_clamp_alloc_bitsize() in __alloc_domheap_pages(). Why is it >>> only applied when the bitsize is explicitly specified by the caller? >>> >>> I think that''s the only thing to fix to allow the 166GB boot-time >>> restriction to be lifted, but am I missing something, Jan? >> >> We had this discussion before - the problem is not restricting the >> allocations a domain does, but pages getting passed to it from other >> domains, which (if they happen to lie outside the 166Gb >range) the domain then can''t control. >> And yes, you said page flipping is basically dead, but this isn''t >> being enforced (and probably can''t as long as you want to support >> older guests potentially using it). >> >> Jan_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel