Vasiliy G Tolstov
2010-Aug-12 10:24 UTC
[Xen-devel] failed to start centos 5 domU with maxmem=30000
Hello. I''m try to start centos domU pvm under sles 11 sp1 xen kernel
with memory=512 and maxmem=30000. Centos doing boot and freeze.
If i''m try to use maxmem=15000 all works.
Can You tell me what''s wrong?
dom0:
xen24:~ # xm info
host : xen24
release : 2.6.32.12-0.7-xen
version : #1 SMP 2010-05-20 11:14:20 +0200
machine : x86_64
nr_cpus : 16
nr_nodes : 2
cores_per_socket : 4
threads_per_core : 2
cpu_mhz : 2266
hw_caps :
bfebfbff:28100800:00000000:00001b40:009ce3bd:00000000:00000001:00000000
virt_caps : hvm hvm_directio
total_memory : 65455
free_memory : 62133
free_cpus : 0
max_free_memory : 62133
max_para_memory : 62129
max_hvm_memory : 61975
node_to_cpu : node0:0-3,8-11
node1:4-7,12-15
node_to_memory : node0:30485
node1:31647
node_to_dma32_mem : node0:2995
node1:0
max_node_id : 1
xen_major : 4
xen_minor : 0
xen_extra : .0_21091_04-0.2
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler : credit
xen_pagesize : 4096
platform_params : virt_start=0xffff800000000000
xen_changeset : 21091
xen_commandline : loglvl=all noreboot dom0_mem=2048M
dom0_max_vcpus=2 dom0_vcpus_pin console=vga vga=current
cc_compiler : gcc version 4.3.4 [gcc-4_3-branch revision
152973] (SUSE Linux)
cc_compile_by : abuild
cc_compile_domain :
cc_compile_date : Thu May 20 11:44:41 UTC 2010
xend_config_format : 4
--
Vasiliy G Tolstov <v.tolstov@selfip.ru>
Selfip.Ru
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Pasi Kärkkäinen
2010-Aug-12 11:37 UTC
Re: [Xen-devel] failed to start centos 5 domU with maxmem=30000
On Thu, Aug 12, 2010 at 02:24:05PM +0400, Vasiliy G Tolstov wrote:> Hello. I''m try to start centos domU pvm under sles 11 sp1 xen kernel > with memory=512 and maxmem=30000. Centos doing boot and freeze. > If i''m try to use maxmem=15000 all works. > > Can You tell me what''s wrong? >Is the centos PV domU 32bit or 64bit ? -- Pasi> dom0: > xen24:~ # xm info > host : xen24 > release : 2.6.32.12-0.7-xen > version : #1 SMP 2010-05-20 11:14:20 +0200 > machine : x86_64 > nr_cpus : 16 > nr_nodes : 2 > cores_per_socket : 4 > threads_per_core : 2 > cpu_mhz : 2266 > hw_caps : > bfebfbff:28100800:00000000:00001b40:009ce3bd:00000000:00000001:00000000 > virt_caps : hvm hvm_directio > total_memory : 65455 > free_memory : 62133 > free_cpus : 0 > max_free_memory : 62133 > max_para_memory : 62129 > max_hvm_memory : 61975 > node_to_cpu : node0:0-3,8-11 > node1:4-7,12-15 > node_to_memory : node0:30485 > node1:31647 > node_to_dma32_mem : node0:2995 > node1:0 > max_node_id : 1 > xen_major : 4 > xen_minor : 0 > xen_extra : .0_21091_04-0.2 > xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 > hvm-3.0-x86_32p hvm-3.0-x86_64 > xen_scheduler : credit > xen_pagesize : 4096 > platform_params : virt_start=0xffff800000000000 > xen_changeset : 21091 > xen_commandline : loglvl=all noreboot dom0_mem=2048M > dom0_max_vcpus=2 dom0_vcpus_pin console=vga vga=current > cc_compiler : gcc version 4.3.4 [gcc-4_3-branch revision > 152973] (SUSE Linux) > cc_compile_by : abuild > cc_compile_domain : > cc_compile_date : Thu May 20 11:44:41 UTC 2010 > xend_config_format : 4 > > > > > -- > Vasiliy G Tolstov <v.tolstov@selfip.ru> > Selfip.Ru > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Vasiliy G Tolstov
2010-Aug-12 11:37 UTC
Re: [Xen-devel] failed to start centos 5 domU with maxmem=30000
В Чтв, 12/08/2010 в 14:37 +0300, Pasi Kärkkäinen пишет:> On Thu, Aug 12, 2010 at 02:24:05PM +0400, Vasiliy G Tolstov wrote: > > Hello. I''m try to start centos domU pvm under sles 11 sp1 xen kernel > > with memory=512 and maxmem=30000. Centos doing boot and freeze. > > If i''m try to use maxmem=15000 all works. > > > > Can You tell me what''s wrong? > > > > Is the centos PV domU 32bit or 64bit ? > > -- Pasi64 bit kernel-xen. I found this in xen sources: /* Simple continuous piecewiese linear function: * max MiB -> min MiB gradient * 0 0 * 16 16 * 32 24 * 128 72 (1/2) * 512 168 (1/4) * 2048 360 (1/8) * 8192 552 (1/32) * 32768 1320 * 131072 4392 */ Why this is not provided in documentation or on web site? -- Vasiliy G Tolstov <v.tolstov@selfip.ru> Selfip.Ru _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Dan Magenheimer
2010-Aug-12 12:17 UTC
RE: [Xen-devel] failed to start centos 5 domU with maxmem=30000
> From: Vasiliy G Tolstov [mailto:v.tolstov@selfip.ru] > Sent: Thursday, August 12, 2010 5:38 AM > To: Pasi Kärkkäinen > Cc: xen-devel@lists.xensource.com > Subject: Re: [Xen-devel] failed to start centos 5 domU with > maxmem=30000 > > В Чтв, 12/08/2010 в 14:37 +0300, Pasi Kärkkäinen пишет: > > On Thu, Aug 12, 2010 at 02:24:05PM +0400, Vasiliy G Tolstov wrote: > > > Hello. I''m try to start centos domU pvm under sles 11 sp1 xen > kernel > > > with memory=512 and maxmem=30000. Centos doing boot and freeze. > > > If i''m try to use maxmem=15000 all works. > > > > > > Can You tell me what''s wrong? > > > > > > > Is the centos PV domU 32bit or 64bit ? > > 64 bit kernel-xen. > > I found this in xen sources: > > /* Simple continuous piecewiese linear function: > * max MiB -> min MiB gradient > * 0 0 > * 16 16 > * 32 24 > * 128 72 (1/2) > * 512 168 (1/4) > * 2048 360 (1/8) > * 8192 552 (1/32) > * 32768 1320 > * 131072 4392 > */ > > Why this is not provided in documentation or on web site?Hi Vasily -- This function limits how far memory can be reduced when ballooning a guest (including dom0). It is only a heuristic but is intended to take into account the various overheads a guest Linux kernel requires to manage memory to avoid out-of-memory conditions. But I think you are correct... the same (or similar) function should be published as it also serves as a guideline for the ratio between memory= and maxmemparameters when creating a guest: If the ratio of maxmem divided by memory is too high, the guest will not even boot. I am curious as to why you would specify memory= so much smaller than maxmem=. Are you trying to overcommit memory for guests that are often idle but sometimes use a very large amount of memory? Thanks, Dan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Vasiliy G Tolstov
2010-Aug-12 12:24 UTC
RE: [Xen-devel] failed to start centos 5 domU with maxmem=30000
В Чтв, 12/08/2010 в 05:17 -0700, Dan Magenheimer пишет:> > From: Vasiliy G Tolstov [mailto:v.tolstov@selfip.ru] > > Sent: Thursday, August 12, 2010 5:38 AM > > To: Pasi Kärkkäinen > > Cc: xen-devel@lists.xensource.com > > Subject: Re: [Xen-devel] failed to start centos 5 domU with > > maxmem=30000 > > > > В Чтв, 12/08/2010 в 14:37 +0300, Pasi Kärkkäinen пишет: > > > On Thu, Aug 12, 2010 at 02:24:05PM +0400, Vasiliy G Tolstov wrote: > > > > Hello. I''m try to start centos domU pvm under sles 11 sp1 xen > > kernel > > > > with memory=512 and maxmem=30000. Centos doing boot and freeze. > > > > If i''m try to use maxmem=15000 all works. > > > > > > > > Can You tell me what''s wrong? > > > > > > > > > > Is the centos PV domU 32bit or 64bit ? > > > > 64 bit kernel-xen. > > > > I found this in xen sources: > > > > /* Simple continuous piecewiese linear function: > > * max MiB -> min MiB gradient > > * 0 0 > > * 16 16 > > * 32 24 > > * 128 72 (1/2) > > * 512 168 (1/4) > > * 2048 360 (1/8) > > * 8192 552 (1/32) > > * 32768 1320 > > * 131072 4392 > > */ > > > > Why this is not provided in documentation or on web site? > > Hi Vasily -- > > This function limits how far memory can be reduced when > ballooning a guest (including dom0). It is only a heuristic > but is intended to take into account the various overheads > a guest Linux kernel requires to manage memory to avoid > out-of-memory conditions. > > But I think you are correct... the same (or similar) > function should be published as it also serves as a > guideline for the ratio between memory= and maxmem> parameters when creating a guest: If the ratio > of maxmem divided by memory is too high, the guest > will not even boot. >Is that possible to use memory=32 and maxmem=60000 ?> I am curious as to why you would specify memory= so > much smaller than maxmem=. Are you trying to overcommit > memory for guests that are often idle but sometimes use > a very large amount of memory?We want to provide ability to use small as possible memory if the guest is idle. And much as possible when the guest under heavy load. Thank You. -- Vasiliy G Tolstov <v.tolstov@selfip.ru> Selfip.Ru _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Dan Magenheimer
2010-Aug-12 18:00 UTC
RE: [Xen-devel] failed to start centos 5 domU with maxmem=30000
> > > Why this is not provided in documentation or on web site? > > > > Hi Vasily -- > > > > This function limits how far memory can be reduced when > > ballooning a guest (including dom0). It is only a heuristic > > but is intended to take into account the various overheads > > a guest Linux kernel requires to manage memory to avoid > > out-of-memory conditions. > > > > But I think you are correct... the same (or similar) > > function should be published as it also serves as a > > guideline for the ratio between memory= and maxmem> > parameters when creating a guest: If the ratio > > of maxmem divided by memory is too high, the guest > > will not even boot. > > Is that possible to use memory=32 and maxmem=60000 ?I think the answer is no. I believe there is a kernel data structure for each 4K page in physical memory I don''t remember the size of this data structure, but assume it is 4 bytes. That means that for 1GB of physical memory (as specified by maxmem) this data structure requires 1MB of physical memory just to track the 1GB. So for your maxmem=60000, memory=60 would only be enough for this one kernel data structure (and the kernel requires many other data structures to be functional). Daniel Kiper is developing virtual hotplug memory for Xen guests. This may be a good use case for it.> > I am curious as to why you would specify memory= so > > much smaller than maxmem=. Are you trying to overcommit > > memory for guests that are often idle but sometimes use > > a very large amount of memory? > > We want to provide ability to use small as possible memory if the guest > is idle. And much as possible when the guest under heavy load.This is a much harder problem than it seems. You may want to look at some of the presentations on Transcendent Memory that explain why it is hard. (Most can be found at http://oss.oracle.com/projects/tmem ) Dan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jeremy Fitzhardinge
2010-Aug-12 18:34 UTC
Re: [Xen-devel] failed to start centos 5 domU with maxmem=30000
On 08/12/2010 05:24 AM, Vasiliy G Tolstov wrote:> В Чтв, 12/08/2010 в 05:17 -0700, Dan Magenheimer пишет: >>> From: Vasiliy G Tolstov [mailto:v.tolstov@selfip.ru] >>> Sent: Thursday, August 12, 2010 5:38 AM >>> To: Pasi Kärkkäinen >>> Cc: xen-devel@lists.xensource.com >>> Subject: Re: [Xen-devel] failed to start centos 5 domU with >>> maxmem=30000 >>> >>> В Чтв, 12/08/2010 в 14:37 +0300, Pasi Kärkkäinen пишет: >>>> On Thu, Aug 12, 2010 at 02:24:05PM +0400, Vasiliy G Tolstov wrote: >>>>> Hello. I''m try to start centos domU pvm under sles 11 sp1 xen >>> kernel >>>>> with memory=512 and maxmem=30000. Centos doing boot and freeze. >>>>> If i''m try to use maxmem=15000 all works. >>>>> >>>>> Can You tell me what''s wrong? >>>>> >>>> Is the centos PV domU 32bit or 64bit ? >>> 64 bit kernel-xen. >>> >>> I found this in xen sources: >>> >>> /* Simple continuous piecewiese linear function: >>> * max MiB -> min MiB gradient >>> * 0 0 >>> * 16 16 >>> * 32 24 >>> * 128 72 (1/2) >>> * 512 168 (1/4) >>> * 2048 360 (1/8) >>> * 8192 552 (1/32) >>> * 32768 1320 >>> * 131072 4392 >>> */ >>> >>> Why this is not provided in documentation or on web site? >> Hi Vasily -- >> >> This function limits how far memory can be reduced when >> ballooning a guest (including dom0). It is only a heuristic >> but is intended to take into account the various overheads >> a guest Linux kernel requires to manage memory to avoid >> out-of-memory conditions. >> >> But I think you are correct... the same (or similar) >> function should be published as it also serves as a >> guideline for the ratio between memory= and maxmem>> parameters when creating a guest: If the ratio >> of maxmem divided by memory is too high, the guest >> will not even boot. >> > Is that possible to use memory=32 and maxmem=60000 ?A factor of ~2000 between smallest and largest size won''t work. The problem is that lots of kernel structures are scaled with the amount of boot-time memory, and they''ll simply be overloaded when you add more memory. For example, the page structure array for 60GB needs ~15.3 million entries, which at 64 bytes per entry will take ~980MB...>> I am curious as to why you would specify memory= so >> much smaller than maxmem=. Are you trying to overcommit >> memory for guests that are often idle but sometimes use >> a very large amount of memory? > We want to provide ability to use small as possible memory if the guest > is idle. And much as possible when the guest under heavy load.This is exactly the use-case for the hotplug memory stuff Daniel is working on - which I guess is why you''ve been testing it. J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Vasiliy G Tolstov
2010-Aug-13 05:54 UTC
Re: [Xen-devel] failed to start centos 5 domU with maxmem=30000
В Чтв, 12/08/2010 в 11:34 -0700, Jeremy Fitzhardinge пишет:> On 08/12/2010 05:24 AM, Vasiliy G Tolstov wrote: > > В Чтв, 12/08/2010 в 05:17 -0700, Dan Magenheimer пишет: > >>> From: Vasiliy G Tolstov [mailto:v.tolstov@selfip.ru] > >>> Sent: Thursday, August 12, 2010 5:38 AM > >>> To: Pasi Kärkkäinen > >>> Cc: xen-devel@lists.xensource.com > >>> Subject: Re: [Xen-devel] failed to start centos 5 domU with > >>> maxmem=30000 > >>> > >>> В Чтв, 12/08/2010 в 14:37 +0300, Pasi Kärkkäinen пишет: > >>>> On Thu, Aug 12, 2010 at 02:24:05PM +0400, Vasiliy G Tolstov wrote: > >>>>> Hello. I''m try to start centos domU pvm under sles 11 sp1 xen > >>> kernel > >>>>> with memory=512 and maxmem=30000. Centos doing boot and freeze. > >>>>> If i''m try to use maxmem=15000 all works. > >>>>> > >>>>> Can You tell me what''s wrong? > >>>>> > >>>> Is the centos PV domU 32bit or 64bit ? > >>> 64 bit kernel-xen. > >>> > >>> I found this in xen sources: > >>> > >>> /* Simple continuous piecewiese linear function: > >>> * max MiB -> min MiB gradient > >>> * 0 0 > >>> * 16 16 > >>> * 32 24 > >>> * 128 72 (1/2) > >>> * 512 168 (1/4) > >>> * 2048 360 (1/8) > >>> * 8192 552 (1/32) > >>> * 32768 1320 > >>> * 131072 4392 > >>> */ > >>> > >>> Why this is not provided in documentation or on web site? > >> Hi Vasily -- > >> > >> This function limits how far memory can be reduced when > >> ballooning a guest (including dom0). It is only a heuristic > >> but is intended to take into account the various overheads > >> a guest Linux kernel requires to manage memory to avoid > >> out-of-memory conditions. > >> > >> But I think you are correct... the same (or similar) > >> function should be published as it also serves as a > >> guideline for the ratio between memory= and maxmem> >> parameters when creating a guest: If the ratio > >> of maxmem divided by memory is too high, the guest > >> will not even boot. > >> > > Is that possible to use memory=32 and maxmem=60000 ? > > A factor of ~2000 between smallest and largest size won''t work. The > problem is that lots of kernel structures are scaled with the amount of > boot-time memory, and they''ll simply be overloaded when you add more > memory. For example, the page structure array for 60GB needs ~15.3 > million entries, which at 64 bytes per entry will take ~980MB... >Ok. Thank You. Bu can some one provide correct ratio for memory and maxmemory that others not to make mistakes and not try to post bugs on that? =)> >> I am curious as to why you would specify memory= so > >> much smaller than maxmem=. Are you trying to overcommit > >> memory for guests that are often idle but sometimes use > >> a very large amount of memory? > > We want to provide ability to use small as possible memory if the guest > > is idle. And much as possible when the guest under heavy load. > > This is exactly the use-case for the hotplug memory stuff Daniel is > working on - which I guess is why you''ve been testing it.This is problem, because a''m using centos, debian, ubuntu, opensuse and other systems in domU, but i read that tmem support needed in dom0 and domU. Debian and Ubuntu i think, does not support this... But i''m try. -- Vasiliy G Tolstov <v.tolstov@selfip.ru> Selfip.Ru _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Vasiliy G Tolstov
2010-Aug-13 05:56 UTC
RE: [Xen-devel] failed to start centos 5 domU with maxmem=30000
В Чтв, 12/08/2010 в 11:00 -0700, Dan Magenheimer пишет:> > > > Why this is not provided in documentation or on web site? > > > > > > Hi Vasily -- > > > > > > This function limits how far memory can be reduced when > > > ballooning a guest (including dom0). It is only a heuristic > > > but is intended to take into account the various overheads > > > a guest Linux kernel requires to manage memory to avoid > > > out-of-memory conditions. > > > > > > But I think you are correct... the same (or similar) > > > function should be published as it also serves as a > > > guideline for the ratio between memory= and maxmem> > > parameters when creating a guest: If the ratio > > > of maxmem divided by memory is too high, the guest > > > will not even boot. > > > > Is that possible to use memory=32 and maxmem=60000 ? > > I think the answer is no. I believe there is a kernel > data structure for each 4K page in physical memory > I don''t remember the size of this data structure, but > assume it is 4 bytes. That means that for 1GB of > physical memory (as specified by maxmem) this data > structure requires 1MB of physical memory just to track > the 1GB. So for your maxmem=60000, memory=60 would > only be enough for this one kernel data structure (and > the kernel requires many other data structures to > be functional). >Ok. Thank''s.> Daniel Kiper is developing virtual hotplug memory for Xen > guests. This may be a good use case for it. >I''m try for my systems. Thank''s> > > I am curious as to why you would specify memory= so > > > much smaller than maxmem=. Are you trying to overcommit > > > memory for guests that are often idle but sometimes use > > > a very large amount of memory? > > > > We want to provide ability to use small as possible memory if the guest > > is idle. And much as possible when the guest under heavy load. > > This is a much harder problem than it seems. You may want > to look at some of the presentations on Transcendent Memory > that explain why it is hard. (Most can be found at > http://oss.oracle.com/projects/tmem )Need to support in domU... this is bad. But i''m try it. -- Vasiliy G Tolstov <v.tolstov@selfip.ru> Selfip.Ru _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Dan Magenheimer
2010-Aug-13 15:19 UTC
RE: [Xen-devel] failed to start centos 5 domU with maxmem=30000
> This is problem, because a''m using centos, debian, ubuntu, opensuse and > other systems in domU, but i read that tmem support needed in dom0 and > domU. Debian and Ubuntu i think, does not support this... But i''m try.I have kernel rpm''s for some versions of Oracle Enterprise Linux that might work on the same version of centos. I''m told that an earlier version of the tmem patch (without self-ballooning) was added to opensuse. Due to upstream Linux kernel change requests, the tmem Linux patches continue to change... cosmetic changes, all core functionality is the same. I''ve been focused on that rather than on keeping the patches for older versions up to date. But if you are eager to try tmem on a particular distro and you have experience in patching kernels for that distro, I would be happy to help. Also, tmem does not require dom0 kernel changes, but the toolset that create domains must be aware of tmem because tmem introduces the concept of "freeable" memory... since tmem eventually uses all free memory in the system, creating a domain may require freeable memory to be freed and the tools need to know how to do that. Xend (xm) works fine as of 4.0.0. Again, efficient utilization of memory is a very hard problem. "Simple" solutions that have been tried to date (such as externally ballooning all guests from the host, and page sharing with host swapping) have unacceptable side-effects. Tmem uses paravirtualization to achieve a much more efficient solution to the problem, but it requires guest changes. Thanks, Dan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel