Hello, i just tried to add a domU with 16 gigs of ram (maxmem = 16384, memory = 16384) with free_memory : 28969 but got an error: libxl: error: libxl.c:3433:libxl_set_memory_target: new target 0 for dom0 is below the minimum threshold I was able to create it with 8 gigs of ram, ending up with free_memory : 20776 Is there something wrong with memory in 4.2? I can''t seem to use all of it. Cheers, Alex
Hello, i just tried to add a domU with 16 gigs of ram (maxmem = 16384, memory = 16384) with free_memory : 28969 but got an error: libxl: error: libxl.c:3433:libxl_set_memory_target: new target 0 for dom0 is below the minimum threshold I was able to create it with 8 gigs of ram, ending up with free_memory : 20776 Is there something wrong with memory in 4.2? I can''t seem to use all of it. Cheers, Alex
On Fri, Nov 23, 2012 at 12:35:26PM +0100, Alexander Bienzeisler wrote:> Hello, > > i just tried to add a domU with 16 gigs of ram (maxmem = 16384, > memory = 16384) with free_memory : 28969 but got an error: > > libxl: error: libxl.c:3433:libxl_set_memory_target: new target 0 for > dom0 is below the minimum threshold > > I was able to create it with 8 gigs of ram, ending up with > free_memory : 20776 > > Is there something wrong with memory in 4.2? I can''t seem to use all of it. >What exact Xen version are you using? what dom0 kernel version? Also please pastebin "xl info" and "xl list" before and after starting the VM. -- Pasi
I am currently using Xen 4.2.1-pre built from git source. Kernel version 3.1.1 xl info before: host : susi-0 release : 3.1.1-xen version : #1 SMP Fri Nov 18 10:16:22 CET 2011 machine : x86_64 nr_cpus : 24 max_cpu_id : 31 nr_nodes : 2 cores_per_socket : 6 threads_per_core : 2 cpu_mhz : 3325 hw_caps : bfebfbff:2c100800:00000000:00003f40:029ee3ff:00000000:00000001:00000000 virt_caps : hvm hvm_directio total_memory : 98291 free_memory : 28969 sharing_freed_memory : 0 sharing_used_memory : 0 free_cpus : 0 xen_major : 4 xen_minor : 2 xen_extra : .1-pre xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 xen_scheduler : credit xen_pagesize : 4096 platform_params : virt_start=0xffff800000000000 xen_changeset : Wed Nov 14 11:46:07 2012 +0000 25922:8ca6372315f8 xen_commandline : placeholder loglvl=all dom0_mem=2048M dom0_max_vcpus=2 dom0_vcpus_pin com2=115200 console=com2,vga cc_compiler : gcc (Debian 4.4.5-8) 4.4.5 cc_compile_by : root cc_compile_domain : wsk.tu-chemnitz.de cc_compile_date : Thu Nov 22 13:50:09 CET 2012 xend_config_format : 4 xl list before: Name ID Mem VCPUs State Time(s) Domain-0 0 2048 2 r----- 3490.0 Squeeze 19 4091 2 -b---- 370.3 win2k8_2_susi6 31 16373 12 -b---- 1619.3 win2k8_susi5 33 32757 12 -b---- 1264.0 win7_susi3 43 4085 4 -b---- 416.0 scientific_susi7 44 8192 16 -b---- 39.9 domU config: name = "centos_susi8" uuid = "2e87b315-fd5a-dfbc-fe9d-6e121fdfa632" maxmem = 16384 memory = 16384 vcpus = 16 builder = "generic" bootloader = "pygrub" on_poweroff = "destroy" on_reboot = "restart" on_crash = "restart" vfb = [ "type=vnc,vncunused=1,keymap=de" ] disk = [ "phy:/dev/mapper/storage-SL5,xvda,w" ] vif = [ "mac=00:50:56:37:61:17,bridge=xenbr0" ] error message after: xl create /root/configs/centos_susi8.cfg Parsing config from /root/configs/centos_susi8.cfg libxl: error: libxl.c:3433:libxl_set_memory_target: new target 0 for dom0 is below the minimum threshold with 8gigs: xl info after: host : susi-0 release : 3.1.1-xen version : #1 SMP Fri Nov 18 10:16:22 CET 2011 machine : x86_64 nr_cpus : 24 max_cpu_id : 31 nr_nodes : 2 cores_per_socket : 6 threads_per_core : 2 cpu_mhz : 3325 hw_caps : bfebfbff:2c100800:00000000:00003f40:029ee3ff:00000000:00000001:00000000 virt_caps : hvm hvm_directio total_memory : 98291 free_memory : 20776 sharing_freed_memory : 0 sharing_used_memory : 0 free_cpus : 0 xen_major : 4 xen_minor : 2 xen_extra : .1-pre xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 xen_scheduler : credit xen_pagesize : 4096 platform_params : virt_start=0xffff800000000000 xen_changeset : Wed Nov 14 11:46:07 2012 +0000 25922:8ca6372315f8 xen_commandline : placeholder loglvl=all dom0_mem=2048M dom0_max_vcpus=2 dom0_vcpus_pin com2=115200 console=com2,vga cc_compiler : gcc (Debian 4.4.5-8) 4.4.5 cc_compile_by : root cc_compile_domain : wsk.tu-chemnitz.de cc_compile_date : Thu Nov 22 13:50:09 CET 2012 xend_config_format : 4 xl list after: Name ID Mem VCPUs State Time(s) Domain-0 0 2048 2 r----- 3550.0 Squeeze 19 4091 2 -b---- 381.5 win2k8_2_susi6 31 16373 12 -b---- 1627.1 win2k8_susi5 33 32757 12 ------ 1269.0 win7_susi3 43 4085 4 -b---- 418.7 scientific_susi7 44 8192 16 -b---- 40.3 centos_susi8 46 8192 16 -b---- 4.4 Am 23.11.2012 12:52, schrieb Pasi Kärkkäinen:> On Fri, Nov 23, 2012 at 12:35:26PM +0100, Alexander Bienzeisler wrote: >> Hello, >> >> i just tried to add a domU with 16 gigs of ram (maxmem = 16384, >> memory = 16384) with free_memory : 28969 but got an error: >> >> libxl: error: libxl.c:3433:libxl_set_memory_target: new target 0 for >> dom0 is below the minimum threshold >> >> I was able to create it with 8 gigs of ram, ending up with >> free_memory : 20776 >> >> Is there something wrong with memory in 4.2? I can''t seem to use all of it. >> > What exact Xen version are you using? what dom0 kernel version? > > Also please pastebin "xl info" and "xl list" before and after starting the VM. > > -- Pasi >
This question belongs on xen-users. On Fri, 2012-11-23 at 11:35 +0000, Alexander Bienzeisler wrote:> Hello, > > i just tried to add a domU with 16 gigs of ram (maxmem = 16384, memory = > 16384) with free_memory : 28969 but got an error: > > libxl: error: libxl.c:3433:libxl_set_memory_target: new target 0 for > dom0 is below the minimum thresholdThe clue is in this message -- you cannot safely balloon down a guest (including dom0) to an arbitrarily small value relative to its initial size. The kernel will have certain fixed datastructures etc which are sized according to the maximum amount of RAM at start of day. If you were to balloon down too low then you would run out of RAM and the OOM killer would step in and start killing stuff, so xl includes a safety catch which prevents this. If you have a huge host and want a small dom0 use the dom0_mem option and disable xl''s autoballoon option or whatever the equivalent xend option is. Ian.
On Fri, Nov 23, 2012 at 12:21:32PM +0000, Ian Campbell wrote:> This question belongs on xen-users. > > On Fri, 2012-11-23 at 11:35 +0000, Alexander Bienzeisler wrote: > > Hello, > > > > i just tried to add a domU with 16 gigs of ram (maxmem = 16384, memory = > > 16384) with free_memory : 28969 but got an error: > > > > libxl: error: libxl.c:3433:libxl_set_memory_target: new target 0 for > > dom0 is below the minimum threshold > > The clue is in this message -- you cannot safely balloon down a guest > (including dom0) to an arbitrarily small value relative to its initial > size. The kernel will have certain fixed datastructures etc which are > sized according to the maximum amount of RAM at start of day. > > If you were to balloon down too low then you would run out of RAM and > the OOM killer would step in and start killing stuff, so xl includes a > safety catch which prevents this. > > If you have a huge host and want a small dom0 use the dom0_mem option > and disable xl''s autoballoon option or whatever the equivalent xend > option is. >I think he was already using dom0_mem. He had 28969 free in Xen as reported by "xl info". -- Pasi
On Fri, 2012-11-23 at 12:28 +0000, Pasi Kärkkäinen wrote:> On Fri, Nov 23, 2012 at 12:21:32PM +0000, Ian Campbell wrote: > > This question belongs on xen-users. > > > > On Fri, 2012-11-23 at 11:35 +0000, Alexander Bienzeisler wrote: > > > Hello, > > > > > > i just tried to add a domU with 16 gigs of ram (maxmem = 16384, memory = > > > 16384) with free_memory : 28969 but got an error: > > > > > > libxl: error: libxl.c:3433:libxl_set_memory_target: new target 0 for > > > dom0 is below the minimum threshold > > > > The clue is in this message -- you cannot safely balloon down a guest > > (including dom0) to an arbitrarily small value relative to its initial > > size. The kernel will have certain fixed datastructures etc which are > > sized according to the maximum amount of RAM at start of day. > > > > If you were to balloon down too low then you would run out of RAM and > > the OOM killer would step in and start killing stuff, so xl includes a > > safety catch which prevents this. > > > > If you have a huge host and want a small dom0 use the dom0_mem option > > and disable xl's autoballoon option or whatever the equivalent xend > > option is. > > > > I think he was already using dom0_mem. He had 28969 free in Xen as > reported by "xl info".Then he's missed disabling autoballoon.> > -- Pasi >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On Fri, Nov 23, 2012 at 12:33:59PM +0000, Ian Campbell wrote:> On Fri, 2012-11-23 at 12:28 +0000, Pasi Kärkkäinen wrote: > > On Fri, Nov 23, 2012 at 12:21:32PM +0000, Ian Campbell wrote: > > > This question belongs on xen-users. > > > > > > On Fri, 2012-11-23 at 11:35 +0000, Alexander Bienzeisler wrote: > > > > Hello, > > > > > > > > i just tried to add a domU with 16 gigs of ram (maxmem = 16384, memory = > > > > 16384) with free_memory : 28969 but got an error: > > > > > > > > libxl: error: libxl.c:3433:libxl_set_memory_target: new target 0 for > > > > dom0 is below the minimum threshold > > > > > > The clue is in this message -- you cannot safely balloon down a guest > > > (including dom0) to an arbitrarily small value relative to its initial > > > size. The kernel will have certain fixed datastructures etc which are > > > sized according to the maximum amount of RAM at start of day. > > > > > > If you were to balloon down too low then you would run out of RAM and > > > the OOM killer would step in and start killing stuff, so xl includes a > > > safety catch which prevents this. > > > > > > If you have a huge host and want a small dom0 use the dom0_mem option > > > and disable xl''s autoballoon option or whatever the equivalent xend > > > option is. > > > > > > > I think he was already using dom0_mem. He had 28969 free in Xen as > > reported by "xl info". > > Then he''s missed disabling autoballoon. >As can be seen from the other email in this thread, he''s using dom0_mem=2048M, and he has 28GB of free memory in Xen. Why is xl trying to balloon down dom0 (and failing) for no reason? -- Pasi
As Ian said, i did not disable autoballooning. It works with autoballooning=0. Thanks everyone and sorry for wasting your time. Am 23.11.2012 13:33, schrieb Ian Campbell:> On Fri, 2012-11-23 at 12:28 +0000, Pasi Kärkkäinen wrote: >> On Fri, Nov 23, 2012 at 12:21:32PM +0000, Ian Campbell wrote: >>> This question belongs on xen-users. >>> >>> On Fri, 2012-11-23 at 11:35 +0000, Alexander Bienzeisler wrote: >>>> Hello, >>>> >>>> i just tried to add a domU with 16 gigs of ram (maxmem = 16384, memory >>>> 16384) with free_memory : 28969 but got an error: >>>> >>>> libxl: error: libxl.c:3433:libxl_set_memory_target: new target 0 for >>>> dom0 is below the minimum threshold >>> The clue is in this message -- you cannot safely balloon down a guest >>> (including dom0) to an arbitrarily small value relative to its initial >>> size. The kernel will have certain fixed datastructures etc which are >>> sized according to the maximum amount of RAM at start of day. >>> >>> If you were to balloon down too low then you would run out of RAM and >>> the OOM killer would step in and start killing stuff, so xl includes a >>> safety catch which prevents this. >>> >>> If you have a huge host and want a small dom0 use the dom0_mem option >>> and disable xl's autoballoon option or whatever the equivalent xend >>> option is. >>> >> I think he was already using dom0_mem. He had 28969 free in Xen as >> reported by "xl info". > Then he's missed disabling autoballoon. > >> -- Pasi >> >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On Fri, 2012-11-23 at 12:41 +0000, Pasi Kärkkäinen wrote:> On Fri, Nov 23, 2012 at 12:33:59PM +0000, Ian Campbell wrote: > > On Fri, 2012-11-23 at 12:28 +0000, Pasi Kärkkäinen wrote: > > > On Fri, Nov 23, 2012 at 12:21:32PM +0000, Ian Campbell wrote: > > > > This question belongs on xen-users. > > > > > > > > On Fri, 2012-11-23 at 11:35 +0000, Alexander Bienzeisler wrote: > > > > > Hello, > > > > > > > > > > i just tried to add a domU with 16 gigs of ram (maxmem = 16384, memory = > > > > > 16384) with free_memory : 28969 but got an error: > > > > > > > > > > libxl: error: libxl.c:3433:libxl_set_memory_target: new target 0 for > > > > > dom0 is below the minimum threshold > > > > > > > > The clue is in this message -- you cannot safely balloon down a guest > > > > (including dom0) to an arbitrarily small value relative to its initial > > > > size. The kernel will have certain fixed datastructures etc which are > > > > sized according to the maximum amount of RAM at start of day. > > > > > > > > If you were to balloon down too low then you would run out of RAM and > > > > the OOM killer would step in and start killing stuff, so xl includes a > > > > safety catch which prevents this. > > > > > > > > If you have a huge host and want a small dom0 use the dom0_mem option > > > > and disable xl's autoballoon option or whatever the equivalent xend > > > > option is. > > > > > > > > > > I think he was already using dom0_mem. He had 28969 free in Xen as > > > reported by "xl info". > > > > Then he's missed disabling autoballoon. > > > > As can be seen from the other email in this thread, > he's using dom0_mem=2048M, and he has 28GB of free memory in Xen. > > Why is xl trying to balloon down dom0 (and failing) for no reason?Because autoballoon is enabled. This has been discussed at length on the list before, please check the archives. Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On Fri, Nov 23, 2012 at 12:47:07PM +0000, Ian Campbell wrote:> On Fri, 2012-11-23 at 12:41 +0000, Pasi Kärkkäinen wrote: > > On Fri, Nov 23, 2012 at 12:33:59PM +0000, Ian Campbell wrote: > > > On Fri, 2012-11-23 at 12:28 +0000, Pasi Kärkkäinen wrote: > > > > On Fri, Nov 23, 2012 at 12:21:32PM +0000, Ian Campbell wrote: > > > > > This question belongs on xen-users. > > > > > > > > > > On Fri, 2012-11-23 at 11:35 +0000, Alexander Bienzeisler wrote: > > > > > > Hello, > > > > > > > > > > > > i just tried to add a domU with 16 gigs of ram (maxmem = 16384, memory = > > > > > > 16384) with free_memory : 28969 but got an error: > > > > > > > > > > > > libxl: error: libxl.c:3433:libxl_set_memory_target: new target 0 for > > > > > > dom0 is below the minimum threshold > > > > > > > > > > The clue is in this message -- you cannot safely balloon down a guest > > > > > (including dom0) to an arbitrarily small value relative to its initial > > > > > size. The kernel will have certain fixed datastructures etc which are > > > > > sized according to the maximum amount of RAM at start of day. > > > > > > > > > > If you were to balloon down too low then you would run out of RAM and > > > > > the OOM killer would step in and start killing stuff, so xl includes a > > > > > safety catch which prevents this. > > > > > > > > > > If you have a huge host and want a small dom0 use the dom0_mem option > > > > > and disable xl''s autoballoon option or whatever the equivalent xend > > > > > option is. > > > > > > > > > > > > > I think he was already using dom0_mem. He had 28969 free in Xen as > > > > reported by "xl info". > > > > > > Then he''s missed disabling autoballoon. > > > > > > > As can be seen from the other email in this thread, > > he''s using dom0_mem=2048M, and he has 28GB of free memory in Xen. > > > > Why is xl trying to balloon down dom0 (and failing) for no reason? > > Because autoballoon is enabled. >Yep, autoballoon=0 fixed the problem.> This has been discussed at length on the list before, please check the > archives. >I tried googling quickly but I didn''t find anything relevant.. To me this behaviour still seems wrong. What''s the point of autoballoon=1 trying to balloon down dom0 if the hypervisor already has enough free memory for the VM ? In this case: - dom0_mem=2G - new VM to launch with size 16 GB. - Xen has 28 GB of free memory. So clearly there''s no need to try to balloon down dom0.. not-yet-implemented check/feature in xl, or a bug? Thanks, -- Pasi
On Fri, 2012-11-23 at 12:55 +0000, Pasi Kärkkäinen wrote:> > This has been discussed at length on the list before, please check the > > archives. > > > > I tried googling quickly but I didn't find anything relevant..I found http://web.archiveorange.com/archive/v/zCKz5T3PLvtyZDSPQc9i in a matter of seconds, then: http://lists.xen.org/archives/html/xen-devel/2011-11/msg01415.html http://lists.xen.org/archives/html/xen-users/2012-05/msg00139.html http://lists.xen.org/archives/html/xen-users/2012-05/msg00146.html> To me this behaviour still seems wrong. What's the point of autoballoon=1 trying to > balloon down dom0 if the hypervisor already has enough free memory for the VM ? > > In this case: > - dom0_mem=2G > - new VM to launch with size 16 GB. > - Xen has 28 GB of free memory. > > So clearly there's no need to try to balloon down dom0..Right, so don't set autoballoon then.> not-yet-implemented check/feature in xl, or a bug?Neither, it is the intended behaviour of xl autoballoon, this option means exactly "take the required memory from dom0". Ian. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On Fri, Nov 23, 2012 at 01:15:53PM +0000, Ian Campbell wrote:> On Fri, 2012-11-23 at 12:55 +0000, Pasi Kärkkäinen wrote: > > > > This has been discussed at length on the list before, please check the > > > archives. > > > > > > > I tried googling quickly but I didn''t find anything relevant.. > > I found > http://web.archiveorange.com/archive/v/zCKz5T3PLvtyZDSPQc9i > in a matter of seconds, then: > http://lists.xen.org/archives/html/xen-devel/2011-11/msg01415.html > http://lists.xen.org/archives/html/xen-users/2012-05/msg00139.html > http://lists.xen.org/archives/html/xen-users/2012-05/msg00146.html > > > To me this behaviour still seems wrong. What''s the point of autoballoon=1 trying to > > balloon down dom0 if the hypervisor already has enough free memory for the VM ? > > > > In this case: > > - dom0_mem=2G > > - new VM to launch with size 16 GB. > > - Xen has 28 GB of free memory. > > > > So clearly there''s no need to try to balloon down dom0.. > > Right, so don''t set autoballoon then. >It''s enabled as a default.. so many people hit this problem.> > not-yet-implemented check/feature in xl, or a bug? > > Neither, it is the intended behaviour of xl autoballoon, this option > means exactly "take the required memory from dom0". >http://xenbits.xen.org/docs/4.2-testing/man/xl.conf.5.html "autoballoon=BOOLEAN If disabled then xl will not attempt to reduce the amount of memory assigned to domain 0 in order to create free memory when starting a new domain. You should set this if you use the dom0_mem hypervisor command line to reduce the amount of memory given to domain 0 by default. Default: 1" I think we should modify that to say "You should set autoballoon=0 if you use the dom0_mem hypervisor command line .." At least I understood that text in the opposite way.. Also: What happens if you have autoballoon=1 and you start some VMs, then stop the VMs, so most of the memory is now free in Xen.. and then you try to start a big VM ? Aren''t you going to hit the same problem as in this thread? -- Pasi
On Fri, 2012-11-23 at 13:29 +0000, Pasi Kärkkäinen wrote:> On Fri, Nov 23, 2012 at 01:15:53PM +0000, Ian Campbell wrote: > > On Fri, 2012-11-23 at 12:55 +0000, Pasi Kärkkäinen wrote: > > > > > > This has been discussed at length on the list before, please check the > > > > archives. > > > > > > > > > > I tried googling quickly but I didn't find anything relevant.. > > > > I found > > http://web.archiveorange.com/archive/v/zCKz5T3PLvtyZDSPQc9i > > in a matter of seconds, then: > > http://lists.xen.org/archives/html/xen-devel/2011-11/msg01415.html > > http://lists.xen.org/archives/html/xen-users/2012-05/msg00139.html > > http://lists.xen.org/archives/html/xen-users/2012-05/msg00146.html > > > > > To me this behaviour still seems wrong. What's the point of autoballoon=1 trying to > > > balloon down dom0 if the hypervisor already has enough free memory for the VM ? > > > > > > In this case: > > > - dom0_mem=2G > > > - new VM to launch with size 16 GB. > > > - Xen has 28 GB of free memory. > > > > > > So clearly there's no need to try to balloon down dom0.. > > > > Right, so don't set autoballoon then. > > > > It's enabled as a default.. so many people hit this problem. > > > > > not-yet-implemented check/feature in xl, or a bug? > > > > Neither, it is the intended behaviour of xl autoballoon, this option > > means exactly "take the required memory from dom0". > > > > http://xenbits.xen.org/docs/4.2-testing/man/xl.conf.5.html > > "autoballoon=BOOLEAN > > If disabled then xl will not attempt to reduce the amount of > memory assigned to domain 0 in order to create free memory when > starting a new domain. You should set this if you use the dom0_mem > hypervisor command line to reduce the amount of memory given to domain > 0 by default. > > Default: 1" > > > I think we should modify that to say "You should set autoballoon=0 if you use the dom0_mem hypervisor command line .." > At least I understood that text in the opposite way..Yes, there should be a s/set/unset/ in there I think.> Also: What happens if you have autoballoon=1 and you start some VMs, then stop the VMs, > so most of the memory is now free in Xen..xl balloons dom0 back up when it destroy domains with autoballoon=1.> and then you try to start a big VM ? > Aren't you going to hit the same problem as in this thread? > > > -- Pasi >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
> From: Ian Campbell [mailto:Ian.Campbell@citrix.com] > Sent: Friday, November 23, 2012 6:34 AM > To: Pasi Kärkkäinen > Cc: xen-devel@lists.xensource.com; Alexander Bienzeisler > Subject: Re: [Xen-devel] can not use all available memory > > On Fri, 2012-11-23 at 13:29 +0000, Pasi Kärkkäinen wrote: > > On Fri, Nov 23, 2012 at 01:15:53PM +0000, Ian Campbell wrote: > > > On Fri, 2012-11-23 at 12:55 +0000, Pasi Kärkkäinen wrote: > > > > > > > > This has been discussed at length on the list before, please check the > > > > > archives. > > > > > > > > > > > > > I tried googling quickly but I didn't find anything relevant.. > > > > > > I found > > > http://web.archiveorange.com/archive/v/zCKz5T3PLvtyZDSPQc9i > > > in a matter of seconds, then: > > > http://lists.xen.org/archives/html/xen-devel/2011-11/msg01415.html > > > http://lists.xen.org/archives/html/xen-users/2012-05/msg00139.html > > > http://lists.xen.org/archives/html/xen-users/2012-05/msg00146.html > > > > > > > To me this behaviour still seems wrong. What's the point of autoballoon=1 trying to > > > > balloon down dom0 if the hypervisor already has enough free memory for the VM ? > > > > > > > > In this case: > > > > - dom0_mem=2G > > > > - new VM to launch with size 16 GB. > > > > - Xen has 28 GB of free memory. > > > > > > > > So clearly there's no need to try to balloon down dom0.. > > > > > > Right, so don't set autoballoon then. > > > > > > > It's enabled as a default.. so many people hit this problem. > > > > > > > > not-yet-implemented check/feature in xl, or a bug? > > > > > > Neither, it is the intended behaviour of xl autoballoon, this option > > > means exactly "take the required memory from dom0". > > > > > > > http://xenbits.xen.org/docs/4.2-testing/man/xl.conf.5.html > > > > "autoballoon=BOOLEAN > > > > If disabled then xl will not attempt to reduce the amount of > > memory assigned to domain 0 in order to create free memory when > > starting a new domain. You should set this if you use the dom0_mem > > hypervisor command line to reduce the amount of memory given to domain > > 0 by default. > > > > Default: 1" > > > > > > I think we should modify that to say "You should set autoballoon=0 if you use the dom0_mem > hypervisor command line .." > > At least I understood that text in the opposite way.. > > Yes, there should be a s/set/unset/ in there I think. > > > Also: What happens if you have autoballoon=1 and you start some VMs, then stop the VMs, > > so most of the memory is now free in Xen.. > > xl balloons dom0 back up when it destroy domains with autoballoon=1. > > > and then you try to start a big VM ? > > Aren't you going to hit the same problem as in this thread?Hmmm... this behavior and default may make sense for the Citrix memory model (single machine, dom0 is "the user" so you want it to always hold most of physical RAM not used by guests). But probably not so for a more cloud-like memory model. Is there any (easy) way to force autoballoon=0 if the hypervisor dom0_mem boot option is specified? Or is there some reasonably sane case I am missing where a user would want both dom0_mem and autoballoon=1? Oracle VM always boots servers with dom0_mem= set so (if/when OVM switches to use xl), OVM will always set autoballoon off. So it's the large number of non-Citrix-non-Oracle Xen-as-a-service providers I am trying to help here. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Monday, November 26, 2012, 5:58:28 PM, you wrote:>> From: Ian Campbell [mailto:Ian.Campbell@citrix.com] >> Sent: Friday, November 23, 2012 6:34 AM >> To: Pasi Kärkkäinen >> Cc: xen-devel@lists.xensource.com; Alexander Bienzeisler >> Subject: Re: [Xen-devel] can not use all available memory >> >> On Fri, 2012-11-23 at 13:29 +0000, Pasi Kärkkäinen wrote: >> > On Fri, Nov 23, 2012 at 01:15:53PM +0000, Ian Campbell wrote: >> > > On Fri, 2012-11-23 at 12:55 +0000, Pasi Kärkkäinen wrote: >> > > >> > > > > This has been discussed at length on the list before, please check the >> > > > > archives. >> > > > > >> > > > >> > > > I tried googling quickly but I didn't find anything relevant.. >> > > >> > > I found >> > > http://web.archiveorange.com/archive/v/zCKz5T3PLvtyZDSPQc9i >> > > in a matter of seconds, then: >> > > http://lists.xen.org/archives/html/xen-devel/2011-11/msg01415.html >> > > http://lists.xen.org/archives/html/xen-users/2012-05/msg00139.html >> > > http://lists.xen.org/archives/html/xen-users/2012-05/msg00146.html >> > > >> > > > To me this behaviour still seems wrong. What's the point of autoballoon=1 trying to >> > > > balloon down dom0 if the hypervisor already has enough free memory for the VM ? >> > > > >> > > > In this case: >> > > > - dom0_mem=2G >> > > > - new VM to launch with size 16 GB. >> > > > - Xen has 28 GB of free memory. >> > > > >> > > > So clearly there's no need to try to balloon down dom0.. >> > > >> > > Right, so don't set autoballoon then. >> > > >> > >> > It's enabled as a default.. so many people hit this problem. >> > >> > >> > > > not-yet-implemented check/feature in xl, or a bug? >> > > >> > > Neither, it is the intended behaviour of xl autoballoon, this option >> > > means exactly "take the required memory from dom0". >> > > >> > >> > http://xenbits.xen.org/docs/4.2-testing/man/xl.conf.5.html >> > >> > "autoballoon=BOOLEAN >> > >> > If disabled then xl will not attempt to reduce the amount of >> > memory assigned to domain 0 in order to create free memory when >> > starting a new domain. You should set this if you use the dom0_mem >> > hypervisor command line to reduce the amount of memory given to domain >> > 0 by default. >> > >> > Default: 1" >> > >> > >> > I think we should modify that to say "You should set autoballoon=0 if you use the dom0_mem >> hypervisor command line .." >> > At least I understood that text in the opposite way.. >> >> Yes, there should be a s/set/unset/ in there I think. >> >> > Also: What happens if you have autoballoon=1 and you start some VMs, then stop the VMs, >> > so most of the memory is now free in Xen.. >> >> xl balloons dom0 back up when it destroy domains with autoballoon=1. >> >> > and then you try to start a big VM ? >> > Aren't you going to hit the same problem as in this thread?> Hmmm... this behavior and default may make sense for the Citrix > memory model (single machine, dom0 is "the user" so you want it > to always hold most of physical RAM not used by guests). But > probably not so for a more cloud-like memory model.> Is there any (easy) way to force autoballoon=0 if the hypervisor > dom0_mem boot option is specified? Or is there some reasonably > sane case I am missing where a user would want both dom0_mem > and autoballoon=1?> Oracle VM always boots servers with dom0_mem= set so (if/when > OVM switches to use xl), OVM will always set autoballoon off. > So it's the large number of non-Citrix-non-Oracle Xen-as-a-service > providers I am trying to help here.Hmm i was bitten by this 2 weeks ago, i found it a bit not intuitive that: - While the guest i was trying to start required less memory than was freely available (according to xentop) outside of dom0 - It would fail, because xl started to try to balloon dom0 down which failed. -- Sander _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
> From: Sander Eikelenboom [mailto:linux@eikelenboom.it] > Subject: Re: [Xen-devel] can not use all available memory > > Monday, November 26, 2012, 5:58:28 PM, you wrote: > > >> From: Ian Campbell [mailto:Ian.Campbell@citrix.com] > >> Sent: Friday, November 23, 2012 6:34 AM > >> To: Pasi Kärkkäinen > >> Cc: xen-devel@lists.xensource.com; Alexander Bienzeisler > >> Subject: Re: [Xen-devel] can not use all available memory > >> > >> On Fri, 2012-11-23 at 13:29 +0000, Pasi Kärkkäinen wrote: > >> > On Fri, Nov 23, 2012 at 01:15:53PM +0000, Ian Campbell wrote: > >> > > On Fri, 2012-11-23 at 12:55 +0000, Pasi Kärkkäinen wrote: > >> > > > >> > > > > This has been discussed at length on the list before, please check the > >> > > > > archives. > >> > > > > > >> > > > > >> > > > I tried googling quickly but I didn't find anything relevant.. > >> > > > >> > > I found > >> > > http://web.archiveorange.com/archive/v/zCKz5T3PLvtyZDSPQc9i > >> > > in a matter of seconds, then: > >> > > http://lists.xen.org/archives/html/xen-devel/2011-11/msg01415.html > >> > > http://lists.xen.org/archives/html/xen-users/2012-05/msg00139.html > >> > > http://lists.xen.org/archives/html/xen-users/2012-05/msg00146.html > >> > > > >> > > > To me this behaviour still seems wrong. What's the point of autoballoon=1 trying to > >> > > > balloon down dom0 if the hypervisor already has enough free memory for the VM ? > >> > > > > >> > > > In this case: > >> > > > - dom0_mem=2G > >> > > > - new VM to launch with size 16 GB. > >> > > > - Xen has 28 GB of free memory. > >> > > > > >> > > > So clearly there's no need to try to balloon down dom0.. > >> > > > >> > > Right, so don't set autoballoon then. > >> > > > >> > > >> > It's enabled as a default.. so many people hit this problem. > >> > > >> > > >> > > > not-yet-implemented check/feature in xl, or a bug? > >> > > > >> > > Neither, it is the intended behaviour of xl autoballoon, this option > >> > > means exactly "take the required memory from dom0". > >> > > > >> > > >> > http://xenbits.xen.org/docs/4.2-testing/man/xl.conf.5.html > >> > > >> > "autoballoon=BOOLEAN > >> > > >> > If disabled then xl will not attempt to reduce the amount of > >> > memory assigned to domain 0 in order to create free memory when > >> > starting a new domain. You should set this if you use the dom0_mem > >> > hypervisor command line to reduce the amount of memory given to domain > >> > 0 by default. > >> > > >> > Default: 1" > >> > > >> > > >> > I think we should modify that to say "You should set autoballoon=0 if you use the dom0_mem > >> hypervisor command line .." > >> > At least I understood that text in the opposite way.. > >> > >> Yes, there should be a s/set/unset/ in there I think. > >> > >> > Also: What happens if you have autoballoon=1 and you start some VMs, then stop the VMs, > >> > so most of the memory is now free in Xen.. > >> > >> xl balloons dom0 back up when it destroy domains with autoballoon=1. > >> > >> > and then you try to start a big VM ? > >> > Aren't you going to hit the same problem as in this thread? > > > Hmmm... this behavior and default may make sense for the Citrix > > memory model (single machine, dom0 is "the user" so you want it > > to always hold most of physical RAM not used by guests). But > > probably not so for a more cloud-like memory model. > > > Is there any (easy) way to force autoballoon=0 if the hypervisor > > dom0_mem boot option is specified? Or is there some reasonably > > sane case I am missing where a user would want both dom0_mem > > and autoballoon=1? > > > Oracle VM always boots servers with dom0_mem= set so (if/when > > OVM switches to use xl), OVM will always set autoballoon off. > > So it's the large number of non-Citrix-non-Oracle Xen-as-a-service > > providers I am trying to help here. > > > > Hmm i was bitten by this 2 weeks ago, i found it a bit not intuitive that: > - While the guest i was trying to start required less memory than was freely available (according to > xentop) outside of dom0 > - It would fail, because xl started to try to balloon dom0 down which failed.Hi Sander -- I could be wrong (and I am confident someone will correct me if I am) but I think this is because the Citrix memory model assumes there is an inference-driven policy engine for load-balancing memory across competing virtual machines ("squeezed"). I suspect squeezed returns unallocated xen "free" memory to dom0. IMHO, such policy engines are good for demos and so salespeople can say "yes, this product supports memory overcommit" but Transcendent Memory goes quite a bit further (albeit not for guests with proprietary OS's). Dan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On Mon, Nov 26, 2012 at 12:20:40PM -0800, Dan Magenheimer wrote:> > From: Sander Eikelenboom [mailto:linux@eikelenboom.it] > > Subject: Re: [Xen-devel] can not use all available memory > > > > Monday, November 26, 2012, 5:58:28 PM, you wrote: > > > > >> From: Ian Campbell [mailto:Ian.Campbell@citrix.com] > > >> Sent: Friday, November 23, 2012 6:34 AM > > >> To: Pasi Kärkkäinen > > >> Cc: xen-devel@lists.xensource.com; Alexander Bienzeisler > > >> Subject: Re: [Xen-devel] can not use all available memory > > >> > > >> On Fri, 2012-11-23 at 13:29 +0000, Pasi Kärkkäinen wrote: > > >> > On Fri, Nov 23, 2012 at 01:15:53PM +0000, Ian Campbell wrote: > > >> > > On Fri, 2012-11-23 at 12:55 +0000, Pasi Kärkkäinen wrote: > > >> > > > > >> > > > > This has been discussed at length on the list before, please check the > > >> > > > > archives. > > >> > > > > > > >> > > > > > >> > > > I tried googling quickly but I didn''t find anything relevant.. > > >> > > > > >> > > I found > > >> > > http://web.archiveorange.com/archive/v/zCKz5T3PLvtyZDSPQc9i > > >> > > in a matter of seconds, then: > > >> > > http://lists.xen.org/archives/html/xen-devel/2011-11/msg01415.html > > >> > > http://lists.xen.org/archives/html/xen-users/2012-05/msg00139.html > > >> > > http://lists.xen.org/archives/html/xen-users/2012-05/msg00146.html > > >> > > > > >> > > > To me this behaviour still seems wrong. What''s the point of autoballoon=1 trying to > > >> > > > balloon down dom0 if the hypervisor already has enough free memory for the VM ? > > >> > > > > > >> > > > In this case: > > >> > > > - dom0_mem=2G > > >> > > > - new VM to launch with size 16 GB. > > >> > > > - Xen has 28 GB of free memory. > > >> > > > > > >> > > > So clearly there''s no need to try to balloon down dom0.. > > >> > > > > >> > > Right, so don''t set autoballoon then. > > >> > > > > >> > > > >> > It''s enabled as a default.. so many people hit this problem. > > >> > > > >> > > > >> > > > not-yet-implemented check/feature in xl, or a bug? > > >> > > > > >> > > Neither, it is the intended behaviour of xl autoballoon, this option > > >> > > means exactly "take the required memory from dom0". > > >> > > > > >> > > > >> > http://xenbits.xen.org/docs/4.2-testing/man/xl.conf.5.html > > >> > > > >> > "autoballoon=BOOLEAN > > >> > > > >> > If disabled then xl will not attempt to reduce the amount of > > >> > memory assigned to domain 0 in order to create free memory when > > >> > starting a new domain. You should set this if you use the dom0_mem > > >> > hypervisor command line to reduce the amount of memory given to domain > > >> > 0 by default. > > >> > > > >> > Default: 1" > > >> > > > >> > > > >> > I think we should modify that to say "You should set autoballoon=0 if you use the dom0_mem > > >> hypervisor command line .." > > >> > At least I understood that text in the opposite way.. > > >> > > >> Yes, there should be a s/set/unset/ in there I think. > > >> > > >> > Also: What happens if you have autoballoon=1 and you start some VMs, then stop the VMs, > > >> > so most of the memory is now free in Xen.. > > >> > > >> xl balloons dom0 back up when it destroy domains with autoballoon=1. > > >> > > >> > and then you try to start a big VM ? > > >> > Aren''t you going to hit the same problem as in this thread? > > > > > Hmmm... this behavior and default may make sense for the Citrix > > > memory model (single machine, dom0 is "the user" so you want it > > > to always hold most of physical RAM not used by guests). But > > > probably not so for a more cloud-like memory model. > > > > > Is there any (easy) way to force autoballoon=0 if the hypervisor > > > dom0_mem boot option is specified? Or is there some reasonably > > > sane case I am missing where a user would want both dom0_mem > > > and autoballoon=1? > > > > > Oracle VM always boots servers with dom0_mem= set so (if/when > > > OVM switches to use xl), OVM will always set autoballoon off. > > > So it''s the large number of non-Citrix-non-Oracle Xen-as-a-service > > > providers I am trying to help here. > > > > > > > > Hmm i was bitten by this 2 weeks ago, i found it a bit not intuitive that: > > - While the guest i was trying to start required less memory than was freely available (according to > > xentop) outside of dom0 > > - It would fail, because xl started to try to balloon dom0 down which failed. > > Hi Sander -- > > I could be wrong (and I am confident someone will correct me if I am) but > I think this is because the Citrix memory model assumes there is an > inference-driven policy engine for load-balancing memory across competing > virtual machines ("squeezed"). I suspect squeezed returns unallocated > xen "free" memory to dom0. > > IMHO, such policy engines are good for demos and so salespeople can > say "yes, this product supports memory overcommit" but Transcendent > Memory goes quite a bit further (albeit not for guests with proprietary > OS''s). >Afaik XenServer/XCP and XenClient both use dom0_mem= option for Xen. -- Pasi
Monday, November 26, 2012, 9:20:40 PM, you wrote:>> From: Sander Eikelenboom [mailto:linux@eikelenboom.it] >> Subject: Re: [Xen-devel] can not use all available memory >> >> Monday, November 26, 2012, 5:58:28 PM, you wrote: >> >> >> From: Ian Campbell [mailto:Ian.Campbell@citrix.com] >> >> Sent: Friday, November 23, 2012 6:34 AM >> >> To: Pasi Kärkkäinen >> >> Cc: xen-devel@lists.xensource.com; Alexander Bienzeisler >> >> Subject: Re: [Xen-devel] can not use all available memory >> >> >> >> On Fri, 2012-11-23 at 13:29 +0000, Pasi Kärkkäinen wrote: >> >> > On Fri, Nov 23, 2012 at 01:15:53PM +0000, Ian Campbell wrote: >> >> > > On Fri, 2012-11-23 at 12:55 +0000, Pasi Kärkkäinen wrote: >> >> > > >> >> > > > > This has been discussed at length on the list before, please check the >> >> > > > > archives. >> >> > > > > >> >> > > > >> >> > > > I tried googling quickly but I didn't find anything relevant.. >> >> > > >> >> > > I found >> >> > > http://web.archiveorange.com/archive/v/zCKz5T3PLvtyZDSPQc9i >> >> > > in a matter of seconds, then: >> >> > > http://lists.xen.org/archives/html/xen-devel/2011-11/msg01415.html >> >> > > http://lists.xen.org/archives/html/xen-users/2012-05/msg00139.html >> >> > > http://lists.xen.org/archives/html/xen-users/2012-05/msg00146.html >> >> > > >> >> > > > To me this behaviour still seems wrong. What's the point of autoballoon=1 trying to >> >> > > > balloon down dom0 if the hypervisor already has enough free memory for the VM ? >> >> > > > >> >> > > > In this case: >> >> > > > - dom0_mem=2G >> >> > > > - new VM to launch with size 16 GB. >> >> > > > - Xen has 28 GB of free memory. >> >> > > > >> >> > > > So clearly there's no need to try to balloon down dom0.. >> >> > > >> >> > > Right, so don't set autoballoon then. >> >> > > >> >> > >> >> > It's enabled as a default.. so many people hit this problem. >> >> > >> >> > >> >> > > > not-yet-implemented check/feature in xl, or a bug? >> >> > > >> >> > > Neither, it is the intended behaviour of xl autoballoon, this option >> >> > > means exactly "take the required memory from dom0". >> >> > > >> >> > >> >> > http://xenbits.xen.org/docs/4.2-testing/man/xl.conf.5.html >> >> > >> >> > "autoballoon=BOOLEAN >> >> > >> >> > If disabled then xl will not attempt to reduce the amount of >> >> > memory assigned to domain 0 in order to create free memory when >> >> > starting a new domain. You should set this if you use the dom0_mem >> >> > hypervisor command line to reduce the amount of memory given to domain >> >> > 0 by default. >> >> > >> >> > Default: 1" >> >> > >> >> > >> >> > I think we should modify that to say "You should set autoballoon=0 if you use the dom0_mem >> >> hypervisor command line .." >> >> > At least I understood that text in the opposite way.. >> >> >> >> Yes, there should be a s/set/unset/ in there I think. >> >> >> >> > Also: What happens if you have autoballoon=1 and you start some VMs, then stop the VMs, >> >> > so most of the memory is now free in Xen.. >> >> >> >> xl balloons dom0 back up when it destroy domains with autoballoon=1. >> >> >> >> > and then you try to start a big VM ? >> >> > Aren't you going to hit the same problem as in this thread? >> >> > Hmmm... this behavior and default may make sense for the Citrix >> > memory model (single machine, dom0 is "the user" so you want it >> > to always hold most of physical RAM not used by guests). But >> > probably not so for a more cloud-like memory model. >> >> > Is there any (easy) way to force autoballoon=0 if the hypervisor >> > dom0_mem boot option is specified? Or is there some reasonably >> > sane case I am missing where a user would want both dom0_mem >> > and autoballoon=1? >> >> > Oracle VM always boots servers with dom0_mem= set so (if/when >> > OVM switches to use xl), OVM will always set autoballoon off. >> > So it's the large number of non-Citrix-non-Oracle Xen-as-a-service >> > providers I am trying to help here. >> >> >> >> Hmm i was bitten by this 2 weeks ago, i found it a bit not intuitive that: >> - While the guest i was trying to start required less memory than was freely available (according to >> xentop) outside of dom0 >> - It would fail, because xl started to try to balloon dom0 down which failed.> Hi Sander --> I could be wrong (and I am confident someone will correct me if I am) but > I think this is because the Citrix memory model assumes there is an > inference-driven policy engine for load-balancing memory across competing > virtual machines ("squeezed"). I suspect squeezed returns unallocated > xen "free" memory to dom0.> IMHO, such policy engines are good for demos and so salespeople can > say "yes, this product supports memory overcommit" but Transcendent > Memory goes quite a bit further (albeit not for guests with proprietary > OS's).> DanYes well my novice assumption was that if i have say 8GB of mem, and set dom0 max mem to say 1GB. That auto ballooning would not interfere unless the total of guests i start go across the 7GB. And it seems when i go under the, say about 1GB of free mem available outside of dom0, some ballooning code seems to interfere with creating guests. Most of the time, the first attempt to start a guests then fails, a second attempt directly thereafter succeeds. Switching off autoballoon in xl.conf makes it all work fine. Not a big problem in itself, but as Pasi said perhaps not quite clear and intuitive for everyone. So your suggestion of disabling autoballooning when dom0_mem is set seems appealing. Although there perhaps is some use case in limiting dom0 to under 4GB for some drivers, but still leave the capability to autoballoon down ? (and yes i don't like all the automatic voodoo in overcommiting memory and "hope for the best" all the algorithms involved can manage it. I like things to predictably fail (and fail direct if possible), so good measures can be taken. A bit old-fashioned perhaps. -- Sander _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
> From: Pasi Kärkkäinen [mailto:pasik@iki.fi] > Subject: Re: [Xen-devel] can not use all available memory > > On Mon, Nov 26, 2012 at 12:20:40PM -0800, Dan Magenheimer wrote: > > > From: Sander Eikelenboom [mailto:linux@eikelenboom.it] > > > Subject: Re: [Xen-devel] can not use all available memory > > > > > > Monday, November 26, 2012, 5:58:28 PM, you wrote: > > > > > > >> From: Ian Campbell [mailto:Ian.Campbell@citrix.com] > > > >> Sent: Friday, November 23, 2012 6:34 AM > > > >> To: Pasi Kärkkäinen > > > >> Cc: xen-devel@lists.xensource.com; Alexander Bienzeisler > > > >> Subject: Re: [Xen-devel] can not use all available memory > > > >> > > > >> On Fri, 2012-11-23 at 13:29 +0000, Pasi Kärkkäinen wrote: > > > >> > On Fri, Nov 23, 2012 at 01:15:53PM +0000, Ian Campbell wrote: > > > >> > > On Fri, 2012-11-23 at 12:55 +0000, Pasi Kärkkäinen wrote: > > > >> > > > > > >> > > > > This has been discussed at length on the list before, please check the > > > >> > > > > archives. > > > >> > > > > > > > >> > > > > > > >> > > > I tried googling quickly but I didn''t find anything relevant.. > > > >> > > > > > >> > > I found > > > >> > > http://web.archiveorange.com/archive/v/zCKz5T3PLvtyZDSPQc9i > > > >> > > in a matter of seconds, then: > > > >> > > http://lists.xen.org/archives/html/xen-devel/2011-11/msg01415.html > > > >> > > http://lists.xen.org/archives/html/xen-users/2012-05/msg00139.html > > > >> > > http://lists.xen.org/archives/html/xen-users/2012-05/msg00146.html > > > >> > > > > > >> > > > To me this behaviour still seems wrong. What''s the point of autoballoon=1 trying to > > > >> > > > balloon down dom0 if the hypervisor already has enough free memory for the VM ? > > > >> > > > > > > >> > > > In this case: > > > >> > > > - dom0_mem=2G > > > >> > > > - new VM to launch with size 16 GB. > > > >> > > > - Xen has 28 GB of free memory. > > > >> > > > > > > >> > > > So clearly there''s no need to try to balloon down dom0.. > > > >> > > > > > >> > > Right, so don''t set autoballoon then. > > > >> > > > > > >> > > > > >> > It''s enabled as a default.. so many people hit this problem. > > > >> > > > > >> > > > > >> > > > not-yet-implemented check/feature in xl, or a bug? > > > >> > > > > > >> > > Neither, it is the intended behaviour of xl autoballoon, this option > > > >> > > means exactly "take the required memory from dom0". > > > >> > > > > > >> > > > > >> > http://xenbits.xen.org/docs/4.2-testing/man/xl.conf.5.html > > > >> > > > > >> > "autoballoon=BOOLEAN > > > >> > > > > >> > If disabled then xl will not attempt to reduce the amount of > > > >> > memory assigned to domain 0 in order to create free memory when > > > >> > starting a new domain. You should set this if you use the dom0_mem > > > >> > hypervisor command line to reduce the amount of memory given to domain > > > >> > 0 by default. > > > >> > > > > >> > Default: 1" > > > >> > > > > >> > > > > >> > I think we should modify that to say "You should set autoballoon=0 if you use the dom0_mem > > > >> hypervisor command line .." > > > >> > At least I understood that text in the opposite way.. > > > >> > > > >> Yes, there should be a s/set/unset/ in there I think. > > > >> > > > >> > Also: What happens if you have autoballoon=1 and you start some VMs, then stop the VMs, > > > >> > so most of the memory is now free in Xen.. > > > >> > > > >> xl balloons dom0 back up when it destroy domains with autoballoon=1. > > > >> > > > >> > and then you try to start a big VM ? > > > >> > Aren''t you going to hit the same problem as in this thread? > > > > > > > Hmmm... this behavior and default may make sense for the Citrix > > > > memory model (single machine, dom0 is "the user" so you want it > > > > to always hold most of physical RAM not used by guests). But > > > > probably not so for a more cloud-like memory model. > > > > > > > Is there any (easy) way to force autoballoon=0 if the hypervisor > > > > dom0_mem boot option is specified? Or is there some reasonably > > > > sane case I am missing where a user would want both dom0_mem > > > > and autoballoon=1? > > > > > > > Oracle VM always boots servers with dom0_mem= set so (if/when > > > > OVM switches to use xl), OVM will always set autoballoon off. > > > > So it''s the large number of non-Citrix-non-Oracle Xen-as-a-service > > > > providers I am trying to help here. > > > > > > > > > > > > Hmm i was bitten by this 2 weeks ago, i found it a bit not intuitive that: > > > - While the guest i was trying to start required less memory than was freely available (according > to > > > xentop) outside of dom0 > > > - It would fail, because xl started to try to balloon dom0 down which failed. > > > > Hi Sander -- > > > > I could be wrong (and I am confident someone will correct me if I am) but > > I think this is because the Citrix memory model assumes there is an > > inference-driven policy engine for load-balancing memory across competing > > virtual machines ("squeezed"). I suspect squeezed returns unallocated > > xen "free" memory to dom0.I forgot... it is called Dynamic Memory Control (DMC), not squeezed in the XenServer product.> > IMHO, such policy engines are good for demos and so salespeople can > > say "yes, this product supports memory overcommit" but Transcendent > > Memory goes quite a bit further (albeit not for guests with proprietary > > OS''s). > > Afaik XenServer/XCP and XenClient both use dom0_mem= option for Xen.Hmmm... looks like you are right. Dan
At 12:37 -0800 on 26 Nov (1353933449), Dan Magenheimer wrote:> > > I could be wrong (and I am confident someone will correct me if I am) but > > > I think this is because the Citrix memory model assumes there is an > > > inference-driven policy engine for load-balancing memory across competing > > > virtual machines ("squeezed"). I suspect squeezed returns unallocated > > > xen "free" memory to dom0. > > I forgot... it is called Dynamic Memory Control (DMC), not squeezed > in the XenServer product.AFAIK XenServer uses dom0_mem= and doesn''t balloon com0 after boot time. The idea of ballooning all free memory into dom0 is a xl-ism, inherited from xend, and not really a "Citrix" one. It''s useful if you''ve installed xen on a machine where dom0 is otherwise your main OS, but not particularly for a dedicated platform. Tim.
> From: Tim Deegan [mailto:tim@xen.org] > Subject: Re: [Xen-devel] can not use all available memory > > At 12:37 -0800 on 26 Nov (1353933449), Dan Magenheimer wrote: > > > > I could be wrong (and I am confident someone will correct me if I am) but > > > > I think this is because the Citrix memory model assumes there is an > > > > inference-driven policy engine for load-balancing memory across competing > > > > virtual machines ("squeezed"). I suspect squeezed returns unallocated > > > > xen "free" memory to dom0. > > > > I forgot... it is called Dynamic Memory Control (DMC), not squeezed > > in the XenServer product. > > AFAIK XenServer uses dom0_mem= and doesn''t balloon com0 after boot time. > The idea of ballooning all free memory into dom0 is a xl-ism, inherited > from xend, and not really a "Citrix" one. It''s useful if you''ve > installed xen on a machine where dom0 is otherwise your main OS, but not > particularly for a dedicated platform."inherited from xend"... was the autoballoon default the same in xend? I don''t recall ever turning it off manually and, when testing tmem, I''m sure I would have had to. Or maybe xend did use hypervisor free memory before trying to autoballoon dom0?
Hi! On Mon, Nov 26, 2012 at 3:47 PM, Dan Magenheimer <dan.magenheimer@oracle.com> wrote:>> From: Tim Deegan [mailto:tim@xen.org] >> Subject: Re: [Xen-devel] can not use all available memory >> >> At 12:37 -0800 on 26 Nov (1353933449), Dan Magenheimer wrote: >> > > > I could be wrong (and I am confident someone will correct me if I am) but >> > > > I think this is because the Citrix memory model assumes there is an >> > > > inference-driven policy engine for load-balancing memory across competing >> > > > virtual machines ("squeezed"). I suspect squeezed returns unallocated >> > > > xen "free" memory to dom0. >> > >> > I forgot... it is called Dynamic Memory Control (DMC), not squeezed >> > in the XenServer product. >> >> AFAIK XenServer uses dom0_mem= and doesn''t balloon com0 after boot time. >> The idea of ballooning all free memory into dom0 is a xl-ism, inherited >> from xend, and not really a "Citrix" one. It''s useful if you''ve >> installed xen on a machine where dom0 is otherwise your main OS, but not >> particularly for a dedicated platform. > > "inherited from xend"... was the autoballoon default the same in xend? > I don''t recall ever turning it off manually and, when testing tmem, > I''m sure I would have had to. Or maybe xend did use hypervisor free > memory before trying to autoballoon dom0?My Xen 3.4 cluster at my old job used available hypervisor memory before ballooning from dom0 on the hosts which had that enabled. So, I believe the behaviour of the xl toolstack is different than that of xend. My newer Xen 4.2 cluster at my new job has autoballoon disabled across all nodes, however. But we are having some problems with running our dom0 environment without udev, which I need to figure out at some point... William
On Mon, 2012-11-26 at 21:47 +0000, Dan Magenheimer wrote:> > From: Tim Deegan [mailto:tim@xen.org] > > Subject: Re: [Xen-devel] can not use all available memory > > > > At 12:37 -0800 on 26 Nov (1353933449), Dan Magenheimer wrote: > > > > > I could be wrong (and I am confident someone will correct me if I am) but > > > > > I think this is because the Citrix memory model assumes there is an > > > > > inference-driven policy engine for load-balancing memory across competing > > > > > virtual machines ("squeezed"). I suspect squeezed returns unallocated > > > > > xen "free" memory to dom0. > > > > > > I forgot... it is called Dynamic Memory Control (DMC), not squeezed > > > in the XenServer product. > > > > AFAIK XenServer uses dom0_mem= and doesn''t balloon com0 after boot time. > > The idea of ballooning all free memory into dom0 is a xl-ism, inherited > > from xend, and not really a "Citrix" one. It''s useful if you''ve > > installed xen on a machine where dom0 is otherwise your main OS, but not > > particularly for a dedicated platform. > > "inherited from xend"... was the autoballoon default the same in xend? > I don''t recall ever turning it off manually and, when testing tmem, > I''m sure I would have had to. Or maybe xend did use hypervisor free > memory before trying to autoballoon dom0?http://wiki.xen.org/wiki/XenBestPractices#Xen_dom0_dedicated_memory_and_preventing_dom0_memory_ballooning indicates that you should have been using "(enable-dom0-ballooning no)" when using dom0_mem= with xend. Perhaps the failure case is not so bad with xend as with xl though, perhaps because xend has a central daemon which lets it make different choices. Ian.