I have been playing with memory= and maxmem=, but I''m not seeing what I think I should be seeing, namely that the memory available to a domU can be increased on the fly. For example, I have: memory = 256 maxmem = 512 in my config file. I can run ''xm mem-set mydomain 128'', and the memory in the domU will drop to 128. I can run ''xm mem-set mydomain 256'' and bring it back up to its original memory level. But if I run ''xm mem-set mydomain 512'', then ''xm list'' will see 512M allocated to the domain, but the domain will only see 256M. However, if I ''reboot'' the domU, when it comes back up, it will have 512M of memory. According to Jayson Vantuyl in a discussion on this list a few days ago: "For example, if you have these options for a domain: memory=1024 maxmem=4096 You will have the kernel boot up with 1GB of memory. At any point you can increase it to, for example, 2 GB with this command: xm mem-set <domain> 2048 The kernel will automagically increase to 2GB of RAM. This is, as you might imagine, extremely cool." Is this true? Is it possible to set the initial memory level to be something less than maxmem, and have the running (linux) domU see the new memory level when it is increased? If so, then is there something I''m missing in my setup? A driver in dom0 or domU, perhaps? Paul Archer -------------------------------------------------------------- "I''ll say this about Linux: it''s the first time I''ve seen Unix on the right platform."--Steve Ballmer, president of Microsoft (NB: Microsoft used to own SCO, which did, and still does, produce a Unix for the Intel platform.) -------------------------------------------------------------- _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Sat, 19 May 2007 23:34:38 -0500 (CDT) Paul Archer <tigger@io.com> wrote:> I have been playing with memory= and maxmem=, but I''m not seeing what > I think I should be seeing, namely that the memory available to a > domU can be increased on the fly. > > For example, I have: > memory = 256 > maxmem = 512 > in my config file. > > I can run ''xm mem-set mydomain 128'', and the memory in the domU will > drop to 128. I can run ''xm mem-set mydomain 256'' and bring it back up > to its original memory level. But if I run ''xm mem-set mydomain 512'', > then ''xm list'' will see 512M allocated to the domain, but the domain > will only see 256M. However, if I ''reboot'' the domU, when it comes > back up, it will have 512M of memory.I''m in the same boat here.> According to Jayson Vantuyl in a discussion on this list a few days > ago: "For example, if you have these options for a domain: > > memory=1024 > maxmem=4096 > > You will have the kernel boot up with 1GB of memory. At any point > you can increase it to, for example, 2 GB with this command: > > xm mem-set <domain> 2048 > > The kernel will automagically increase to 2GB of RAM. This is, as > you might imagine, extremely cool." > > Is this true? Is it possible to set the initial memory level to be > something less than maxmem, and have the running (linux) domU see the > new memory level when it is increased? If so, then is there something > I''m missing in my setup? A driver in dom0 or domU, perhaps?I wasn''t able to achieve this using xen-3.1. The maximum amount of RAM appears to be dictated by the value that the machine is booted with. Has anyone else managed to increase the amount of memory allocated to a domU to more than what the domU was booted with and have the domU utilise this memory? I.e. show up with free -m on the domU? Cheers, Brad _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi everyone , I have also been facing the same problem for the past few days. The max-mem parameter in the domU config file is completely ignored.Even if we specifiy maxmem parameter in the config file at boot time , boot time allocated memory is taken as the max memory possible for the domain. Also i observed that the max vcpu parameter also behaves the same way. VCPU Observations: 1]We can play around with only the no of VCPUs allotted at boot time or less. Allocating max no of VCPUs dynamically is not reflected. i.e we can make max VCPUs of a domain from 5 to 10 dynamically BUT the 5 new ones are never used...they are always paused (possibly a bug in open source xen ?) Is there a fix around these problems? Thanks..... Suyash> Paul Archer <tigger@io.com> wrote: > > > I have been playing with memory= and maxmem=, but I''m not seeing what > > I think I should be seeing, namely that the memory available to a > > domU can be increased on the fly. > > > >Brad wrote :> >I wasn''t able to achieve this using xen-3.1. The maximum amount of RAM > appears to be dictated by the value that >the machine is booted with. Has > anyone else managed to increase the amount of memory allocated to a domU to > >more than what the domU was booted with and have the domU utilise this > memory? I.e. show up with free -m on the >domU?_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
After doing some more research, I finally found out what''s going on. In order to allocate more memory to a domU, it has to be freed from somewhere else first. In other words, if memory=256 and maxmem=512, then before you run ''xm mem-set mydomU 512'', you have to run ''xm mem-set 0 <current memory for dom0 - 256>''. I suppose this would work just as well by taking the memory from another domU (or some combination of domU''s and dom0), but I haven''t tried that. I haven''t played with VCPU''s, but it probably works on the same principle. Paul Tomorrow, suyash jape wrote:> Hi everyone , I have also been facing the same problem for the past few > days. The max-mem parameter in the domU config file is completely > ignored.Even if we specifiy maxmem parameter in the config file at boot time > , boot time allocated memory is taken as the max memory possible for the > domain. > Also i observed that the max vcpu parameter also behaves the same > way. > VCPU Observations: > 1]We can play around with only the no of VCPUs allotted at boot time or > less. > > Allocating max no of VCPUs dynamically is not reflected. i.e we can make > max VCPUs of a domain from 5 to 10 dynamically BUT the 5 new ones are never > used...they are always paused (possibly a bug in open source xen ?) > > Is there a fix around these problems? > > Thanks..... > > Suyash > > > >> Paul Archer <tigger@io.com> wrote: >> >> > I have been playing with memory= and maxmem=, but I''m not seeing what >> > I think I should be seeing, namely that the memory available to a >> > domU can be increased on the fly. >> > >> > > > > Brad wrote : >> >I wasn''t able to achieve this using xen-3.1. The maximum amount of RAM >> appears to be dictated by the value that >the machine is booted with. Has >> anyone else managed to increase the amount of memory allocated to a domU to >> >more than what the domU was booted with and have the domU utilise this >> memory? I.e. show up with free -m on the >domU? >--------Brady''s First Law of Problem Solving:------- When confronted by a difficult problem, you can solve it more easily by reducing it to the question, "How would the Lone Ranger have handled this?" ---------------------------------------------------- _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi Paul, On Mon, 21 May 2007 21:56:22 -0500 (CDT) Paul Archer <tigger@io.com> wrote:> After doing some more research, I finally found out what''s going on. > In order to allocate more memory to a domU, it has to be freed from > somewhere else first. In other words, if memory=256 and maxmem=512, > then before you run ''xm mem-set mydomU 512'', you have to run ''xm > mem-set 0 <current memory for dom0 - 256>''. > I suppose this would work just as well by taking the memory from > another domU (or some combination of domU''s and dom0), but I haven''t > tried that.I don''t think this is the issue here since the dom0 memory usage will balloon down automatically to free up more RAM for the domU. When I attempted to get this to work yesterday, dom0 had over 2GB of RAM allocated to it, so it had heaps to spare when it ballooned down. The problem isn''t with trying to allocate memory to a domU, it is with trying to get domU to use or recognise the extra memory. I was able to allocate the extra memory ok, but running free -m in the domU would not show the extra memory as being available. Cheers, Brad _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
The parameter is maxmem, not max-mem. That could be part of it. You can get a complete list of options from "xm create --help_config". When I found this out it was a huge help for me. Also, it is important that you NOT specify mem=XXX on the kernel commandline (via the extra option). That would completely break this behavior. In other works, DO NOT specify anything like ''extra="mem=512"''. One other thing I''ve done that may affect this is that I have limited my Dom0 memory at boot time. Perhaps if it has to borrow the memory from the Dom0 it stops short. I do this by passing the hypervisor the parameter dom0_mem=524288. Here''s the relevant section of my grub.conf: root (hd0,0) kernel /boot/xen.gz dom0_mem=524288 com1=38400,8n1 nmi=dom0 module /boot/vmlinuz-2.6.16.38-xen0 root=/dev/uba1 netloop.nloopbacks=2 xencons=ttyS libusual.bias="ub" And here''s a lightly modified copy of the .xen file that I just tested: # -*- mode: python; -*- kernel = ''/boot/vmlinuz-2.6.16.38-xenU'' memory = 2000 maxmem = 4096 name = ''XXX'' vif = [ ''bridge=xenbr0'' ] disk = [ ''phy:/dev/XXX0,sda1,w'', ''phy:/dev/XXX1,sda2,w'', ''phy:/dev/XXX2,sdb1,w!'', ] root = ''/dev/sda1 ro'' vcpus = 2 Here''s the output of a session. The dates are to help match up with the following screenshots: hostname ~ # date ; xm info | grep free Mon May 21 20:10:21 PDT 2007 free_memory : 307 hostname ~ # xm mem-set XXX 2250 ; date Mon May 21 20:12:13 PDT 2007 hostname ~ # xm mem-set XXX 2000 ; date Mon May 21 20:12:43 PDT 2007 hostname ~ # Here are screenshots of before adjusting the memory, after upping it, and after dropping it again: http://picasaweb.google.com/jvantuyl/XenDebuggingScreenShots/ photo#5067220233699865074 http://picasaweb.google.com/jvantuyl/XenDebuggingScreenShots/ photo#5067220233699865090 http://picasaweb.google.com/jvantuyl/XenDebuggingScreenShots/ photo#5067220237994832402 Here is uname -a from the DomU: Linux XXX 2.6.16.38-xenU #1 SMP Mon Mar 26 03:15:23 PDT 2007 x86_64 Dual Core AMD Opteron(tm) Processor 280 GNU/Linux And xm info on the Dom0: host : XXW release : 2.6.16.38-xen0 version : #4 SMP Mon Mar 26 22:56:07 PDT 2007 machine : x86_64 nr_cpus : 4 nr_nodes : 1 sockets_per_node : 2 cores_per_socket : 2 threads_per_core : 1 cpu_mhz : 2394 hw_caps : 178bfbff:e3d3fbff: 00000000:00000010:00000001:00000000:00000003 total_memory : 16319 free_memory : 307 xen_major : 3 xen_minor : 0 xen_extra : .4-1 xen_caps : xen-3.0-x86_64 xen_pagesize : 4096 platform_params : virt_start=0xffff800000000000 xen_changeset : Thu Feb 15 11:34:58 2007 +0000 13139:3341afbb1953 cc_compiler : gcc version 4.1.1 (Gentoo 4.1.1-r1) cc_compile_by : root cc_compile_domain : XXW.com cc_compile_date : Mon Mar 26 02:54:15 PDT 2007 xend_config_format : 3 Hopefully some of this will help somebody...but it really does work. I swear! On May 21, 2007, at 6:10 PM, suyash jape wrote:> Hi everyone , I have also been facing the same problem for the > past few days. The max-mem parameter in the domU config file is > completely ignored.Even if we specifiy maxmem parameter in the > config file at boot time , boot time allocated memory is taken as > the max memory possible for the domain. > Also i observed that the max vcpu parameter also behaves > the same way. > VCPU Observations: > 1]We can play around with only the no of VCPUs allotted at boot > time or less. > Allocating max no of VCPUs dynamically is not reflected. i.e we > can make max VCPUs of a domain from 5 to 10 dynamically BUT the 5 > new ones are never used...they are always paused (possibly a bug in > open source xen ?) > > Is there a fix around these problems? > > Thanks..... > > Suyash > > > > Paul Archer <tigger@io.com> wrote: > > > I have been playing with memory= and maxmem=, but I''m not seeing > what > > I think I should be seeing, namely that the memory available to a > > domU can be increased on the fly. > > > > > > Brad wrote : > >I wasn''t able to achieve this using xen-3.1. The maximum amount of > RAM appears to be dictated by the value that >the machine is booted > with. Has anyone else managed to increase the amount of memory > allocated to a domU to >more than what the domU was booted with and > have the domU utilise this memory? I.e. show up with free -m on the > >domU? > > > > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users-- Jayson Vantuyl Systems Architect Engine Yard jvantuyl@engineyard.com _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Tue, 22 May 2007 13:26:09 +1000 Brad Plant <bplant@iinet.net.au> wrote:> On Mon, 21 May 2007 21:56:22 -0500 (CDT) > Paul Archer <tigger@io.com> wrote: > > > After doing some more research, I finally found out what''s going on. > > In order to allocate more memory to a domU, it has to be freed from > > somewhere else first. In other words, if memory=256 and maxmem=512, > > then before you run ''xm mem-set mydomU 512'', you have to run ''xm > > mem-set 0 <current memory for dom0 - 256>''. > > I suppose this would work just as well by taking the memory from > > another domU (or some combination of domU''s and dom0), but I haven''t > > tried that. > > I don''t think this is the issue here since the dom0 memory usage will > balloon down automatically to free up more RAM for the domU. When I > attempted to get this to work yesterday, dom0 had over 2GB of RAM > allocated to it, so it had heaps to spare when it ballooned down. > > The problem isn''t with trying to allocate memory to a domU, it is > with trying to get domU to use or recognise the extra memory. I was > able to allocate the extra memory ok, but running free -m in the domU > would not show the extra memory as being available.Sorry Paul, you were right. It seems that dom0 didn''t automatically balloon down to free up the extra ram. I am now passing the dom0_mem variable to the hypervisor so that I don''t have to worry about reducing its usage first. Cheers, Brad _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Thanks for the very informative post Jayson, All is sweet on my end now. Cheers, Brad P.S. Good to see some other Gentoo/Xen users out there On Mon, 21 May 2007 20:33:19 -0700 Jayson Vantuyl <jvantuyl@engineyard.com> wrote:> The parameter is maxmem, not max-mem. That could be part of it. > You can get a complete list of options from "xm create > --help_config". When I found this out it was a huge help for me. > > Also, it is important that you NOT specify mem=XXX on the kernel > commandline (via the extra option). That would completely break > this behavior. In other works, DO NOT specify anything like > ''extra="mem=512"''. > > One other thing I''ve done that may affect this is that I have > limited my Dom0 memory at boot time. Perhaps if it has to borrow the > memory from the Dom0 it stops short. I do this by passing the > hypervisor the parameter dom0_mem=524288. > > Here''s the relevant section of my grub.conf: > > root (hd0,0) > kernel /boot/xen.gz dom0_mem=524288 com1=38400,8n1 nmi=dom0 > module /boot/vmlinuz-2.6.16.38-xen0 root=/dev/uba1 > netloop.nloopbacks=2 xencons=ttyS libusual.bias="ub" > > And here''s a lightly modified copy of the .xen file that I just > tested: > > # -*- mode: python; -*- > kernel = ''/boot/vmlinuz-2.6.16.38-xenU'' > memory = 2000 > maxmem = 4096 > name = ''XXX'' > vif = [ ''bridge=xenbr0'' ] > disk = [ > ''phy:/dev/XXX0,sda1,w'', > ''phy:/dev/XXX1,sda2,w'', > ''phy:/dev/XXX2,sdb1,w!'', > ] > root = ''/dev/sda1 ro'' > vcpus = 2 > > Here''s the output of a session. The dates are to help match up with > the following screenshots: > > hostname ~ # date ; xm info | grep free > Mon May 21 20:10:21 PDT 2007 > free_memory : 307 > hostname ~ # xm mem-set XXX 2250 ; date > Mon May 21 20:12:13 PDT 2007 > hostname ~ # xm mem-set XXX 2000 ; date > Mon May 21 20:12:43 PDT 2007 > hostname ~ # > > Here are screenshots of before adjusting the memory, after upping > it, and after dropping it again: > > http://picasaweb.google.com/jvantuyl/XenDebuggingScreenShots/ > photo#5067220233699865074 > http://picasaweb.google.com/jvantuyl/XenDebuggingScreenShots/ > photo#5067220233699865090 > http://picasaweb.google.com/jvantuyl/XenDebuggingScreenShots/ > photo#5067220237994832402 > > Here is uname -a from the DomU: > > Linux XXX 2.6.16.38-xenU #1 SMP Mon Mar 26 03:15:23 PDT 2007 x86_64 > Dual Core AMD Opteron(tm) Processor 280 GNU/Linux > > And xm info on the Dom0: > > host : XXW > release : 2.6.16.38-xen0 > version : #4 SMP Mon Mar 26 22:56:07 PDT 2007 > machine : x86_64 > nr_cpus : 4 > nr_nodes : 1 > sockets_per_node : 2 > cores_per_socket : 2 > threads_per_core : 1 > cpu_mhz : 2394 > hw_caps : 178bfbff:e3d3fbff: > 00000000:00000010:00000001:00000000:00000003 > total_memory : 16319 > free_memory : 307 > xen_major : 3 > xen_minor : 0 > xen_extra : .4-1 > xen_caps : xen-3.0-x86_64 > xen_pagesize : 4096 > platform_params : virt_start=0xffff800000000000 > xen_changeset : Thu Feb 15 11:34:58 2007 +0000 > 13139:3341afbb1953 > cc_compiler : gcc version 4.1.1 (Gentoo 4.1.1-r1) > cc_compile_by : root > cc_compile_domain : XXW.com > cc_compile_date : Mon Mar 26 02:54:15 PDT 2007 > xend_config_format : 3 > > Hopefully some of this will help somebody...but it really does > work. I swear! > > On May 21, 2007, at 6:10 PM, suyash jape wrote: > > > Hi everyone , I have also been facing the same problem for the > > past few days. The max-mem parameter in the domU config file is > > completely ignored.Even if we specifiy maxmem parameter in the > > config file at boot time , boot time allocated memory is taken as > > the max memory possible for the domain. > > Also i observed that the max vcpu parameter also > > behaves the same way. > > VCPU Observations: > > 1]We can play around with only the no of VCPUs allotted at boot > > time or less. > > Allocating max no of VCPUs dynamically is not reflected. i.e we > > can make max VCPUs of a domain from 5 to 10 dynamically BUT the 5 > > new ones are never used...they are always paused (possibly a bug > > in open source xen ?) > > > > Is there a fix around these problems? > > > > Thanks..... > > > > Suyash > > > > > > > > Paul Archer <tigger@io.com> wrote: > > > > > I have been playing with memory= and maxmem=, but I''m not seeing > > what > > > I think I should be seeing, namely that the memory available to a > > > domU can be increased on the fly. > > > > > > > > > > Brad wrote : > > >I wasn''t able to achieve this using xen-3.1. The maximum amount > > >of > > RAM appears to be dictated by the value that >the machine is > > booted with. Has anyone else managed to increase the amount of > > memory allocated to a domU to >more than what the domU was booted > > with and have the domU utilise this memory? I.e. show up with free > > -m on the > > >domU? > > > > > > > > > > > > _______________________________________________ > > Xen-users mailing list > > Xen-users@lists.xensource.com > > http://lists.xensource.com/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> On Tue, 22 May 2007 13:26:09 +1000 > Brad Plant <bplant@iinet.net.au> wrote: > > > On Mon, 21 May 2007 21:56:22 -0500 (CDT) > > Paul Archer <tigger@io.com> wrote: > > > > > After doing some more research, I finally found out what''s going on. > > > In order to allocate more memory to a domU, it has to be freed from > > > somewhere else first. In other words, if memory=256 and maxmem=512, > > > then before you run ''xm mem-set mydomU 512'', you have to run ''xm > > > mem-set 0 <current memory for dom0 - 256>''. > > > I suppose this would work just as well by taking the memory from > > > another domU (or some combination of domU''s and dom0), but I haven''t > > > tried that. > > > > I don''t think this is the issue here since the dom0 memory usage will > > balloon down automatically to free up more RAM for the domU. When I > > attempted to get this to work yesterday, dom0 had over 2GB of RAM > > allocated to it, so it had heaps to spare when it ballooned down. > > > > The problem isn''t with trying to allocate memory to a domU, it is > > with trying to get domU to use or recognise the extra memory. I was > > able to allocate the extra memory ok, but running free -m in the domU > > would not show the extra memory as being available. > > Sorry Paul, you were right. It seems that dom0 didn''t automatically > balloon down to free up the extra ram. I am now passing the dom0_mem > variable to the hypervisor so that I don''t have to worry about reducing > its usage first. >No problem. I thought this was a bug at first, but I realized that it makes some sense, because you might want to pull memory from one domU to give it to another, so you have to do the allocation yourself. What is messed up is the reporting. ''xm list'' doesn''t necessarily reflect reality: tiva:/etc/xen # xm mem-set 0 999999 tiva:/etc/xen # xm list Name ID Mem VCPUs State Time(s) Domain-0 0 999999 1 r----- 12119.9 c5 10 256 1 -b---- 2.6 centos5 11 256 1 -b---- 2.5 tiva:/etc/xen # cat /proc/meminfo |head MemTotal: 1545148 kB MemFree: 276396 kB Buffers: 35080 kB Cached: 599996 kB SwapCached: 0 kB Active: 549724 kB Inactive: 530676 kB HighTotal: 1203788 kB HighFree: 34820 kB LowTotal: 341360 kB Paul _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users