Jia Rao
2009-Nov-10 16:23 UTC
[Xen-devel] How does the self-ballooning daemon calculate the selftarget?
Hi all, I am wondering how does the self-ballooning daemon calculate the selftarget for the guest OS.>From the README of balloon driver, it seems that selftarget is determined byCommitted_AS in meminfo. I did some test on tpcc benchmark. the OS is initially allocated 2048MB of memory, and then I enabled the selfballooning by echo 1 > /proc/xen/balloon. The configuration of xenballoond.conf is the default except that I enabled self-balloon. The following is the meminfo MemTotal: 1939996 kB MemFree: 1183888 kB Buffers: 13260 kB Cached: 396492 kB SwapCached: 0 kB Active: 386912 kB Inactive: 213584 kB HighTotal: 0 kB HighFree: 0 kB LowTotal: 1939996 kB LowFree: 1183888 kB SwapTotal: 2097136 kB SwapFree: 2097132 kB Dirty: 78432 kB Writeback: 0 kB AnonPages: 190668 kB Mapped: 14228 kB Slab: 53148 kB PageTables: 4836 kB NFS_Unstable: 0 kB Bounce: 0 kB CommitLimit: 3067132 kB Committed_AS: 524484 kB VmallocTotal: 34359738367 kB VmallocUsed: 1208 kB VmallocChunk: 34359737159 kB It seems that the self-balloon driver is reluctant to give up free memory from the guest although the Committed_AS is small. The self-ballooning did changed the selftarget to something like 1.6GB but quickly changed to and fix at 193xxxxKB. According to the meminfo, we should be able to give up at least 800MB back to the VMM. FYI: xen 3.3.1, linux 2.6.18.8 in both dom0 and domu. Any ideas? Thanks, Jia _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Dan Magenheimer
2009-Nov-10 17:43 UTC
RE: [Xen-devel] How does the self-ballooning daemon calculate the selftarget?
Hi Jia -- There are changes in newer versions of the balloon driver to ensure that ballooning does not take too much memory away from the kernel, which would result in kernel crashes. These changes set a different minimum value depending on how much memory the kernel originally discovered. If the self-ballooning scripts run with an "old" balloon driver and one changes certain parameters in the scripts, kernel crashes might occur. So the self-ballooning scripts implement a minimum memory algorithm similar to the newer balloon driver to avoid this situation. Look for XENBALLOON_MINMEM in the xenballoon config file if you want to change this. Dan -----Original Message----- From: Jia Rao [mailto:rickenrao@gmail.com] Sent: Tuesday, November 10, 2009 9:23 AM To: xen-devel@lists.xensource.com Subject: [Xen-devel] How does the self-ballooning daemon calculate the selftarget? Hi all, I am wondering how does the self-ballooning daemon calculate the selftarget for the guest OS.>From the README of balloon driver, it seems that selftarget is determined by Committed_AS in meminfo.I did some test on tpcc benchmark. the OS is initially allocated 2048MB of memory, and then I enabled the selfballooning by echo 1 > /proc/xen/balloon. The configuration of xenballoond.conf is the default except that I enabled self-balloon. The following is the meminfo MemTotal: 1939996 kB MemFree: 1183888 kB Buffers: 13260 kB Cached: 396492 kB SwapCached: 0 kB Active: 386912 kB Inactive: 213584 kB HighTotal: 0 kB HighFree: 0 kB LowTotal: 1939996 kB LowFree: 1183888 kB SwapTotal: 2097136 kB SwapFree: 2097132 kB Dirty: 78432 kB Writeback: 0 kB AnonPages: 190668 kB Mapped: 14228 kB Slab: 53148 kB PageTables: 4836 kB NFS_Unstable: 0 kB Bounce: 0 kB CommitLimit: 3067132 kB Committed_AS: 524484 kB VmallocTotal: 34359738367 kB VmallocUsed: 1208 kB VmallocChunk: 34359737159 kB It seems that the self-balloon driver is reluctant to give up free memory from the guest although the Committed_AS is small. The self-ballooning did changed the selftarget to something like 1.6GB but quickly changed to and fix at 193xxxxKB. According to the meminfo, we should be able to give up at least 800MB back to the VMM. FYI: xen 3.3.1, linux 2.6.18.8 in both dom0 and domu. Any ideas? Thanks, Jia _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jia Rao
2009-Nov-10 19:18 UTC
Re: [Xen-devel] How does the self-ballooning daemon calculate the selftarget?
Hi Dan, Thanks for the reply. The XENBALLOON_MINMEM was set to 0 in my case. I am wondering if the /proc/meminfo is a reliable source for memory utilization of a guest. In my case, the guest stays in idle for several hours, the metrics in /proc/meminfo is seldom changed. For example, the Committed_AS and Active metrics stay almost the same for several hours. In earlier kernels, the Active and Inactive metrics are periodically updated by a kernel thread kscand, which seems to has been removed from the 2.6 kernel.>From what I have observed, the meminfo is correctly updated if the memorydemand from the guest is increasing. However, once the memory demand drops or be almost zero (idle), the meminfo is not periodically updated. This might be a problem for the self-balloon daemon to deflate the balloon. Thanks, Jia On Tue, Nov 10, 2009 at 12:43 PM, Dan Magenheimer < dan.magenheimer@oracle.com> wrote:> Hi Jia -- > > There are changes in newer versions of the balloon driver to > ensure that ballooning does not take too much memory away > from the kernel, which would result in kernel crashes. > These changes set a different minimum value depending on > how much memory the kernel originally discovered. > > If the self-ballooning scripts run with an "old" balloon > driver and one changes certain parameters > in the scripts, kernel crashes might occur. So the > self-ballooning scripts implement a minimum memory > algorithm similar to the newer balloon driver to > avoid this situation. > > Look for XENBALLOON_MINMEM in the xenballoon config file > if you want to change this. > > Dan > > -----Original Message----- > From: Jia Rao [mailto:rickenrao@gmail.com] > Sent: Tuesday, November 10, 2009 9:23 AM > To: xen-devel@lists.xensource.com > Subject: [Xen-devel] How does the self-ballooning daemon calculate the > selftarget? > > > Hi all, > > I am wondering how does the self-ballooning daemon calculate the selftarget > for the guest OS. > From the README of balloon driver, it seems that selftarget is determined > by Committed_AS in meminfo. > > I did some test on tpcc benchmark. the OS is initially allocated 2048MB of > memory, and then I enabled the selfballooning by echo 1 > /proc/xen/balloon. > The configuration of xenballoond.conf is the default except that I enabled > self-balloon. > > The following is the meminfo > > MemTotal: 1939996 kB > MemFree: 1183888 kB > Buffers: 13260 kB > Cached: 396492 kB > SwapCached: 0 kB > Active: 386912 kB > Inactive: 213584 kB > HighTotal: 0 kB > HighFree: 0 kB > LowTotal: 1939996 kB > LowFree: 1183888 kB > SwapTotal: 2097136 kB > SwapFree: 2097132 kB > Dirty: 78432 kB > Writeback: 0 kB > AnonPages: 190668 kB > Mapped: 14228 kB > Slab: 53148 kB > PageTables: 4836 kB > NFS_Unstable: 0 kB > Bounce: 0 kB > CommitLimit: 3067132 kB > Committed_AS: 524484 kB > VmallocTotal: 34359738367 kB > VmallocUsed: 1208 kB > VmallocChunk: 34359737159 kB > > It seems that the self-balloon driver is reluctant to give up free memory > from the guest although the Committed_AS is small. The self-ballooning did > changed the selftarget to something like 1.6GB but quickly changed to and > fix at 193xxxxKB. > > According to the meminfo, we should be able to give up at least 800MB back > to the VMM. > FYI: xen 3.3.1, linux 2.6.18.8 in both dom0 and domu. > > Any ideas? > > Thanks, > Jia >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Dan Magenheimer
2009-Nov-10 20:45 UTC
RE: [Xen-devel] How does the self-ballooning daemon calculate the selftarget?
Hi Jia -- First, please note that when XENBALLOON_MINMEM is set to 0, that does NOT mean 0MB. As described in the text in the xenballoon config file, XENBALLOON_MINMEM==0 means that the heuristic is used which is in my previous reply. As for Committed_AS, it is certainly not a perfect metric. It was just an adequate one (and much better than no metric at all). I''m not sure, but your observation of meminfo not being periodically updated may be a result of a balloon driver bug that was fixed in a patch by Ian Campbell here: http://xenbits.xensource.com/linux-2.6.18-xen.hg?rev/329ea0ccb344 If you are interested in the area of memory utilization optimization, I''d encourage you to try tmem, which is in xen-unstable and the Oracle VM 2.2 product. (See http://oss.oracle.com/projects/tmem for more info.) Dan P.S. The minimum memory heuristic patch for the balloon driver is here: http://xenbits.xensource.com/linux-2.6.18-xen.hg?rev/b61443b1bf76 -----Original Message----- From: Jia Rao [mailto:rickenrao@gmail.com] Sent: Tuesday, November 10, 2009 12:19 PM To: Dan Magenheimer Cc: xen-devel@lists.xensource.com Subject: Re: [Xen-devel] How does the self-ballooning daemon calculate the selftarget? Hi Dan, Thanks for the reply. The XENBALLOON_MINMEM was set to 0 in my case. I am wondering if the /proc/meminfo is a reliable source for memory utilization of a guest. In my case, the guest stays in idle for several hours, the metrics in /proc/meminfo is seldom changed. For example, the Committed_AS and Active metrics stay almost the same for several hours. In earlier kernels, the Active and Inactive metrics are periodically updated by a kernel thread kscand, which seems to has been removed from the 2.6 kernel. From what I have observed, the meminfo is correctly updated if the memory demand from the guest is increasing. However, once the memory demand drops or be almost zero (idle), the meminfo is not periodically updated. This might be a problem for the self-balloon daemon to deflate the balloon. Thanks, Jia On Tue, Nov 10, 2009 at 12:43 PM, Dan Magenheimer <dan.magenheimer@oracle.com> wrote: Hi Jia -- There are changes in newer versions of the balloon driver to ensure that ballooning does not take too much memory away from the kernel, which would result in kernel crashes. These changes set a different minimum value depending on how much memory the kernel originally discovered. If the self-ballooning scripts run with an "old" balloon driver and one changes certain parameters in the scripts, kernel crashes might occur. So the self-ballooning scripts implement a minimum memory algorithm similar to the newer balloon driver to avoid this situation. Look for XENBALLOON_MINMEM in the xenballoon config file if you want to change this. Dan -----Original Message----- From: Jia Rao [mailto:rickenrao@gmail.com] Sent: Tuesday, November 10, 2009 9:23 AM To: xen-devel@lists.xensource.com Subject: [Xen-devel] How does the self-ballooning daemon calculate the selftarget? Hi all, I am wondering how does the self-ballooning daemon calculate the selftarget for the guest OS. >From the README of balloon driver, it seems that selftarget is determined by Committed_AS in meminfo. I did some test on tpcc benchmark. the OS is initially allocated 2048MB of memory, and then I enabled the selfballooning by echo 1 > /proc/xen/balloon. The configuration of xenballoond.conf is the default except that I enabled self-balloon. The following is the meminfo MemTotal: 1939996 kB MemFree: 1183888 kB Buffers: 13260 kB Cached: 396492 kB SwapCached: 0 kB Active: 386912 kB Inactive: 213584 kB HighTotal: 0 kB HighFree: 0 kB LowTotal: 1939996 kB LowFree: 1183888 kB SwapTotal: 2097136 kB SwapFree: 2097132 kB Dirty: 78432 kB Writeback: 0 kB AnonPages: 190668 kB Mapped: 14228 kB Slab: 53148 kB PageTables: 4836 kB NFS_Unstable: 0 kB Bounce: 0 kB CommitLimit: 3067132 kB Committed_AS: 524484 kB VmallocTotal: 34359738367 kB VmallocUsed: 1208 kB VmallocChunk: 34359737159 kB It seems that the self-balloon driver is reluctant to give up free memory from the guest although the Committed_AS is small. The self-ballooning did changed the selftarget to something like 1.6GB but quickly changed to and fix at 193xxxxKB. According to the meminfo, we should be able to give up at least 800MB back to the VMM. FYI: xen 3.3.1, linux 2.6.18.8 in both dom0 and domu. Any ideas? Thanks, Jia _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Jia Rao
2009-Nov-11 01:21 UTC
Re: [Xen-devel] How does the self-ballooning daemon calculate the selftarget?
Hi Dan, That is because the bug you mentioned here. I ran my guest under my friend''s xen 3.4 which has a updated 2.6.18.8 dom0, the self ballooning worked fine (I didnt patch the guest kernel). I am going to patch the fix by Ian Campbell in my xen 3.3 machines. Do I need to patch the guest kernel too? or just patch the dom0 kernel? Thanks, Jia On Tue, Nov 10, 2009 at 3:45 PM, Dan Magenheimer <dan.magenheimer@oracle.com> wrote:> Hi Jia -- > > First, please note that when XENBALLOON_MINMEM is set to 0, that does NOT > mean 0MB. As described in the text in the xenballoon config > file, XENBALLOON_MINMEM==0 means that the heuristic is used which is in my > previous reply. > > As for Committed_AS, it is certainly not a perfect metric. It was just an > adequate one (and much better than no metric at all). > > I''m not sure, but your observation of meminfo not being periodically > updated may be a result of a balloon driver bug that was fixed in a patch by > Ian Campbell here: > > http://xenbits.xensource.com/linux-2.6.18-xen.hg?rev/329ea0ccb344 > > If you are interested in the area of memory utilization optimization, I''d > encourage you to try tmem, which is in xen-unstable and the Oracle VM 2.2 > product. (See http://oss.oracle.com/projects/tmem for more info.) > > Dan > > P.S. The minimum memory heuristic patch for the balloon driver is here: > > http://xenbits.xensource.com/linux-2.6.18-xen.hg?rev/b61443b1bf76 > > -----Original Message----- > *From:* Jia Rao [mailto:rickenrao@gmail.com] > *Sent:* Tuesday, November 10, 2009 12:19 PM > *To:* Dan Magenheimer > *Cc:* xen-devel@lists.xensource.com > *Subject:* Re: [Xen-devel] How does the self-ballooning daemon calculate > the selftarget? > > Hi Dan, > > Thanks for the reply. > The XENBALLOON_MINMEM was set to 0 in my case. > > I am wondering if the /proc/meminfo is a reliable source for memory > utilization of a guest. In my case, the guest stays in idle for several > hours, the metrics in /proc/meminfo is seldom changed. > For example, the Committed_AS and Active metrics stay almost the same for > several hours. > > In earlier kernels, the Active and Inactive metrics are periodically > updated by a kernel thread kscand, which seems to has been removed from the > 2.6 kernel. > > From what I have observed, the meminfo is correctly updated if the memory > demand from the guest is increasing. However, once the memory demand drops > or be almost zero (idle), the meminfo is not periodically updated. This > might be a problem for the self-balloon daemon to deflate the balloon. > > Thanks, > Jia > > On Tue, Nov 10, 2009 at 12:43 PM, Dan Magenheimer < > dan.magenheimer@oracle.com> wrote: > >> Hi Jia -- >> >> There are changes in newer versions of the balloon driver to >> ensure that ballooning does not take too much memory away >> from the kernel, which would result in kernel crashes. >> These changes set a different minimum value depending on >> how much memory the kernel originally discovered. >> >> If the self-ballooning scripts run with an "old" balloon >> driver and one changes certain parameters >> in the scripts, kernel crashes might occur. So the >> self-ballooning scripts implement a minimum memory >> algorithm similar to the newer balloon driver to >> avoid this situation. >> >> Look for XENBALLOON_MINMEM in the xenballoon config file >> if you want to change this. >> >> Dan >> >> -----Original Message----- >> From: Jia Rao [mailto:rickenrao@gmail.com] >> Sent: Tuesday, November 10, 2009 9:23 AM >> To: xen-devel@lists.xensource.com >> Subject: [Xen-devel] How does the self-ballooning daemon calculate the >> selftarget? >> >> >> Hi all, >> >> I am wondering how does the self-ballooning daemon calculate the >> selftarget for the guest OS. >> >From the README of balloon driver, it seems that selftarget is determined >> by Committed_AS in meminfo. >> >> I did some test on tpcc benchmark. the OS is initially allocated 2048MB of >> memory, and then I enabled the selfballooning by echo 1 > /proc/xen/balloon. >> The configuration of xenballoond.conf is the default except that I enabled >> self-balloon. >> >> The following is the meminfo >> >> MemTotal: 1939996 kB >> MemFree: 1183888 kB >> Buffers: 13260 kB >> Cached: 396492 kB >> SwapCached: 0 kB >> Active: 386912 kB >> Inactive: 213584 kB >> HighTotal: 0 kB >> HighFree: 0 kB >> LowTotal: 1939996 kB >> LowFree: 1183888 kB >> SwapTotal: 2097136 kB >> SwapFree: 2097132 kB >> Dirty: 78432 kB >> Writeback: 0 kB >> AnonPages: 190668 kB >> Mapped: 14228 kB >> Slab: 53148 kB >> PageTables: 4836 kB >> NFS_Unstable: 0 kB >> Bounce: 0 kB >> CommitLimit: 3067132 kB >> Committed_AS: 524484 kB >> VmallocTotal: 34359738367 kB >> VmallocUsed: 1208 kB >> VmallocChunk: 34359737159 kB >> >> It seems that the self-balloon driver is reluctant to give up free memory >> from the guest although the Committed_AS is small. The self-ballooning did >> changed the selftarget to something like 1.6GB but quickly changed to and >> fix at 193xxxxKB. >> >> According to the meminfo, we should be able to give up at least 800MB back >> to the VMM. >> FYI: xen 3.3.1, linux 2.6.18.8 in both dom0 and domu. >> >> Any ideas? >> >> Thanks, >> Jia >> > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Dan Magenheimer
2009-Nov-11 03:08 UTC
RE: [Xen-devel] How does the self-ballooning daemon calculate the selftarget?
Hmmmm... I think Ian Campbell''s patch needs to be applied in the guest. So it is a mystery to me why your guest worked on your friend''s 3.4 machine and not on your 3.3 machine. -----Original Message----- From: Jia Rao [mailto:rickenrao@gmail.com] Sent: Tuesday, November 10, 2009 6:22 PM To: Dan Magenheimer Cc: xen-devel@lists.xensource.com Subject: Re: [Xen-devel] How does the self-ballooning daemon calculate the selftarget? Hi Dan, That is because the bug you mentioned here. I ran my guest under my friend''s xen 3.4 which has a updated 2.6.18.8 dom0, the self ballooning worked fine (I didnt patch the guest kernel). I am going to patch the fix by Ian Campbell in my xen 3.3 machines. Do I need to patch the guest kernel too? or just patch the dom0 kernel? Thanks, Jia On Tue, Nov 10, 2009 at 3:45 PM, Dan Magenheimer <dan.magenheimer@oracle.com> wrote: Hi Jia -- First, please note that when XENBALLOON_MINMEM is set to 0, that does NOT mean 0MB. As described in the text in the xenballoon config file, XENBALLOON_MINMEM==0 means that the heuristic is used which is in my previous reply. As for Committed_AS, it is certainly not a perfect metric. It was just an adequate one (and much better than no metric at all). I''m not sure, but your observation of meminfo not being periodically updated may be a result of a balloon driver bug that was fixed in a patch by Ian Campbell here: http://xenbits.xensource.com/linux-2.6.18-xen.hg?rev/329ea0ccb344 If you are interested in the area of memory utilization optimization, I''d encourage you to try tmem, which is in xen-unstable and the Oracle VM 2.2 product. (See http://oss.oracle.com/projects/tmem for more info.) Dan P.S. The minimum memory heuristic patch for the balloon driver is here: http://xenbits.xensource.com/linux-2.6.18-xen.hg?rev/b61443b1bf76 -----Original Message----- From: Jia Rao [mailto:rickenrao@gmail.com] Sent: Tuesday, November 10, 2009 12:19 PM To: Dan Magenheimer Cc: xen-devel@lists.xensource.com Subject: Re: [Xen-devel] How does the self-ballooning daemon calculate the selftarget? Hi Dan, Thanks for the reply. The XENBALLOON_MINMEM was set to 0 in my case. I am wondering if the /proc/meminfo is a reliable source for memory utilization of a guest. In my case, the guest stays in idle for several hours, the metrics in /proc/meminfo is seldom changed. For example, the Committed_AS and Active metrics stay almost the same for several hours. In earlier kernels, the Active and Inactive metrics are periodically updated by a kernel thread kscand, which seems to has been removed from the 2.6 kernel. From what I have observed, the meminfo is correctly updated if the memory demand from the guest is increasing. However, once the memory demand drops or be almost zero (idle), the meminfo is not periodically updated. This might be a problem for the self-balloon daemon to deflate the balloon. Thanks, Jia On Tue, Nov 10, 2009 at 12:43 PM, Dan Magenheimer <dan.magenheimer@oracle.com> wrote: Hi Jia -- There are changes in newer versions of the balloon driver to ensure that ballooning does not take too much memory away from the kernel, which would result in kernel crashes. These changes set a different minimum value depending on how much memory the kernel originally discovered. If the self-ballooning scripts run with an "old" balloon driver and one changes certain parameters in the scripts, kernel crashes might occur. So the self-ballooning scripts implement a minimum memory algorithm similar to the newer balloon driver to avoid this situation. Look for XENBALLOON_MINMEM in the xenballoon config file if you want to change this. Dan -----Original Message----- From: Jia Rao [mailto:rickenrao@gmail.com] Sent: Tuesday, November 10, 2009 9:23 AM To: xen-devel@lists.xensource.com Subject: [Xen-devel] How does the self-ballooning daemon calculate the selftarget? Hi all, I am wondering how does the self-ballooning daemon calculate the selftarget for the guest OS. >From the README of balloon driver, it seems that selftarget is determined by Committed_AS in meminfo. I did some test on tpcc benchmark. the OS is initially allocated 2048MB of memory, and then I enabled the selfballooning by echo 1 > /proc/xen/balloon. The configuration of xenballoond.conf is the default except that I enabled self-balloon. The following is the meminfo MemTotal: 1939996 kB MemFree: 1183888 kB Buffers: 13260 kB Cached: 396492 kB SwapCached: 0 kB Active: 386912 kB Inactive: 213584 kB HighTotal: 0 kB HighFree: 0 kB LowTotal: 1939996 kB LowFree: 1183888 kB SwapTotal: 2097136 kB SwapFree: 2097132 kB Dirty: 78432 kB Writeback: 0 kB AnonPages: 190668 kB Mapped: 14228 kB Slab: 53148 kB PageTables: 4836 kB NFS_Unstable: 0 kB Bounce: 0 kB CommitLimit: 3067132 kB Committed_AS: 524484 kB VmallocTotal: 34359738367 kB VmallocUsed: 1208 kB VmallocChunk: 34359737159 kB It seems that the self-balloon driver is reluctant to give up free memory from the guest although the Committed_AS is small. The self-ballooning did changed the selftarget to something like 1.6GB but quickly changed to and fix at 193xxxxKB. According to the meminfo, we should be able to give up at least 800MB back to the VMM. FYI: xen 3.3.1, linux 2.6.18.8 in both dom0 and domu. Any ideas? Thanks, Jia _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel