I have xen 3.04 running on SLES10SP1. It''s working great, but I''m wondering if I have some memory issues with dom0. I have 7 domU''s, 1 is win2k, the other 6 are win2k3. I''ve had a lot of machines up and down with a reboot of dom0 in quite a while. I''m wondering if dom0 has not given up memory during the process. I added 5 win2k3 domU''s today. They simply run reports, but the report engine basically runs them one by one. Before adding and turning them on dom0 was using over 8 GB of RAM. The server has 16 GB of RAM. Right now all 7 domU''s are turned on. According to virt-manager my memory usage is: dom0 8.68 GB win2k 1.01 GB win2k3_1 1.01 GB win2k3_2 519 MB win2k3_3 519 MB win2k3_4 519 MB win2k3_5 519 MB win2k3_6 519 MB I''m wondering if I shut down all the domU''s and restart if I''ll gain some RAM back from dom0? Or, is this typical? Thanks, James _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Tue, May 20, 2008 at 7:43 PM, James Pifer <jep@obrien-pifer.com> wrote:> I''m wondering if I shut down all the domU''s and restart if I''ll gain > some RAM back from dom0? Or, is this typical?That sounds about right to me... at least in my experience. The memory ballooning code in Xen is ... strange and does not always behave as you may expect it to. It''s also been my experience that it''s really just best practice to limit dom0 to a small amount of ram by default leaving all the rest for guests... Typically, I limit dom0 to either 1024 or 2048 MB by using the dom0_mem= parameter. You pass this on the kernel line in grub. That would be the line that actually says kernel and boots the hypervisor, NOT the line that says module and boots dom0''s kernel. I''ve seen all sorts of fun things with memory ballooning, from memory not being recovered (which I think is expected behaviour) to the hypervisor refusing to allocate or balloon memory for a new domU. -- ------------------> Jeffrey Lane - W4KDH <------------------- www.jefflane.org Another cog in the great Corporate Wheel The internet has no government, no constitution, no laws, no rights, no police, no courts. Don''t talk about fairness or innocence, and don''t talk about what should be done. Instead, talk about what is being done and what will be done by the amorphous unreachable undefinable blob called "the internet user base." -Paul Vixie _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Wed, 2008-05-21 at 09:15 -0400, Jeff Lane wrote:> On Tue, May 20, 2008 at 7:43 PM, James Pifer <jep@obrien-pifer.com> wrote: > > > I''m wondering if I shut down all the domU''s and restart if I''ll gain > > some RAM back from dom0? Or, is this typical? > > That sounds about right to me... at least in my experience. The > memory ballooning code in Xen is ... strange and does not always > behave as you may expect it to. It''s also been my experience that > it''s really just best practice to limit dom0 to a small amount of ram > by default leaving all the rest for guests... > > Typically, I limit dom0 to either 1024 or 2048 MB by using the > dom0_mem= parameter. You pass this on the kernel line in grub. That > would be the line that actually says kernel and boots the hypervisor, > NOT the line that says module and boots dom0''s kernel. > > I''ve seen all sorts of fun things with memory ballooning, from memory > not being recovered (which I think is expected behaviour) to the > hypervisor refusing to allocate or balloon memory for a new domU. > >Okay, so mine has: kernel /boot/xen.gz module /boot/vmlinuz-2.6.16.46-0.12-xen root=/dev/cciss/c0d0p2 vga=0x317 resume=/dev/cciss/c0d0p1 splash=silent showopts module /boot/initrd-2.6.16.46-0.12-xen That would become: kernel /boot/xen.gz dom0_mem=2048 module /boot/vmlinuz-2.6.16.46-0.12-xen root=/dev/cciss/c0d0p2 vga=0x317 resume=/dev/cciss/c0d0p1 splash=silent showopts module /boot/initrd-2.6.16.46-0.12-xen Are there any negative side affects with doing this? Thanks, James _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Wed, May 21, 2008 at 9:45 AM, James Pifer <jep@obrien-pifer.com> wrote: Correction:> That would become: > kernel /boot/xen.gz dom0_mem=2048M > module /boot/vmlinuz-2.6.16.46-0.12-xen root=/dev/cciss/c0d0p2 vga=0x317 resume=/dev/cciss/c0d0p1 splash=silent showopts > module /boot/initrd-2.6.16.46-0.12-xenI am not positive that the default for that parameter is in Megabytes, BUT the regular mem= parameter for Linux kernels defaults to bytes if no modifier is specified. Given that, its good practice to explicitly state whether you want M(egabytes), K(ilobytes) or G(gigabytes)... that way you know for a fact what you are allocating and not just assuming that the kernel or hypervisor knows what you mean.> Are there any negative side affects with doing this?Nope... the only thing that happens with this is that, at boot, Domain0 is brought up at 2048M or 2GB. I am pretty sure that this will also overide the max-mem setting for domain 0, but to be sure, you can set that as well to ensure Domain0 does not try to grow beyond the 2GB limit you set in grub. Now, as I said, I''ve seen all sorts of weirdness happen related to the ballooning mechanism in Xen. In some cases, I''ve seen where you can start X number of guests, then you try starting X+1 and the hypervisor spits out an error saying "Cannot allocate memory" even though you''ve still got more than enough ram to share. In others, I''ve seen where the machine would simply freeze... this was a race condition where starting a bunch of guests at the same time (using xm create from a script within a loop, for example) would cause the system to freeze because the guests were grabbing memory faster than Xen could take it from Domain 0. And there have been other things in between. I think the race condition got fixed, I haven''t seen that one pop up in a while. As for negative side effects, I''ve not noticed any. I''ve run as many as 80 guests on a system, all using 512 or 1024 MB each, wiht domain0 limited to either 2 or 4 GB at boot time. Granted, I could play wiht that a bit because I had a system with 256GB and 64 cores. My general rule with setting up a Xen server is Domain0 gets either 1GB (for small systems with 16GB or less) and 2GB for > 16GB (and I do use 4GB for the big systems (256+ GB). I''ve run anywhere from 4 to 80 domains at one time in these configs and while the guests start getting slower and more twitchy as the number of them increases, I have found no real issues that seem to be directly related to the amount of RAM that Domain0 uses. Cheers Jeff _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users