I''ve had this problem on several occasions and it''s a pain. When I try configure another domU using virt-manager I get a can''t allocate memory error. My dom0 is limited with the startup parameter dom0_mem=2048M. The system has 14 gig of RAM. It''s running SLES 10SP1. It let''s me select the amount of RAM and says 8GB is available. In this case I''m only choosing 512meg. I know if I restart the server, or stop and restart xend and xendomains it will clear up this issue, for a while anyway. Is there any way to clear things up without a restart? I really don''t want to bring down the other running domU''s. Thanks, James _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Tue, Jul 8, 2008 at 11:49 AM, James Pifer <jep@obrien-pifer.com> wrote:> I''ve had this problem on several occasions and it''s a pain. When I try > configure another domU using virt-manager I get a can''t allocate memory > error. My dom0 is limited with the startup parameter dom0_mem=2048M. The > system has 14 gig of RAM. It''s running SLES 10SP1. It let''s me select > the amount of RAM and says 8GB is available. In this case I''m only > choosing 512meg. > > I know if I restart the server, or stop and restart xend and xendomains > it will clear up this issue, for a while anyway. > > Is there any way to clear things up without a restart? I really don''t > want to bring down the other running domU''s.None that I know of. I''ve run into this issue many times in test, with both RHEL and SLES and it''s an ongoing problem that no one seems to be able to resolve. The other issue is that this is not necessarily easy to reproduce either. The oddity here is that you are still seeing the problem. In every case I''ve run across, limiting dom0 like you''ve done resolved the problem. This makes me wonder a few things... 1: how many domUs do you have running, and what are their memory limits? 2: Have you tried a full restart, along with limiting the max memory for each domU to ensure that the guests don''t try grabbing too much memory? As I recall, besides limiting dom0''s max, if you don''t limit the domU max they too will be able to grab additional RAM. 3: Are you in a scenario where you are starting and stopping them repeatedly? That''s more a curiosity for me, as I wonder if you are stopping some guests and their memory is not being released to the pool for other guests to grab as they are started. 4: Not that it makes THAT much difference, but are you running 32bit or 64bit SLES? 5: Finally, again, not that it should make too much difference, but if this is a NUMA capable machine, are you enabling NUMA at boot time? I don''t know that NUMA would do anything to help or hinder here, but it''s a thought at least. I am not a Xen developer or anything, but I''m curious about this as well. As I said, I''ve seen this myself on both RHEL and SLES, going back to RHEL 5.0 and SLES 10 all the way through RHEL 5.2 and SLES 10 SP2 in testing. Good luck... Jeff _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> None that I know of. I''ve run into this issue many times in test, > with both RHEL and SLES and it''s an ongoing problem that no one seems > to be able to resolve. The other issue is that this is not > necessarily easy to reproduce either. > > The oddity here is that you are still seeing the problem. In every > case I''ve run across, limiting dom0 like you''ve done resolved the > problem. > > This makes me wonder a few things... > > 1: how many domUs do you have running, and what are their memory limits?Total system RAM 14gb. dom0 = 2gb server1 (full) = 3gb server2 (full) = 1gb server3 (para) = 3gb (NOT running) server4 (para) = 1gb server5 (full) = 512meg (NOT running) server6 (full) = 512meg server7 (full) = 1gb (NOT running) server8 (full) = 512mg (NOT running) full = windows full virt para = linux para virt> 2: Have you tried a full restart, along with limiting the max memory > for each domU to ensure that the guests don''t try grabbing too much > memory? As I recall, besides limiting dom0''s max, if you don''t limit > the domU max they too will be able to grab additional RAM.Was not aware you could/should limit the domU''s. Can you tell me that setting? I''m willing to try it, but according to virt-manager they are all using what they are supposed to.> 3: Are you in a scenario where you are starting and stopping them > repeatedly? That''s more a curiosity for me, as I wonder if you are > stopping some guests and their memory is not being released to the > pool for other guests to grab as they are started.Not really, but their certainly has been a number of stops and restarts.> 4: Not that it makes THAT much difference, but are you running 32bit > or 64bit SLES?64bit> 5: Finally, again, not that it should make too much difference, but > if this is a NUMA capable machine, are you enabling NUMA at boot time? > I don''t know that NUMA would do anything to help or hinder here, but > it''s a thought at least.Don''t know if it''s NUMA capable. This machine is an HP DL360-G5. I don''t see any NUMA parameter in the boot parameters. Thanks, James _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hello, James, May be it will be helpful. Try> xm delete <DomU_Name> > xm new <DomU_Name>and than start it. It is some less destructive than reboot of Dom0. Tuesday, July 8, 2008, 8:52:23 PM, you wrote:>> None that I know of. I''ve run into this issue many times in test, >> with both RHEL and SLES and it''s an ongoing problem that no one seems >> to be able to resolve. The other issue is that this is not >> necessarily easy to reproduce either. >> >> The oddity here is that you are still seeing the problem. In every >> case I''ve run across, limiting dom0 like you''ve done resolved the >> problem. >> >> This makes me wonder a few things... >> >> 1: how many domUs do you have running, and what are their memory limits?JP> Total system RAM 14gb. JP> dom0 = 2gb JP> server1 (full) = 3gb JP> server2 (full) = 1gb JP> server3 (para) = 3gb (NOT running) JP> server4 (para) = 1gb JP> server5 (full) = 512meg ? (NOT running) JP> server6 (full) = 512meg JP> server7 (full) = 1gb ? (NOT running) JP> server8 (full) = 512mg ? (NOT running) JP> full = windows full virt JP> para = linux para virt>> 2: Have you tried a full restart, along with limiting the max memory >> for each domU to ensure that the guests don''t try grabbing too much >> memory? As I recall, besides limiting dom0''s max, if you don''t limit >> the domU max they too will be able to grab additional RAM.JP> Was not aware you could/should limit the domU''s. Can you tell me that JP> setting? I''m willing to try it, but according to virt-manager they are JP> all using what they are supposed to.>> 3: Are you in a scenario where you are starting and stopping them >> repeatedly? That''s more a curiosity for me, as I wonder if you are >> stopping some guests and their memory is not being released to the >> pool for other guests to grab as they are started.JP> Not really, but their certainly has been a number of stops and JP> restarts.>> 4: Not that it makes THAT much difference, but are you running 32bit >> or 64bit SLES?JP> 64bit>> 5: Finally, again, not that it should make too much difference, but >> if this is a NUMA capable machine, are you enabling NUMA at boot time? >> I don''t know that NUMA would do anything to help or hinder here, but >> it''s a thought at least.JP> Don''t know if it''s NUMA capable. This machine is an HP DL360-G5. I don''t JP> see any NUMA parameter in the boot parameters. JP> Thanks, JP> James JP> _______________________________________________ JP> Xen-users mailing list JP> Xen-users@lists.xensource.com JP> http://lists.xensource.com/xen-users -- С уважением, Вылуско И.М. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Tue, Jul 8, 2008 at 12:52 PM, James Pifer <jep@obrien-pifer.com> wrote:> Total system RAM 14gb. > dom0 = 2gb > server1 (full) = 3gb > server2 (full) = 1gb > server3 (para) = 3gb (NOT running) > server4 (para) = 1gb > server5 (full) = 512meg (NOT running) > server6 (full) = 512meg > server7 (full) = 1gb (NOT running) > server8 (full) = 512mg (NOT running)So overall, 12.5GB worth of Domains, dom0 inclusive, and 14GB total system ram... doesn't Xen also have some overhead in the neighborhood of 1 - 2 GB for the Hypervisor as well? By "NOT running" do you mean that they WERE running but are stopped now, or that none of those will start at all? I'm leaning toward the "memory not being released" hypothesis here... I've never seen Igor's suggestion before, but xm delete SHOULD free any resources reserved by the domUs being deleted. In any case, Why not? :-)> Was not aware you could/should limit the domU's. Can you tell me that > setting? I'm willing to try it, but according to virt-manager they are > all using what they are supposed to.I've never had to before, but you can limit the domU's max ram allowance via a line in the domUs config file in /etc/xen or via xm. I'm not positive what the actual parameter is of the top of my head though, but it should be listed in docs somewhere... and if you do an xm --help or something along those lines, you should get a full list of xm parameters to let you limit them on the fly. supposedly, man 5 xmdomain.cfg should give you the domain config format. Also, this may help too: http://wiki.rpath.com/wiki/Xen_DomU_Guide I don't know that limiting max mem for a guest is absolutely necessary, but since you are really getting the most out of your available RAM, it certainly wouldn't hurt to put some limits in place. At least, that "should" guarantee that guests are not grabbing any more than necessary.> Not really, but their certainly has been a number of stops and > restarts.Yep... now I am really betting on ram not being freed up after domains are shut down. Again... bug, IMO.>> 4: Not that it makes THAT much difference, but are you running 32bit >> or 64bit SLES? > > 64bitOk... like I said, not that it matters, really, but at least that rules out any 32Bit memory handling weirdness.> Don't know if it's NUMA capable. This machine is an HP DL360-G5. I don't > see any NUMA parameter in the boot parameters.I have no idea. Google didn't even turn up anything useful. You could always try passing numa=on to the kernel line in grub... I am pretty sure that on a non-NUMA system, that won't do anything. But on a NUMA system, it will at least enable it. SLES and RHEL do not enable NUMA by default in Xen, so you have to explicitly turn it on. BUT, to be honest, I couldn't even find it mentioned on the HP website for the DL-360 G5, and I'd imagine that that is something they would certainly include in their marketing schwag as a feature... Of course, I saw some post somewhere about having a "Memory Affinity" option in BIOS setup if the system is NUMA aware... I have all IBM boxen, and I am pretty sure they do not have a Memory Affinity option in BIOS, so YMMV. But certainly try Igor's suggestion. And let me know if that works for you, as that could help my and my people out as well. Cheers Jeff -- ------------------> Jeffrey Lane - W4KDH <------------------- www.jefflane.org Another cog in the great Corporate Wheel The internet has no government, no constitution, no laws, no rights, no police, no courts. Don't talk about fairness or innocence, and don't talk about what should be done. Instead, talk about what is being done and what will be done by the amorphous unreachable undefinable blob called "the internet user base." -Paul Vixie _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Tue, 2008-07-08 at 21:43 +0400, Igor Vylusko wrote:> Hello, James, > > May be it will be helpful. > > Try > > xm delete <DomU_Name>Sorry, I''m a bit confused. What would I be deleting? I''m not trying to remove any existing domU''s, just add a new one. James _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Tue, 2008-07-08 at 21:43 +0400, Igor Vylusko wrote:> Hello, James, > > May be it will be helpful. > > Try > > xm delete <DomU_Name> > > xm new <DomU_Name> > and than start it. >Okay, I deleted the server that had 3gig but was not running using the xm delete command. Then I restart virt-manager and was able to create the new one I was trying to create. Thanks, James _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hello, James, I have faced with similar problem on starting existing DomU. And this solution helps me. But, I really read your initial message not carefully. Sorry. :) Tuesday, July 8, 2008, 10:00:04 PM, you wrote: JP> On Tue, 2008-07-08 at 21:43 +0400, Igor Vylusko wrote:>> Hello, James, >> >> May be it will be helpful. >> >> Try >> > xm delete <DomU_Name>JP> Sorry, I''m a bit confused. What would I be deleting? I''m not trying to JP> remove any existing domU''s, just add a new one. JP> James Igor. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Tue, Jul 8, 2008 at 2:12 PM, James Pifer <jep@obrien-pifer.com> wrote:>> Try >> > xm delete <DomU_Name> >> > xm new <DomU_Name> >> and than start it. >> > > Okay, I deleted the server that had 3gig but was not running using the > xm delete command. Then I restart virt-manager and was able to create > the new one I was trying to create.Cool... Thanks for the update. I''m going to be sure that gets added to our list of tricks as I am sure we''ll see this happen again. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users