I''ve been running into this from time to time... usually when xm creating a new domU, /var/log/messages for dom0 spits out: <snip> May 3 02:56:04 tweak kernel: printk: 4 messages suppressed. May 3 02:56:04 tweak kernel: xen_net: Memory squeeze in netback driver. </snip> And at this point all of the network connectivity for *all* domU''s controlled for that dom0 vanish. I have to xm shutdown and xm create them again to get the network back. This doesn''t happen all of the time, seems rather random really. Its *really* annoying. Is there a way to prevent this from happening? --jason _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Is there any rule for how much memory dom0_mem should be set to? I''ve got 16g in this machine, and run ~12 1g domU''s. Is more memory for dom0_mem better? or is less better? --jason On May 3, 2007, at 8:14 AM, Adam Jacob wrote:> On Thu, May 03, 2007 at 03:03:43AM -0700, Jason Dillon wrote: >> I''ve been running into this from time to time... usually when xm >> creating a new domU, /var/log/messages for dom0 spits out: >> >> <snip> >> May 3 02:56:04 tweak kernel: printk: 4 messages suppressed. >> May 3 02:56:04 tweak kernel: xen_net: Memory squeeze in netback >> driver. >> </snip> >> >> And at this point all of the network connectivity for *all* domU''s >> controlled for that dom0 vanish. I have to xm shutdown and xm create >> them again to get the network back. This doesn''t happen all of the >> time, seems rather random really. >> >> Its *really* annoying. Is there a way to prevent this from >> happening? > > You need to set the dom0_mem parameter at boot time. Something like: > > kernel /xen-3.0.3-1-amd64.gz dom0_mem=256M > > Has been working for us. > > Adam > > -- > HJK Solutions - We Launch Startups - http://www.hjksolutions.com > Adam Jacob, Senior Partner > T: (206) 508-4759 E: adam@hjksolutions.com_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> Is there any rule for how much memory dom0_mem should be set to? > I''ve got 16g in this machine, and run ~12 1g domU''s. > > Is more memory for dom0_mem better? or is less better? > > --jason > > > On May 3, 2007, at 8:14 AM, Adam Jacob wrote: > >> On Thu, May 03, 2007 at 03:03:43AM -0700, Jason Dillon wrote: >>> I''ve been running into this from time to time... usually when xm >>> creating a new domU, /var/log/messages for dom0 spits out: >>> >>> <snip> >>> May 3 02:56:04 tweak kernel: printk: 4 messages suppressed. >>> May 3 02:56:04 tweak kernel: xen_net: Memory squeeze in netback >>> driver. >>> </snip> >>> >>> And at this point all of the network connectivity for *all* domU''s >>> controlled for that dom0 vanish. I have to xm shutdown and xm create >>> them again to get the network back. This doesn''t happen all of the >>> time, seems rather random really. >>> >>> Its *really* annoying. Is there a way to prevent this from >>> happening? >> >> You need to set the dom0_mem parameter at boot time. Something like: >> >> kernel /xen-3.0.3-1-amd64.gz dom0_mem=256M >> >> Has been working for us. >> >> Adam >>Have any of you tried setting the dom0_min_mem variable? I had this problem back in 3.0.2 and if I remember right the solution was to set dom0_min_mem so that there wasa minimum that Dom0 would balloon down. I think that fixed it. Ryan _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
I''ve not tried setting dom0_min_mem, this is also a kernel param? Any suggestions on what that size should be? --jason On May 3, 2007, at 4:03 PM, Ryan Burke wrote:>> Is there any rule for how much memory dom0_mem should be set to? >> I''ve got 16g in this machine, and run ~12 1g domU''s. >> >> Is more memory for dom0_mem better? or is less better? >> >> --jason >> >> >> On May 3, 2007, at 8:14 AM, Adam Jacob wrote: >> >>> On Thu, May 03, 2007 at 03:03:43AM -0700, Jason Dillon wrote: >>>> I''ve been running into this from time to time... usually when xm >>>> creating a new domU, /var/log/messages for dom0 spits out: >>>> >>>> <snip> >>>> May 3 02:56:04 tweak kernel: printk: 4 messages suppressed. >>>> May 3 02:56:04 tweak kernel: xen_net: Memory squeeze in netback >>>> driver. >>>> </snip> >>>> >>>> And at this point all of the network connectivity for *all* domU''s >>>> controlled for that dom0 vanish. I have to xm shutdown and xm >>>> create >>>> them again to get the network back. This doesn''t happen all of the >>>> time, seems rather random really. >>>> >>>> Its *really* annoying. Is there a way to prevent this from >>>> happening? >>> >>> You need to set the dom0_mem parameter at boot time. Something >>> like: >>> >>> kernel /xen-3.0.3-1-amd64.gz dom0_mem=256M >>> >>> Has been working for us. >>> >>> Adam >>> > > > Have any of you tried setting the dom0_min_mem variable? I had this > problem back in 3.0.2 and if I remember right the solution was to set > dom0_min_mem so that there wasa minimum that Dom0 would balloon > down. I > think that fixed it. > > Ryan_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> I''ve not tried setting dom0_min_mem, this is also a kernel param? > Any suggestions on what that size should be?It''s a setting in /etc/xen/xend-config.sxp. Set it to something reasonable to make sure that dom0 doesn''t get ballooned down too far. I think for a production machine a sensible measure would be to set dom0-min-mem=0 to disable auto-ballooning entirely - then set dom0_mem on the Xen command line. On my test box where I run up to about 3-4 domUs, not heavily loaded, I have been running with dom0-min-mem=196M and dom0_mem=256M. I should probably just disable ballooning altogether on this machine. A static dom0 memory allocation should really be just fine on a server, the auto-ballooning is more useful for desktop use or for playing around and getting the feel of things without having to reconfigure manually all the time. Cheers, Mark> --jason > > On May 3, 2007, at 4:03 PM, Ryan Burke wrote: > >> Is there any rule for how much memory dom0_mem should be set to? > >> I''ve got 16g in this machine, and run ~12 1g domU''s. > >> > >> Is more memory for dom0_mem better? or is less better? > >> > >> --jason > >> > >> On May 3, 2007, at 8:14 AM, Adam Jacob wrote: > >>> On Thu, May 03, 2007 at 03:03:43AM -0700, Jason Dillon wrote: > >>>> I''ve been running into this from time to time... usually when xm > >>>> creating a new domU, /var/log/messages for dom0 spits out: > >>>> > >>>> <snip> > >>>> May 3 02:56:04 tweak kernel: printk: 4 messages suppressed. > >>>> May 3 02:56:04 tweak kernel: xen_net: Memory squeeze in netback > >>>> driver. > >>>> </snip> > >>>> > >>>> And at this point all of the network connectivity for *all* domU''s > >>>> controlled for that dom0 vanish. I have to xm shutdown and xm > >>>> create > >>>> them again to get the network back. This doesn''t happen all of the > >>>> time, seems rather random really. > >>>> > >>>> Its *really* annoying. Is there a way to prevent this from > >>>> happening? > >>> > >>> You need to set the dom0_mem parameter at boot time. Something > >>> like: > >>> > >>> kernel /xen-3.0.3-1-amd64.gz dom0_mem=256M > >>> > >>> Has been working for us. > >>> > >>> Adam > > > > Have any of you tried setting the dom0_min_mem variable? I had this > > problem back in 3.0.2 and if I remember right the solution was to set > > dom0_min_mem so that there wasa minimum that Dom0 would balloon > > down. I > > think that fixed it. > > > > Ryan > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users-- Dave: Just a question. What use is a unicyle with no seat? And no pedals! Mark: To answer a question with a question: What use is a skateboard? Dave: Skateboards have wheels. Mark: My wheel has a wheel! _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> > I''ve not tried setting dom0_min_mem, this is also a kernel param? > > Any suggestions on what that size should be? > > It''s a setting in /etc/xen/xend-config.sxp. Set it to something reasonable > to make sure that dom0 doesn''t get ballooned down too far. I think for a > production machine a sensible measure would be to set dom0-min-mem=0 to > disable auto-ballooning entirely - then set dom0_mem on the Xen command > line. > > On my test box where I run up to about 3-4 domUs, not heavily loaded, I > have been running with dom0-min-mem=196M and dom0_mem=256M. I should > probably just disable ballooning altogether on this machine. > > A static dom0 memory allocation should really be just fine on a server, the > auto-ballooning is more useful for desktop use or for playing around and > getting the feel of things without having to reconfigure manually all the > time.I''d just like to add at this point that I suspect it''s possible that ballooning dom0 to too small a size *or* setting dom0_mem to too small a size could cause memory squeezes. But also, they have been known to occur in the past by dom0 being ballooned down by too many MB when the Linux memory manager confuses the ballooning with memory pressure - even though dom0 still ought to have enough memory free. I''m not sure if this can still happen, but this is why I recommend using a static allocation on a production server. Cheers, Mark> Cheers, > Mark > > > --jason > > > > On May 3, 2007, at 4:03 PM, Ryan Burke wrote: > > >> Is there any rule for how much memory dom0_mem should be set to? > > >> I''ve got 16g in this machine, and run ~12 1g domU''s. > > >> > > >> Is more memory for dom0_mem better? or is less better? > > >> > > >> --jason > > >> > > >> On May 3, 2007, at 8:14 AM, Adam Jacob wrote: > > >>> On Thu, May 03, 2007 at 03:03:43AM -0700, Jason Dillon wrote: > > >>>> I''ve been running into this from time to time... usually when xm > > >>>> creating a new domU, /var/log/messages for dom0 spits out: > > >>>> > > >>>> <snip> > > >>>> May 3 02:56:04 tweak kernel: printk: 4 messages suppressed. > > >>>> May 3 02:56:04 tweak kernel: xen_net: Memory squeeze in netback > > >>>> driver. > > >>>> </snip> > > >>>> > > >>>> And at this point all of the network connectivity for *all* domU''s > > >>>> controlled for that dom0 vanish. I have to xm shutdown and xm > > >>>> create > > >>>> them again to get the network back. This doesn''t happen all of the > > >>>> time, seems rather random really. > > >>>> > > >>>> Its *really* annoying. Is there a way to prevent this from > > >>>> happening? > > >>> > > >>> You need to set the dom0_mem parameter at boot time. Something > > >>> like: > > >>> > > >>> kernel /xen-3.0.3-1-amd64.gz dom0_mem=256M > > >>> > > >>> Has been working for us. > > >>> > > >>> Adam > > > > > > Have any of you tried setting the dom0_min_mem variable? I had this > > > problem back in 3.0.2 and if I remember right the solution was to set > > > dom0_min_mem so that there wasa minimum that Dom0 would balloon > > > down. I > > > think that fixed it. > > > > > > Ryan > > > > _______________________________________________ > > Xen-users mailing list > > Xen-users@lists.xensource.com > > http://lists.xensource.com/xen-users-- Dave: Just a question. What use is a unicyle with no seat? And no pedals! Mark: To answer a question with a question: What use is a skateboard? Dave: Skateboards have wheels. Mark: My wheel has a wheel! _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Tue, May 08, 2007, Mark Williamson wrote:> > I''ve not tried setting dom0_min_mem, this is also a kernel param? > > Any suggestions on what that size should be? > > It''s a setting in /etc/xen/xend-config.sxp. Set it to something reasonable to > make sure that dom0 doesn''t get ballooned down too far. I think for a > production machine a sensible measure would be to set dom0-min-mem=0 to > disable auto-ballooning entirely - then set dom0_mem on the Xen command line. > > On my test box where I run up to about 3-4 domUs, not heavily loaded, I have > been running with dom0-min-mem=196M and dom0_mem=256M. I should probably > just disable ballooning altogether on this machine. > > A static dom0 memory allocation should really be just fine on a server, the > auto-ballooning is more useful for desktop use or for playing around and > getting the feel of things without having to reconfigure manually all the > time.I''ve tried running a couple of heavy network servers at 256meg dom0 but I keep getting memory squeeze errors. I wouldn''t mind it -that- much if it didn''t then just lock up all domU networking. I can trigger it pretty readily by running rsync''s from a domU to an external server at ~50-100mbit. I wonder how hard it''d be to fix that.. Adrian _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> > On my test box where I run up to about 3-4 domUs, not heavily loaded, I > > have been running with dom0-min-mem=196M and dom0_mem=256M. I should > > probably just disable ballooning altogether on this machine. > > > > A static dom0 memory allocation should really be just fine on a server, > > the auto-ballooning is more useful for desktop use or for playing around > > and getting the feel of things without having to reconfigure manually all > > the time. > > I''ve tried running a couple of heavy network servers at 256meg dom0 but I > keep getting memory squeeze errors. I wouldn''t mind it -that- much if it > didn''t then just lock up all domU networking. I can trigger it pretty > readily by running rsync''s from a domU to an external server at > ~50-100mbit. > > I wonder how hard it''d be to fix that..Are you running the network servers in separate domUs or in dom0 itself? Are the domUs using network servers in dom0? Is this with a statically allocated dom0 memory footprint, or autoballooning? Don''t you just hate people who ask lots of questions? ;-) Cheers, Mark -- Dave: Just a question. What use is a unicyle with no seat? And no pedals! Mark: To answer a question with a question: What use is a skateboard? Dave: Skateboards have wheels. Mark: My wheel has a wheel! _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Tue, May 08, 2007, Mark Williamson wrote:> > > On my test box where I run up to about 3-4 domUs, not heavily loaded, I > > > have been running with dom0-min-mem=196M and dom0_mem=256M. I should > > > probably just disable ballooning altogether on this machine. > > > > > > A static dom0 memory allocation should really be just fine on a server, > > > the auto-ballooning is more useful for desktop use or for playing around > > > and getting the feel of things without having to reconfigure manually all > > > the time. > > > > I''ve tried running a couple of heavy network servers at 256meg dom0 but I > > keep getting memory squeeze errors. I wouldn''t mind it -that- much if it > > didn''t then just lock up all domU networking. I can trigger it pretty > > readily by running rsync''s from a domU to an external server at > > ~50-100mbit. > > > > I wonder how hard it''d be to fix that.. > > Are you running the network servers in separate domUs or in dom0 itself? Are > the domUs using network servers in dom0? Is this with a statically allocated > dom0 memory footprint, or autoballooning?* Network servers (eg rsync) are in domU, not dom0. dom0 just does devices, bridging, munin-node and other small bits of housekeeping. The rsync is going to an external machine, not into dom0. * Oh, its with minimal ballooning - I manually set dom0 mem down to 256meg at boot and let it stay there. The new servers are at 384meg.> Don''t you just hate people who ask lots of questions? ;-)Nope! Adrian _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> > > I wonder how hard it''d be to fix that.. > > > > Are you running the network servers in separate domUs or in dom0 itself? > > Are the domUs using network servers in dom0? Is this with a statically > > allocated dom0 memory footprint, or autoballooning? > > * Network servers (eg rsync) are in domU, not dom0. dom0 just does devices, > bridging, munin-node and other small bits of housekeeping. The rsync is > going to an external machine, not into dom0.OK. The reason I asked was that there was a problem with some older versions of Xen when weird stuff would happen if domUs were using network services that were running in dom0... I can''t remember what error this gave, but it seemed worth ruling out.> * Oh, its with minimal ballooning - I manually set dom0 mem down to 256meg > at boot and let it stay there. The new servers are at 384meg.If you set dom0_mem on the the command line and don''t use any ballooning, do you still get this problem?> > Don''t you just hate people who ask lots of questions? ;-) > > Nope!That''s definitely good - but what else could I say? ;-) Cheers, Mark -- Dave: Just a question. What use is a unicyle with no seat? And no pedals! Mark: To answer a question with a question: What use is a skateboard? Dave: Skateboards have wheels. Mark: My wheel has a wheel! _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Tue, May 08, 2007, Mark Williamson wrote:> OK. The reason I asked was that there was a problem with some older versions > of Xen when weird stuff would happen if domUs were using network services > that were running in dom0... I can''t remember what error this gave, but it > seemed worth ruling out.Ah. I remember some weird stuff - like UDP NFS mounts from domU -> dom0 going weird - but I haven''t done any of that recently.> > * Oh, its with minimal ballooning - I manually set dom0 mem down to 256meg > > at boot and let it stay there. The new servers are at 384meg. > > If you set dom0_mem on the the command line and don''t use any ballooning, do > you still get this problem?I don''t -think- so. I''ll try it out with my test server at home over the weekend and report back anything interesting. Still, I wouldn''t mind at least having the memory squeeze condition recoverable somehow. I haven''t looked into it all lately. Adrian _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users