Pim van Riezen
2011-Feb-02 13:00 UTC
[Xen-users] Xen 4.0.1 hangs when trying to create vps
Good day, we recently migrated to Xen4.0.1 on one of our clusters. It solves a lot of problem, but it also adds one. Whenever we try to xm create a guest that oversteps the total amount of available memory, xend completely hangs itself up and stops responding to any rpc calls.. The last lines in xend.log are: [2011-02-02 10:00:22 8199] DEBUG (balloon:172) Balloon: tmem relinquished -1 KiB of 20100116 KiB requested. [2011-02-02 10:00:22 8199] DEBUG (balloon:226) Balloon: 13454316 KiB free; 0 to scrub; need 33554432; retries: 115. [2011-02-02 10:00:22 8199] DEBUG (balloon:226) Balloon: 13454316 KiB free; 0 to scrub; need 33554432; retries: 210. is there a way to recover from this? Cheers, Pim van Riezen _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Pasi Kärkkäinen
2011-Feb-21 09:48 UTC
Re: [Xen-users] Xen 4.0.1 hangs when trying to create vps
On Wed, Feb 02, 2011 at 02:00:36PM +0100, Pim van Riezen wrote:> Good day, > > we recently migrated to Xen4.0.1 on one of our clusters. It solves a lot of problem, but it also adds one. Whenever we try to xm create a guest that oversteps the total amount of available memory, xend completely hangs itself up and stops responding to any rpc calls.. > > The last lines in xend.log are: > > [2011-02-02 10:00:22 8199] DEBUG (balloon:172) Balloon: tmem relinquished -1 KiB of 20100116 KiB requested. > [2011-02-02 10:00:22 8199] DEBUG (balloon:226) Balloon: 13454316 KiB free; 0 to scrub; need 33554432; retries: 115. > [2011-02-02 10:00:22 8199] DEBUG (balloon:226) Balloon: 13454316 KiB free; 0 to scrub; need 33554432; retries: 210. > > is there a way to recover from this? >Did you dedicate fixed amount of memory for dom0? Are you using tmem? -- Pasi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Pim van Riezen
2011-Feb-21 09:52 UTC
Re: [Xen-users] Xen 4.0.1 hangs when trying to create vps
On Feb 21, 2011, at 10:48 , Pasi Kärkkäinen wrote:> On Wed, Feb 02, 2011 at 02:00:36PM +0100, Pim van Riezen wrote: >> Good day, >> >> we recently migrated to Xen4.0.1 on one of our clusters. It solves a lot of problem, but it also adds one. Whenever we try to xm create a guest that oversteps the total amount of available memory, xend completely hangs itself up and stops responding to any rpc calls.. >> >> The last lines in xend.log are: >> >> [2011-02-02 10:00:22 8199] DEBUG (balloon:172) Balloon: tmem relinquished -1 KiB of 20100116 KiB requested. >> [2011-02-02 10:00:22 8199] DEBUG (balloon:226) Balloon: 13454316 KiB free; 0 to scrub; need 33554432; retries: 115. >> [2011-02-02 10:00:22 8199] DEBUG (balloon:226) Balloon: 13454316 KiB free; 0 to scrub; need 33554432; retries: 210. >> >> is there a way to recover from this? >> > > Did you dedicate fixed amount of memory for dom0?Yes.> Are you using tmem?No. Cheers, Pi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Pasi Kärkkäinen
2011-Feb-21 10:19 UTC
Re: [Xen-users] Xen 4.0.1 hangs when trying to create vps
On Mon, Feb 21, 2011 at 10:52:49AM +0100, Pim van Riezen wrote:> > On Feb 21, 2011, at 10:48 , Pasi Kärkkäinen wrote: > > > On Wed, Feb 02, 2011 at 02:00:36PM +0100, Pim van Riezen wrote: > >> Good day, > >> > >> we recently migrated to Xen4.0.1 on one of our clusters. It solves a lot of problem, but it also adds one. Whenever we try to xm create a guest that oversteps the total amount of available memory, xend completely hangs itself up and stops responding to any rpc calls.. > >> > >> The last lines in xend.log are: > >> > >> [2011-02-02 10:00:22 8199] DEBUG (balloon:172) Balloon: tmem relinquished -1 KiB of 20100116 KiB requested. > >> [2011-02-02 10:00:22 8199] DEBUG (balloon:226) Balloon: 13454316 KiB free; 0 to scrub; need 33554432; retries: 115. > >> [2011-02-02 10:00:22 8199] DEBUG (balloon:226) Balloon: 13454316 KiB free; 0 to scrub; need 33554432; retries: 210. > >> > >> is there a way to recover from this? > >> > > > > Did you dedicate fixed amount of memory for dom0? > > Yes. > > > Are you using tmem? > > No. >Ok. Did you disable dom0 ballooning? http://wiki.xensource.com/xenwiki/XenBestPractices It looks like xend is trying to balloon down dom0, which seems to fail and crash things. -- Pasi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Pim van Riezen
2011-Feb-21 10:39 UTC
Re: [Xen-users] Xen 4.0.1 hangs when trying to create vps
On Feb 21, 2011, at 11:19 , Pasi Kärkkäinen wrote:> On Mon, Feb 21, 2011 at 10:52:49AM +0100, Pim van Riezen wrote: >> >> On Feb 21, 2011, at 10:48 , Pasi Kärkkäinen wrote: >> >>> On Wed, Feb 02, 2011 at 02:00:36PM +0100, Pim van Riezen wrote: >>>> Good day, >>>> >>>> we recently migrated to Xen4.0.1 on one of our clusters. It solves a lot of problem, but it also adds one. Whenever we try to xm create a guest that oversteps the total amount of available memory, xend completely hangs itself up and stops responding to any rpc calls.. >>>> >>>> The last lines in xend.log are: >>>> >>>> [2011-02-02 10:00:22 8199] DEBUG (balloon:172) Balloon: tmem relinquished -1 KiB of 20100116 KiB requested. >>>> [2011-02-02 10:00:22 8199] DEBUG (balloon:226) Balloon: 13454316 KiB free; 0 to scrub; need 33554432; retries: 115. >>>> [2011-02-02 10:00:22 8199] DEBUG (balloon:226) Balloon: 13454316 KiB free; 0 to scrub; need 33554432; retries: 210. >>>> >>>> is there a way to recover from this? >>>> >>> >>> Did you dedicate fixed amount of memory for dom0? >> >> Yes. >> >>> Are you using tmem? >> >> No. >> > > Ok. Did you disable dom0 ballooning? > http://wiki.xensource.com/xenwiki/XenBestPractices > > It looks like xend is trying to balloon down dom0, which seems to fail and crash things.This option was indeed not explicitly set. I think htis is a result of our migration from an earlier Xen 3 install. This used to be the documentation for dom0-min-mem in xend-config.sxp: # Dom0 will balloon out when needed to free memory for domU. # dom0-min-mem is the lowest memory level (in MB) dom0 will get down to. # If dom0-min-mem=0, dom0 will never balloon out. i.e., the "no ballooning" used to be explicitly linked to the dom0 min mem and there was no enable-dom0-ballooning. I''ll fix the options in our configuration now, thanks for the heads-up. My only suggestion left is that xend becoming completely unresponsive is probably not the best response to such a configuration foul-up. Should I report that as a possible bug somewhere? Cheers, Pi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Pasi Kärkkäinen
2011-Feb-21 11:04 UTC
Re: [Xen-users] Xen 4.0.1 hangs when trying to create vps
On Mon, Feb 21, 2011 at 11:39:03AM +0100, Pim van Riezen wrote:> > On Feb 21, 2011, at 11:19 , Pasi Kärkkäinen wrote: > > > On Mon, Feb 21, 2011 at 10:52:49AM +0100, Pim van Riezen wrote: > >> > >> On Feb 21, 2011, at 10:48 , Pasi Kärkkäinen wrote: > >> > >>> On Wed, Feb 02, 2011 at 02:00:36PM +0100, Pim van Riezen wrote: > >>>> Good day, > >>>> > >>>> we recently migrated to Xen4.0.1 on one of our clusters. It solves a lot of problem, but it also adds one. Whenever we try to xm create a guest that oversteps the total amount of available memory, xend completely hangs itself up and stops responding to any rpc calls.. > >>>> > >>>> The last lines in xend.log are: > >>>> > >>>> [2011-02-02 10:00:22 8199] DEBUG (balloon:172) Balloon: tmem relinquished -1 KiB of 20100116 KiB requested. > >>>> [2011-02-02 10:00:22 8199] DEBUG (balloon:226) Balloon: 13454316 KiB free; 0 to scrub; need 33554432; retries: 115. > >>>> [2011-02-02 10:00:22 8199] DEBUG (balloon:226) Balloon: 13454316 KiB free; 0 to scrub; need 33554432; retries: 210. > >>>> > >>>> is there a way to recover from this? > >>>> > >>> > >>> Did you dedicate fixed amount of memory for dom0? > >> > >> Yes. > >> > >>> Are you using tmem? > >> > >> No. > >> > > > > Ok. Did you disable dom0 ballooning? > > http://wiki.xensource.com/xenwiki/XenBestPractices > > > > It looks like xend is trying to balloon down dom0, which seems to fail and crash things. > > This option was indeed not explicitly set. I think htis is a result of our migration from an earlier Xen 3 install. This used to be the documentation for dom0-min-mem in xend-config.sxp: > > # Dom0 will balloon out when needed to free memory for domU. > # dom0-min-mem is the lowest memory level (in MB) dom0 will get down to. > # If dom0-min-mem=0, dom0 will never balloon out. > > i.e., the "no ballooning" used to be explicitly linked to the dom0 min mem and there was no enable-dom0-ballooning. > > I''ll fix the options in our configuration now, thanks for the heads-up. > > My only suggestion left is that xend becoming completely unresponsive is probably not the best response to such a configuration foul-up. Should I report that as a possible bug somewhere? >I''m suspecting it''s actually dom0 kernel issue, and not xend issue. If you want to investigate further and track down the actual problem, feel free. -- Pasi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Pim van Riezen
2011-Feb-21 11:34 UTC
Re: [Xen-users] Xen 4.0.1 hangs when trying to create vps
On Feb 21, 2011, at 12:04 , Pasi Kärkkäinen wrote:> I''m suspecting it''s actually dom0 kernel issue, and not xend issue.The rest of the dom0 is fine. All services that accept sockets still work. I can ssh. I can start new services. It''s absolutely only xend that stops working. If I find some time this week I''ll see if I can create a reproducible set-up in our lab. Thanks, Pi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Pasi Kärkkäinen
2011-Feb-21 13:30 UTC
Re: [Xen-users] Xen 4.0.1 hangs when trying to create vps
On Mon, Feb 21, 2011 at 12:34:04PM +0100, Pim van Riezen wrote:> > On Feb 21, 2011, at 12:04 , Pasi Kärkkäinen wrote: > > > I''m suspecting it''s actually dom0 kernel issue, and not xend issue. > > The rest of the dom0 is fine. All services that accept sockets still work. I can ssh. I can start new services. It''s absolutely only xend that stops working. If I find some time this week I''ll see if I can create a reproducible set-up in our lab. >Ok. Please report back if you time to repro it. -- Pasi _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users