Sean Dague
2005-Sep-14 18:53 UTC
[Xen-devel] extremely long xend startup time causing /etc/init.d/xendomains not to run?
I''ve started to notice the following issue, and would like to see if anyone else has seen it before digging too deep. When rebooting a xen system that has auto starting domains, they mostly don''t start. On one system that tries to start 5 DomU instances, it appears only the last one (alphabetically) ends up starting. On the system with only 2 DomU instances, nothing starts. Neither of these are very speedy machines (700 & 800 Mhz respectively), so a timing problem with /etc/init.d/xend starting is probably exacerbated in this environment. Sanity confirmation on this would be really useful, so if anyone else has seen this please also post back. :) -Sean -- __________________________________________________________________ Sean Dague Mid-Hudson Valley sean at dague dot net Linux Users Group http://dague.net http://mhvlug.org There is no silver bullet. Plus, werewolves make better neighbors than zombies, and they tend to keep the vampire population down. __________________________________________________________________ _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Anthony Liguori
2005-Sep-14 19:04 UTC
Re: [Xen-devel] extremely long xend startup time causing /etc/init.d/xendomains not to run?
Sean Dague wrote:>I''ve started to notice the following issue, and would like to see if anyone >else has seen it before digging too deep. > >When rebooting a xen system that has auto starting domains, they mostly >don''t start. > >On one system that tries to start 5 DomU instances, it appears only the last >one (alphabetically) ends up starting. On the system with only 2 DomU >instances, nothing starts. Neither of these are very speedy machines (700 & >800 Mhz respectively), so a timing problem with /etc/init.d/xend starting is >probably exacerbated in this environment. > >Sanity confirmation on this would be really useful, so if anyone else has >seen this please also post back. :) > >I see this a lot in QEMU. xm is unresponsive for a bit after xend starts. We really need to make sure that xend start does not return until xend is ready to receive connections. We also have to look at making the xend startup time a bit faster. Regards, Anthony Liguori> -Sean > > > >------------------------------------------------------------------------ > >_______________________________________________ >Xen-devel mailing list >Xen-devel@lists.xensource.com >http://lists.xensource.com/xen-devel > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ryan Harper
2005-Sep-14 19:13 UTC
Re: [Xen-devel] extremely long xend startup time causing /etc/init.d/xendomains not to run?
* Sean Dague <sean@dague.net> [2005-09-14 13:55]:> I''ve started to notice the following issue, and would like to see if anyone > else has seen it before digging too deep. > > When rebooting a xen system that has auto starting domains, they mostly > don''t start. > > On one system that tries to start 5 DomU instances, it appears only the last > one (alphabetically) ends up starting. On the system with only 2 DomU > instances, nothing starts. Neither of these are very speedy machines (700 & > 800 Mhz respectively), so a timing problem with /etc/init.d/xend starting is > probably exacerbated in this environment. > > Sanity confirmation on this would be really useful, so if anyone else has > seen this please also post back. :)I''ve seen this a lot when working bug#197 As the number of entries in /var/lib/xenstored/store/domain/$uuid increase, the time for xend to get up and running increases. When I had over 500 stale uuid directories in the store, it took several *minutes* for xend to full come up, and /var/lib/xenstored was several hundred MB. The only workaround I know is to: /etc/init.d/xend stop killall xenstored xenconsoled rm -rf /var/lib/xenstored/* /etc/init.d/xend start -- Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx (512) 838-9253 T/L: 678-9253 ryanh@us.ibm.com _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Sean Dague
2005-Sep-14 20:18 UTC
[Xen-devel] Re: extremely long xend startup time causing /etc/init.d/xendomains not to run?
On Wed, Sep 14, 2005 at 02:13:14PM -0500, Ryan Harper wrote:> * Sean Dague <sean@dague.net> [2005-09-14 13:55]: > > I''ve started to notice the following issue, and would like to see if anyone > > else has seen it before digging too deep. > > > > When rebooting a xen system that has auto starting domains, they mostly > > don''t start. > > > > On one system that tries to start 5 DomU instances, it appears only the last > > one (alphabetically) ends up starting. On the system with only 2 DomU > > instances, nothing starts. Neither of these are very speedy machines (700 & > > 800 Mhz respectively), so a timing problem with /etc/init.d/xend starting is > > probably exacerbated in this environment. > > > > Sanity confirmation on this would be really useful, so if anyone else has > > seen this please also post back. :) > > I''ve seen this a lot when working bug#197 As the number of entries in > /var/lib/xenstored/store/domain/$uuid increase, the time for xend to get > up and running increases. When I had over 500 stale uuid directories in > the store, it took several *minutes* for xend to full come up, and > /var/lib/xenstored was several hundred MB. > > The only workaround I know is to: > > /etc/init.d/xend stop > killall xenstored xenconsoled > rm -rf /var/lib/xenstored/* > /etc/init.d/xend startI tried that, and nstartup still has xend in an odd state after about 15 minutes. It looks like a few domains attempted to autostart, but never made it. I''ll compile this into a bugzilla tonight. -Sean -- __________________________________________________________________ Sean Dague Mid-Hudson Valley sean at dague dot net Linux Users Group http://dague.net http://mhvlug.org There is no silver bullet. Plus, werewolves make better neighbors than zombies, and they tend to keep the vampire population down. __________________________________________________________________ _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Anthony Liguori
2005-Sep-15 01:44 UTC
[Xen-devel] Re: [Xen-tools] Re: extremely long xend startup time causing /etc/init.d/xendomains not to run?
Sean Dague wrote:>>/etc/init.d/xend stop >>killall xenstored xenconsoled >>rm -rf /var/lib/xenstored/* >>/etc/init.d/xend start >> >> > >I tried that, and nstartup still has xend in an odd state after about 15 >minutes. It looks like a few domains attempted to autostart, but never >made it. I''ll compile this into a bugzilla tonight. > >You should try with my latest patches. Without them, I don''t think there''s much hope that the xendomains script would work on a slower system. That doesn''t mean that I think it will work, but rather that there is at least a chance now :-) Regards, Anthony Liguori> -Sean > > > >------------------------------------------------------------------------ > >_______________________________________________ >Xen-tools mailing list >Xen-tools@lists.xensource.com >http://lists.xensource.com/xen-tools >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Sean Dague
2005-Sep-15 03:55 UTC
[Xen-devel] Re: [Xen-tools] Re: extremely long xend startup time causing /etc/init.d/xendomains not to run?
On Wed, Sep 14, 2005 at 08:44:40PM -0500, Anthony Liguori wrote:> Sean Dague wrote: > > >>/etc/init.d/xend stop > >>killall xenstored xenconsoled > >>rm -rf /var/lib/xenstored/* > >>/etc/init.d/xend start > >> > >> > > > >I tried that, and nstartup still has xend in an odd state after about 15 > >minutes. It looks like a few domains attempted to autostart, but never > >made it. I''ll compile this into a bugzilla tonight. > > > > > You should try with my latest patches. Without them, I don''t think > there''s much hope that the xendomains script would work on a slower system. > > That doesn''t mean that I think it will work, but rather that there is at > least a chance now :-)Well, that part works now. My 5 DomU machine now actually starts 5 DomUs on boot! :) Thanks Anthony! Networking is hosed in some way still, but it''s been a couple weeks since I brought this box up to sync so that may just require some fiddling. -Sean -- __________________________________________________________________ Sean Dague Mid-Hudson Valley sean at dague dot net Linux Users Group http://dague.net http://mhvlug.org There is no silver bullet. Plus, werewolves make better neighbors than zombies, and they tend to keep the vampire population down. __________________________________________________________________ _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel