We found that cs 11554 increases some of timeout to fix the hot plug issue when starting 4 VMX simultaneously. Thanks for the quick reponse to nightly status report. But we found that when in an extremely condition, like creating 12 VMX domains and 4 XENU domains at the same time, sometimes we still can meet this bug. Attached is a demo script to start many domains. So we just wonder changing the timout value could not resolve this issue completely. At the same time, we found when main_loop_wait is increased to 100 ms, windows guest''s keyboard and mouse response is significantly slower. Ping _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 26/9/06 8:36 am, "Yu, Ping Y" <ping.y.yu@intel.com> wrote:> But we found that when in an extremely condition, like > creating 12 VMX domains and 4 XENU domains at the same > time, sometimes we still can meet this bug. Attached is a > demo script to start many domains. So we just wonder changing > the timout value could not resolve this issue completely.How long do the creation commands appear to hang before timing out? With the new timeouts it ought to be at least a minute, which ought to be plenty of time to create a dozen domains.> At the same time, we found when main_loop_wait is increased to > 100 ms, windows guest''s keyboard and mouse response is > significantly slower.Perhaps we need a single polling daemon that can demux kb/mouse events to all qemu instances in a way that can be select()ed on. That would avoid unscalable polling overheads. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Any extreme case is going to stress the crap out of the dom0 hotplug subsystem; it''s not clear it was designed for the sort of workloads xen puts on it :-( BTW, your attachment was stripped, but I think we can probably work out what it does. cheers, S. ----- Original Message ----- From: "Yu, Ping Y" <ping.y.yu@intel.com> To: <xen-devel@lists.xensource.com> Cc: <Christian.Limpach@xensource.com>; <sos22@cam.ac.uk>; "Steven Hand" <steven.hand@cl.cam.ac.uk> Sent: Tuesday, September 26, 2006 8:36 AM Subject: about hot plug issue We found that cs 11554 increases some of timeout to fix the hot plug issue when starting 4 VMX simultaneously. Thanks for the quick reponse to nightly status report. But we found that when in an extremely condition, like creating 12 VMX domains and 4 XENU domains at the same time, sometimes we still can meet this bug. Attached is a demo script to start many domains. So we just wonder changing the timout value could not resolve this issue completely. At the same time, we found when main_loop_wait is increased to 100 ms, windows guest''s keyboard and mouse response is significantly slower. Ping _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Yeah, in fact, the script is quite simple, and it is just trying to create many VMXs or XENUs at the same time. Another issue we found is that windows guest are not adapt with the new timeout value. #!/bin/bash xm vcpu-set 0 1 sleep 5 xm cr config.vmx1 & xm cr config.vmx2 & xm cr config.vmx3 & xm cr config.vmx4 & xm cr config.vmx5 & xm cr config.vmx6 & xm cr config.vmx7 & xm cr config.vmx8 & xm cr config.vmx9 & xm cr config.vmx10 & xm cr config.vmx11 & xm cr config.vmx12 & xterm -display :1 -e "xm cr -c config.xenu1" & xterm -display :1 -e "xm cr -c config.xenu2" & xterm -display :1 -e "xm cr -c config.xenu3" & xterm -display :1 -e "xm cr -c config.xenu4" & Ping>-----Original Message----- >From: Steven M. Hand [mailto:smh22@hermes.cam.ac.uk] On Behalf Of Steven Hand >Sent: 2006年9月26日 15:48 >To: Yu, Ping Y; xen-devel@lists.xensource.com >Cc: Christian.Limpach@xensource.com; sos22@cam.ac.uk >Subject: Re: about hot plug issue > >Any extreme case is going to stress the crap out of the dom0 >hotplug subsystem; it''s not clear it was designed for the sort >of workloads xen puts on it :-( > >BTW, your attachment was stripped, but I think we can probably >work out what it does. > >cheers, > >S. > > >----- Original Message ----- >From: "Yu, Ping Y" <ping.y.yu@intel.com> >To: <xen-devel@lists.xensource.com> >Cc: <Christian.Limpach@xensource.com>; <sos22@cam.ac.uk>; "Steven Hand" ><steven.hand@cl.cam.ac.uk> >Sent: Tuesday, September 26, 2006 8:36 AM >Subject: about hot plug issue > > >We found that cs 11554 increases some of timeout to fix >the hot plug issue when starting 4 VMX simultaneously. >Thanks for the quick reponse to nightly status report. > >But we found that when in an extremely condition, like >creating 12 VMX domains and 4 XENU domains at the same >time, sometimes we still can meet this bug. Attached is a >demo script to start many domains. So we just wonder changing >the timout value could not resolve this issue completely. >At the same time, we found when main_loop_wait is increased to >100 ms, windows guest''s keyboard and mouse response is >significantly slower. > >Ping_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> >How long do the creation commands appear to hang before timing out?With the>new timeouts it ought to be at least a minute, which ought to be plentyof>time to create a dozen domains.The failure time is about 140 second.> >> At the same time, we found when main_loop_wait is increased to >> 100 ms, windows guest''s keyboard and mouse response is >> significantly slower. > >Perhaps we need a single polling daemon that can demux kb/mouse eventsto>all qemu instances in a way that can be select()ed on. That would avoid >unscalable polling overheads.Good solution. Could you make a patch for this? :-)> > -- Keir > > > >_______________________________________________ >Xen-devel mailing list >Xen-devel@lists.xensource.com >http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 26/9/06 09:26, "Yu, Ping Y" <ping.y.yu@intel.com> wrote:>> How long do the creation commands appear to hang before timing out? > With the >> new timeouts it ought to be at least a minute, which ought to be plenty > of >> time to create a dozen domains. > The failure time is about 140 second.Well, dom0 is probably hosed for those 140 seconds and you''ve reached the limit of work that can be done in single-VCPU domain 0 in that amount of time. Also, a user is not going to want to wait an unbounded length of time for his request to be processed -- it seems reasonable to time out, but the error should probably be ''too busy'' rather than ''unknown fatal error''. And the timeout should be more easily user-adjustable (2 minutes is a long time for a user to stare at an idle console). Or we could work out how we are chewing up dom0 for 10s per created domain. Is the total time to create 12 domains less if you create them one after another (serially) rather than issuing the xm requests all at the same time? I wonder if creating many domains in parallel is thrashing domain 0 somehow.>>> At the same time, we found when main_loop_wait is increased to >>> 100 ms, windows guest''s keyboard and mouse response is >>> significantly slower. >> >> Perhaps we need a single polling daemon that can demux kb/mouse events > to >> all qemu instances in a way that can be select()ed on. That would avoid >> unscalable polling overheads. > Good solution. Could you make a patch for this? :-)I need to discuss with other Cambridge folks. Moving to qemu stub domain and split driver for kb/mouse input will solve this issue anyway in the longer term. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>Or we could work out how we are chewing up dom0 for 10s per createddomain.>Is the total time to create 12 domains less if you create them oneafter>another (serially) rather than issuing the xm requests all at the sametime?>I wonder if creating many domains in parallel is thrashing domain 0somehow. We have tried to start 14 domains in serial, and the total time is 2 minutes 21 seconds, and if staring in parallel, the total time is 1 minutes 51 second, counting from "xm create" to message "Started domain domain_name) Hope this info is useful to you. :-)> >>>> At the same time, we found when main_loop_wait is increased to >>>> 100 ms, windows guest''s keyboard and mouse response is >>>> significantly slower. >>> >>> Perhaps we need a single polling daemon that can demux kb/mouseevents>> to >>> all qemu instances in a way that can be select()ed on. That wouldavoid>>> unscalable polling overheads. >> Good solution. Could you make a patch for this? :-) > >I need to discuss with other Cambridge folks. Moving to qemu stubdomain and>split driver for kb/mouse input will solve this issue anyway in thelonger>term. > > -- Keir_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 26/9/06 11:20, "Yu, Ping Y" <ping.y.yu@intel.com> wrote:>> Or we could work out how we are chewing up dom0 for 10s per created > domain. >> Is the total time to create 12 domains less if you create them one > after >> another (serially) rather than issuing the xm requests all at the same > time? >> I wonder if creating many domains in parallel is thrashing domain 0 > somehow. > > We have tried to start 14 domains in serial, and the total time is 2 > minutes > 21 seconds, and if staring in parallel, the total time is 1 minutes 51 > second, > counting from "xm create" to message "Started domain domain_name) > Hope this info is useful to you. :-)Well, it means we''re not thrashing dom0 by doing parallel creates. That means the answer (if we care) is to make HVM guest creation less slow. :-) -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel