Hi, Stephan,
Latest hg tree has set your sEDF scheduler as default, and then
I saw some strange behavior as mentioned in subject on XEN/IA64. The
source tree is based on xen-ia64-unstable.hg set up by Dan, which was
just newly created yesterday and had default scheduler change as above
too.
An example is: (No DomU created)
Total cpu time
IDLE 93def9294f
Dom0 5420288a1b
Shouldn''t IDLE domain not be scheduled for most time? Because
idle task will call into PAL for power save on XEN/IA64, the performance
is really, really bad to boot Dom0. The net effect is about ten times
slower. After adding "sched=bvt", everything back to normal.
I''m not sure whether same problem also exists on x86, maybe not
that obvious. Is there any special sEDF parameter setting to be done
arch specific...?
For previous BVT scheduler, the evt (effective virtual time) for
IDLE task is set as -1, which has ideal effect to always schedule Dom0
as long as it''s runable. IDLE domain should never be scheduled after
Dom0 is up, unless explicit sleep request issued from Dom0.
However after looking into sEDF code, it seems that current
running ED is very likely to be moved into wait queue after a period,
while IDLE is always on run queue. Then regarding case with only Dom0
and IDLE, IDLE will be scheduled more frequently than expected, even
when dom0 doesn''t request to sleep explicitly. When multiple domains
exist, that phenomenon is instead not that obvious, since domains will
gear to run queue alternatively and prevent IDLE to be scheduled.
So maybe some special consideration should be added to IDLE
domain... I''m actually a stranger to this new scheduler, and please
correct me if I throw water on incorrect direction. ;-)
Thanks,
Kevin
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Dan Magenheimer
2005-Jul-08 14:53 UTC
[Xen-devel] Re: IDLE domain is scheduled more than dom0
> Shouldn''t IDLE domain not be scheduled for most time? Because > idle task will call into PAL for power save on XEN/IA64, the performance > is really, really bad to boot Dom0. The net effect is about ten times > slower. After adding "sched=bvt", everything back to normal.If the sedf scheduler is scheduling the idle domain when domain0 is runnable, surely this is affecting performance on x86 also and is a bug that should be fixed? Has anyone done any performance testing (on x86) since sedf was checked in as the default? Dan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Andrew Theurer
2005-Jul-08 16:33 UTC
Re: [Xen-devel] Re: IDLE domain is scheduled more than dom0
On Friday 08 July 2005 09:53, Dan Magenheimer wrote:> > Shouldn''t IDLE domain not be scheduled for most time? Because > > idle task will call into PAL for power save on XEN/IA64, the > > performance is really, really bad to boot Dom0. The net effect is > > about ten times slower. After adding "sched=bvt", everything back > > to normal. > > If the sedf scheduler is scheduling the idle domain when > domain0 is runnable, surely this is affecting performance > on x86 also and is a bug that should be fixed? > > Has anyone done any performance testing (on x86) since > sedf was checked in as the default?Just tried launching some cpu bound tasks in dom0, and I get only 75% cpu util for dom0. I''ll try the other domain scheduler and see if it clears it up. -Andrew cpu0: [075.3] d0-0[075.3] cpu1: [075.3] d0-1[075.3] cpu2: [075.3] d0-2[075.3] cpu3: [074.8] d0-3[074.8] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Andrew Theurer
2005-Jul-08 16:40 UTC
Re: [Xen-devel] Re: IDLE domain is scheduled more than dom0
On Friday 08 July 2005 11:33, Andrew Theurer wrote:> On Friday 08 July 2005 09:53, Dan Magenheimer wrote: > > > Shouldn''t IDLE domain not be scheduled for most time? Because > > > idle task will call into PAL for power save on XEN/IA64, the > > > performance is really, really bad to boot Dom0. The net effect is > > > about ten times slower. After adding "sched=bvt", everything back > > > to normal. > > > > If the sedf scheduler is scheduling the idle domain when > > domain0 is runnable, surely this is affecting performance > > on x86 also and is a bug that should be fixed? > > > > Has anyone done any performance testing (on x86) since > > sedf was checked in as the default? > > Just tried launching some cpu bound tasks in dom0, and I get only 75% > cpu util for dom0. I''ll try the other domain scheduler and see if it > clears it up.OK, just confirmed bvt works as expected in ia32. -Andrew _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
I think it''s quite likely that by default SEDF doesn''t allow domains to use slack CPU time, i.e. non work-conserving. Each domain is given an absolute percentage of the total CPU time. Try use command "xm sedf dom-id 0 0 0 1 0" to give a domain access to slack CPU time. Hopefully this could restore the performance. BTW, on my uniprocessor test machine with latest xen-unstable, xenlinux 2.6.11.12, domU sees significant drop in network throughputs (~40% less!) I''m interested in whether other people encounter similar situations, especially on SMP machines. Cheers, Bin On 7/8/05, Andrew Theurer <habanero@us.ibm.com> wrote:> On Friday 08 July 2005 11:33, Andrew Theurer wrote: > > On Friday 08 July 2005 09:53, Dan Magenheimer wrote: > > > > Shouldn''t IDLE domain not be scheduled for most time? Because > > > > idle task will call into PAL for power save on XEN/IA64, the > > > > performance is really, really bad to boot Dom0. The net effect is > > > > about ten times slower. After adding "sched=bvt", everything back > > > > to normal. > > > > > > If the sedf scheduler is scheduling the idle domain when > > > domain0 is runnable, surely this is affecting performance > > > on x86 also and is a bug that should be fixed? > > > > > > Has anyone done any performance testing (on x86) since > > > sedf was checked in as the default? > > > > Just tried launching some cpu bound tasks in dom0, and I get only 75% > > cpu util for dom0. I''ll try the other domain scheduler and see if it > > clears it up. > > OK, just confirmed bvt works as expected in ia32. > > -Andrew > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Magenheimer, Dan (HP Labs Fort Collins)
2005-Jul-08 19:22 UTC
RE: [Xen-devel] Re: IDLE domain is scheduled more than dom0
Academically, this may not be a bug, but I think its fair to argue that domain0 should be a special case for using "slack" CPU time, at least by default. Indeed, an argument could/should be made that the whole concept of an idle domain should go away entirely. Dan> -----Original Message----- > From: Bin Ren [mailto:bin.ren@gmail.com] > Sent: Friday, July 08, 2005 1:07 PM > To: Andrew Theurer > Cc: Magenheimer, Dan (HP Labs Fort Collins); > xen-devel@lists.xensource.com > Subject: Re: [Xen-devel] Re: IDLE domain is scheduled more than dom0 > > I think it''s quite likely that by default SEDF doesn''t allow domains > to use slack CPU time, i.e. non work-conserving. Each domain is given > an absolute percentage of the total CPU time. Try use command "xm sedf > dom-id 0 0 0 1 0" to give a domain access to slack CPU time. Hopefully > this could restore the performance. > > BTW, on my uniprocessor test machine with latest xen-unstable, > xenlinux 2.6.11.12, domU sees significant drop in network throughputs > (~40% less!) I''m interested in whether other people encounter similar > situations, especially on SMP machines. > > Cheers, > Bin > > On 7/8/05, Andrew Theurer <habanero@us.ibm.com> wrote: > > On Friday 08 July 2005 11:33, Andrew Theurer wrote: > > > On Friday 08 July 2005 09:53, Dan Magenheimer wrote: > > > > > Shouldn''t IDLE domain not be scheduled for most > time? Because > > > > > idle task will call into PAL for power save on XEN/IA64, the > > > > > performance is really, really bad to boot Dom0. The > net effect is > > > > > about ten times slower. After adding "sched=bvt", > everything back > > > > > to normal. > > > > > > > > If the sedf scheduler is scheduling the idle domain when > > > > domain0 is runnable, surely this is affecting performance > > > > on x86 also and is a bug that should be fixed? > > > > > > > > Has anyone done any performance testing (on x86) since > > > > sedf was checked in as the default? > > > > > > Just tried launching some cpu bound tasks in dom0, and I > get only 75% > > > cpu util for dom0. I''ll try the other domain scheduler > and see if it > > > clears it up. > > > > OK, just confirmed bvt works as expected in ia32. > > > > -Andrew > > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xensource.com > > http://lists.xensource.com/xen-devel > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stephan Diestelhorst
2005-Jul-09 18:14 UTC
Re: [Xen-devel] Re: IDLE domain is scheduled more than dom0
Andrew Theurer schrieb:>On Friday 08 July 2005 09:53, Dan Magenheimer wrote: > > >>> Shouldn''t IDLE domain not be scheduled for most time? Because >>>idle task will call into PAL for power save on XEN/IA64, the >>>performance is really, really bad to boot Dom0. The net effect is >>>about ten times slower. After adding "sched=bvt", everything back >>>to normal. >>> >>> >>If the sedf scheduler is scheduling the idle domain when >>domain0 is runnable, surely this is affecting performance >>on x86 also and is a bug that should be fixed? >> >>Has anyone done any performance testing (on x86) since >>sedf was checked in as the default? >> >> > >Just tried launching some cpu bound tasks in dom0, and I get only 75% >cpu util for dom0. I''ll try the other domain scheduler and see if it >clears it up. > >-Andrew > > > cpu0: [075.3] d0-0[075.3] > cpu1: [075.3] d0-1[075.3] > cpu2: [075.3] d0-2[075.3] > cpu3: [074.8] d0-3[074.8] > >This is the current default behaviour of domain 0, it gets 75% of the CPU time and no slack time. This is easily changeable, and I guess it is usefull to do so! Stephan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stephan Diestelhorst
2005-Jul-09 18:17 UTC
Re: [Xen-devel] Re: IDLE domain is scheduled more than dom0
>BTW, on my uniprocessor test machine with latest xen-unstable,>xenlinux 2.6.11.12, domU sees significant drop in network throughputs >(~40% less!) I''m interested in whether other people encounter similar >situations, especially on SMP machines. > > > >Mhmhm, this is interesting, what are the timing parameters for your domains? Thanks, Stephan>On 7/8/05, Andrew Theurer <habanero@us.ibm.com> wrote: > > >>On Friday 08 July 2005 11:33, Andrew Theurer wrote: >> >> >>>On Friday 08 July 2005 09:53, Dan Magenheimer wrote: >>> >>> >>>>> Shouldn''t IDLE domain not be scheduled for most time? Because >>>>>idle task will call into PAL for power save on XEN/IA64, the >>>>>performance is really, really bad to boot Dom0. The net effect is >>>>>about ten times slower. After adding "sched=bvt", everything back >>>>>to normal. >>>>> >>>>> >>>>If the sedf scheduler is scheduling the idle domain when >>>>domain0 is runnable, surely this is affecting performance >>>>on x86 also and is a bug that should be fixed? >>>> >>>>Has anyone done any performance testing (on x86) since >>>>sedf was checked in as the default? >>>> >>>> >>>Just tried launching some cpu bound tasks in dom0, and I get only 75% >>>cpu util for dom0. I''ll try the other domain scheduler and see if it >>>clears it up. >>> >>> >>OK, just confirmed bvt works as expected in ia32. >> >>-Andrew >> >>_______________________________________________ >>Xen-devel mailing list >>Xen-devel@lists.xensource.com >>http://lists.xensource.com/xen-devel >> >> >> > >_______________________________________________ >Xen-devel mailing list >Xen-devel@lists.xensource.com >http://lists.xensource.com/xen-devel > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Default SEDF settings. I have never changed any SEDF parameters. It''s uniprocessor. With some netback driver instrumentation, I find out that SEDF preempts a lot more aggresive even than BVT. For example, when dom1 is sending data to an external host, during each context switch the number of pages dom0 takes off the Tx ring and delivers to the NIC always (yes, 100% of the time) fall between 1-8. With BVT, 70% of batch-sizes fall between 1-8 with the rest scattered around 20-40. In short, there are too frequent preemption/context switches with SEDF on uniprocessors. - Bin On 7/9/05, Stephan Diestelhorst <sd386@cam.ac.uk> wrote:> >BTW, on my uniprocessor test machine with latest xen-unstable, > > >xenlinux 2.6.11.12, domU sees significant drop in network throughputs > >(~40% less!) I''m interested in whether other people encounter similar > >situations, especially on SMP machines. > > > > > > > > > Mhmhm, this is interesting, what are the timing parameters for your domains? > > Thanks, > Stephan > > >On 7/8/05, Andrew Theurer <habanero@us.ibm.com> wrote: > > > > > >>On Friday 08 July 2005 11:33, Andrew Theurer wrote: > >> > >> > >>>On Friday 08 July 2005 09:53, Dan Magenheimer wrote: > >>> > >>> > >>>>> Shouldn''t IDLE domain not be scheduled for most time? Because > >>>>>idle task will call into PAL for power save on XEN/IA64, the > >>>>>performance is really, really bad to boot Dom0. The net effect is > >>>>>about ten times slower. After adding "sched=bvt", everything back > >>>>>to normal. > >>>>> > >>>>> > >>>>If the sedf scheduler is scheduling the idle domain when > >>>>domain0 is runnable, surely this is affecting performance > >>>>on x86 also and is a bug that should be fixed? > >>>> > >>>>Has anyone done any performance testing (on x86) since > >>>>sedf was checked in as the default? > >>>> > >>>> > >>>Just tried launching some cpu bound tasks in dom0, and I get only 75% > >>>cpu util for dom0. I''ll try the other domain scheduler and see if it > >>>clears it up. > >>> > >>> > >>OK, just confirmed bvt works as expected in ia32. > >> > >>-Andrew > >> > >>_______________________________________________ > >>Xen-devel mailing list > >>Xen-devel@lists.xensource.com > >>http://lists.xensource.com/xen-devel > >> > >> > >> > > > >_______________________________________________ > >Xen-devel mailing list > >Xen-devel@lists.xensource.com > >http://lists.xensource.com/xen-devel > > > > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Magenheimer, Dan (HP Labs Fort Collins)
2005-Jul-09 19:32 UTC
RE: [Xen-devel] Re: IDLE domain is scheduled more than dom0
Um, why was the default scheduler changed? Is all this easily fixed or should the default be changed back to bvt until sedf demonstrates equal (or preferably better) performance on some range of benchmarks?> -----Original Message----- > From: Bin Ren [mailto:bin.ren@gmail.com] > Sent: Saturday, July 09, 2005 12:29 PM > To: Stephan Diestelhorst > Cc: Andrew Theurer; Magenheimer, Dan (HP Labs Fort Collins); > xen-devel@lists.xensource.com > Subject: Re: [Xen-devel] Re: IDLE domain is scheduled more than dom0 > > Default SEDF settings. I have never changed any SEDF parameters. It''s > uniprocessor. With some netback driver instrumentation, I find out > that SEDF preempts a lot more aggresive even than BVT. For example, > when dom1 is sending data to an external host, during each context > switch the number of pages dom0 takes off the Tx ring and delivers to > the NIC always (yes, 100% of the time) fall between 1-8. With BVT, 70% > of batch-sizes fall between 1-8 with the rest scattered around 20-40. > In short, there are too frequent preemption/context switches with SEDF > on uniprocessors. > > - Bin > > On 7/9/05, Stephan Diestelhorst <sd386@cam.ac.uk> wrote: > > >BTW, on my uniprocessor test machine with latest xen-unstable, > > > > >xenlinux 2.6.11.12, domU sees significant drop in network > throughputs > > >(~40% less!) I''m interested in whether other people > encounter similar > > >situations, especially on SMP machines. > > > > > > > > > > > > > > Mhmhm, this is interesting, what are the timing parameters > for your domains? > > > > Thanks, > > Stephan > > > > >On 7/8/05, Andrew Theurer <habanero@us.ibm.com> wrote: > > > > > > > > >>On Friday 08 July 2005 11:33, Andrew Theurer wrote: > > >> > > >> > > >>>On Friday 08 July 2005 09:53, Dan Magenheimer wrote: > > >>> > > >>> > > >>>>> Shouldn''t IDLE domain not be scheduled for most time? Because > > >>>>>idle task will call into PAL for power save on XEN/IA64, the > > >>>>>performance is really, really bad to boot Dom0. The > net effect is > > >>>>>about ten times slower. After adding "sched=bvt", > everything back > > >>>>>to normal. > > >>>>> > > >>>>> > > >>>>If the sedf scheduler is scheduling the idle domain when > > >>>>domain0 is runnable, surely this is affecting performance > > >>>>on x86 also and is a bug that should be fixed? > > >>>> > > >>>>Has anyone done any performance testing (on x86) since > > >>>>sedf was checked in as the default? > > >>>> > > >>>> > > >>>Just tried launching some cpu bound tasks in dom0, and I > get only 75% > > >>>cpu util for dom0. I''ll try the other domain scheduler > and see if it > > >>>clears it up. > > >>> > > >>> > > >>OK, just confirmed bvt works as expected in ia32. > > >> > > >>-Andrew > > >> > > >>_______________________________________________ > > >>Xen-devel mailing list > > >>Xen-devel@lists.xensource.com > > >>http://lists.xensource.com/xen-devel > > >> > > >> > > >> > > > > > >_______________________________________________ > > >Xen-devel mailing list > > >Xen-devel@lists.xensource.com > > >http://lists.xensource.com/xen-devel > > > > > > > > > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2005-Jul-10 17:14 UTC
Re: [Xen-devel] Re: IDLE domain is scheduled more than dom0
On 9 Jul 2005, at 19:14, Stephan Diestelhorst wrote:> This is the current default behaviour of domain 0, it gets 75% of the > CPU time and no slack time. This is easily changeable, and I guess it > is > usefull to do so!The default for all domains should be that they are eligible for slack time. Alternatively (or perhaps additionally), CPU time should be given to ineligible domains in preference to idling the CPU. There''s no good reason for the scheduler not to be work conserving, even if no domains are marked as wanting slack time. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ian Pratt
2005-Jul-10 17:17 UTC
RE: [Xen-devel] Re: IDLE domain is scheduled more than dom0
> > cpu0: [075.3] d0-0[075.3] > > cpu1: [075.3] d0-1[075.3] > > cpu2: [075.3] d0-2[075.3] > > cpu3: [074.8] d0-3[074.8] > > > > > This is the current default behaviour of domain 0, it gets > 75% of the CPU time and no slack time. This is easily > changeable, and I guess it is usefull to do so!All domains should certainly be eligible for slack time by default otherwise we''ll just confuse people. If there are parameter tweaks that help dom0 work better as an IO domain that''s fine, but we should at least it make work as expected for computer bound stuff too (not that people should really be doing such stuff in dom0..) [I suspect most people are only interested in using the weight parameter rather than setting actual gurantees anyhow] I suspect the logic behind giving dom0 a large guarantee is that it that when it blocks (which it will frequently) its large guarantee will likely cause it to be scheduled immediately it unblocks, which is "often" good for IO (not always, as promoting batching may be better in other circumstances). We need to get more benchmark numbers that compare sedf and bvt under more realistic workloads. I''ll be particularly interested in reaim and kernel build numbers. Stephan: please can you post a fix to the default parameters. Thanks, Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>From: sd386@hermes.cam.ac.uk [mailto:sd386@hermes.cam.ac.uk] On >Behalf Of Stephan Diestelhorst >Sent: Friday, July 08, 2005 11:15 PM > >> >> An example is: (No DomU created) >> Total cpu time >> IDLE 93def9294f >> Dom0 5420288a1b > >Hi Kevin, > this is quite an interesting result. >In "sedf_add_task" dom0 is set to 15ms every 20ms, so this means thatit gets>75% of the cpu_time. During boottime this might vary due to lots of I/Obut a>ratio of 33% for dom0 and 66% for the idle task is strange. Could youplease>try two things: >a) give the domain extratime by modfying the line >inf->stauts = EXTRA_NONE | SEDF_ASLEEP; >to >inf->status = EXTRA_AWARE | SEDF_ASLEEP; >in "sedf_add_task" "case 0" in sched_sedf.c >This can''t help, with almost same statistics: Total cpu time IDLE 19CE88F3E7 Dom0 1040A91728 Not sure whether that strange coming from the fact that IDLE is also initialized with period as WEIGHT_PERIOD(100ms)?>b) set inf->slice = MILLIESECS(20);Instead this approach has exactly same effect as previous bvt: Total cpu time IDLE 48FD92BF Dom0 2D61AF8D5D Then IDLE wasn''t scheduled any more after Dom0 is up, except explicit request. Though Dom0 is placed on waitq when in the start of do_schedule, it will be moved back to runq when update_queues. The reason is that the start of next period of Dom0 is always earlier than NOW(), when inf->slice == inf->period. Then scheduler is always happy to choose Dom0 instead of IDLE, since both in runq. So, to make b) change can give smooth boot for Dom0, but not sure whether this will affect multiple domains since your intent is to set Dom0 as 75% threshold. Then how about adding a special check in update_queues, upon the case where IDLE is only candidate on runq? In that case, it''s better to pull one from waitq which is run-able there only being slice consumed up, to begin a new round immediately... Thanks, Kevin _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stephan Diestelhorst
2005-Jul-11 12:59 UTC
[Xen-devel] Re: IDLE domain is scheduled more than dom0
> >From: sd386@hermes.cam.ac.uk [mailto:sd386@hermes.cam.ac.uk] On > >Behalf Of Stephan Diestelhorst > >Sent: Friday, July 08, 2005 11:15 PM > > > >> An example is: (No DomU created) > >> Total cpu time > >> IDLE 93def9294f > >> Dom0 5420288a1b > > > >Hi Kevin, > > this is quite an interesting result. > >In "sedf_add_task" dom0 is set to 15ms every 20ms, so this means that > > it gets > > >75% of the cpu_time. During boottime this might vary due to lots of I/O > > but a > > >ratio of 33% for dom0 and 66% for the idle task is strange. Could you > > please > > >try two things: > >a) give the domain extratime by modfying the line > >inf->stauts = EXTRA_NONE | SEDF_ASLEEP; > >to > >inf->status = EXTRA_AWARE | SEDF_ASLEEP; > >in "sedf_add_task" "case 0" in sched_sedf.c > > This can''t help, with almost same statistics: > Total cpu time > IDLE 19CE88F3E7 > Dom0 1040A91728 > > Not sure whether that strange coming from the fact that IDLE is also > initialized with period as WEIGHT_PERIOD(100ms)?This will not matter, and in fact add_domain will not get called for the idle task.> >b) set inf->slice = MILLIESECS(20); > > Instead this approach has exactly same effect as previous bvt: > Total cpu time > IDLE 48FD92BF > Dom0 2D61AF8D5D > > Then IDLE wasn''t scheduled any more after Dom0 is up, except explicit > request. Though Dom0 is placed on waitq when in the start of > do_schedule, it will be moved back to runq when update_queues. The > reason is that the start of next period of Dom0 is always earlier than > NOW(), when inf->slice == inf->period. Then scheduler is always happy to > choose Dom0 instead of IDLE, since both in runq.Okay, thanks for your investigation! The thing is that the idle domain IS the lowest prioritised domain in the system! So if you set the EXTRA_AWARE flag, it should almost never get scheduled. The only case when this will happen is when dom0 does I/O, as this is treated differently whether it occurs in EDF-time or in extra-time. I will have a closer look at this, as this behaviuor is not acceptable.> So, to make b) change can give smooth boot for Dom0, but not sure > whether this will affect multiple domains since your intent is to set > Dom0 as 75% threshold. Then how about adding a special check in > update_queues, upon the case where IDLE is only candidate on runq? In > that case, it''s better to pull one from waitq which is run-able there > only being slice consumed up, to begin a new round immediately...This is exactly what happens if you set the extra-flag! As I said, I will have a closer look on what happens with I/O during extratime. Thanks for your efforts! Stephan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>From: sd386@hermes.cam.ac.uk [mailto:sd386@hermes.cam.ac.uk] On >Behalf Of Stephan Diestelhorst >Sent: Monday, July 11, 2005 7:26 PM >> >> >> Not sure whether that strange coming from the fact that IDLE is also >> initialized with period as WEIGHT_PERIOD(100ms)? >This will not matter, and in fact add_domain will not get called forthe idle>task.Ah, yes, slice of IDLE is always 0, which means arbitration will take effect on every tick when current is IDLE.>> >b) set inf->slice = MILLIESECS(20); > >Okay, thanks for your investigation! >The thing is that the idle domain IS the lowest prioritised domain inthe>system! So if you set the EXTRA_AWARE flag, it should almost never get >scheduled. The only case when this will happen is when dom0 does I/O,as>this >is treated differently whether it occurs in EDF-time or in extra-time.I will>have a closer look at this, as this behaviuor is not acceptable.And one thing I''d like you to know is, my test environment is with XEN/VTI domain0, which is whole unmodified one without any explicit request to do_block. So actually within this model, you can consider Dom0 as always runable... Not sure this different behavior as xen0 will cause any issue in sedf scheduler which, however can be considered as an extreme example for IDLE never to be scheduled. ;-) Thanks, Kevin _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stephan Diestelhorst
2005-Jul-11 14:02 UTC
[Xen-devel] Re: IDLE domain is scheduled more than dom0
On Monday 11 July 2005 14:19, Tian, Kevin wrote:> >From: sd386@hermes.cam.ac.uk [mailto:sd386@hermes.cam.ac.uk] On > >Behalf Of Stephan Diestelhorst > >Sent: Monday, July 11, 2005 7:26 PM > > > >> Not sure whether that strange coming from the fact that IDLE is also > >> initialized with period as WEIGHT_PERIOD(100ms)? > > > >This will not matter, and in fact add_domain will not get called for >> the idle task. > > Ah, yes, slice of IDLE is always 0, which means arbitration will take > effect on every tick when current is IDLE.Not quite. Arbitration points are set by the value of ret.task in do_schedule, for the idle-task this value is estimated by the time the first domain on the waitq gets ready. (Bear in mind that the idle task only runs if dom0 is blocked once you specify the EXTRA_AWARE flag!) If the waitq is empty the idle task runs for a relatively long time (one second), but gets immediately descheduled when the domain wakes up!> >> >b) set inf->slice = MILLIESECS(20); > > > >Okay, thanks for your investigation! > >The thing is that the idle domain IS the lowest prioritised domain in the > >system! So if you set the EXTRA_AWARE flag, it should almost never get > >scheduled. The only case when this will happen is when dom0 does I/O, > > as this is treated differently whether it occurs in EDF-time or in > > extra-time. > > I will have a closer look at this, as this behaviour is not acceptable. > > And one thing I''d like you to know is, my test environment is with > XEN/VTI domain0, which is whole unmodified one without any explicit > request to do_block. So actually within this model, you can consider > Dom0 as always runable... Not sure this different behavior as xen0 will > cause any issue in sedf scheduler which, however can be considered as an > extreme example for IDLE never to be scheduled. ;-)But still it should work! Stephan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stephan Diestelhorst
2005-Jul-11 15:14 UTC
Re: [Xen-devel] Re: IDLE domain is scheduled more than dom0 => [PATCH]
> We need to get more benchmark numbers that compare sedf and bvt under > more realistic workloads. I''ll be particularly interested in reaim and > kernel build numbers.Working on it!> Stephan: please can you post a fix to the default parameters.Attached! Signed-off by: Stephan.Diestelhorst@{cl.cam.ac.uk, inf.tu-dresden.de} Stephan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stephan Diestelhorst
2005-Jul-11 15:30 UTC
[Xen-devel] Re: IDLE domain is scheduled more than dom0
Kevin, I just made my own measurements on my machine (intel dual xeon 2.4 GhZ SMT aka. HT) results: My machine: SEDF: dom0 (15ms/20ms,extra on) dom0 cpu-time after boot: 7796144577 ~ 7.8 sec idle -"- 40933867915 ~ 40.9 sec -> total boot-time: 48.7 sec ratio: 0.191 SEDF: dom0 (5ms/20ms,xon) dom0 cpu-time after boot: 8300470815 ~ 8.3 sec idle -"- 38639540787 ~ 38.6 sec -> total boot-time: 46.9 sec ratio: 0.215 With BVT: dom0 cpu-time after boot: 0x1F20E0DF4 = 8355974644 ~ 8.4 sec idle -"- 0x7D5F045FA = 33654064634 ~ 33.7 sec -> total boot-time: 42.1 sec ratio: 0.248 Kevin Tian''s measurements (on IA64 hardware): SEDF: dom0 (15/20, xoff) Dom0 0x5420288a1b = 361316780571 ~ 361.3 sec IDLE 0x93def9294f = 635101063503 ~ 635.1 sec -> total boot time: 996.4 sec ?! ratio: 0.57 SEDF: dom0(15/20, xon) Dom0 0x1040A91728 = 69804300072 ~ 69.8 sec IDLE 0x19CE88F3E7 = 110839264231 ~ 110.8 sec -> total boot time: 180.6 sec ratio: 0.63 SEDF: dom0(20/20, xoff) Dom0 0x2D61AF8D5D = 194912423261 ~ 194.9 sec IDLE 0x48FD92BF = 1224577727 ~ 1.2 sec -> total boot time: 196.1 sec ratio: 162.42 As you see, for me the fiddling with the parameters of sedf doesn''t make much difference (even to BVT) and the idle-task always has 4-5 times as much CPU time as dom0. In my setup this is due to mounting of NFS devices, which takes quite a while, where dom0 is blocked most of the times. So our times might not be comparable. Did you measure your times at roughly the same stage during / after boot? I''m not quite sure, what causes the weird problems. But I''ll def. look into I/O performance of extra-time, probably these artifacts vanish! Thanks, Stephan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stephan Diestelhorst
2005-Jul-11 15:34 UTC
[Xen-devel] Re: IDLE domain is scheduled more than dom0
Kevin, I just made my own measurements on my machine (intel dual xeon 2.4 GhZ SMT aka. HT) results: My machine: SEDF: dom0 (15ms/20ms,extra on) dom0 cpu-time after boot: 7796144577 ~ 7.8 sec idle -"- 40933867915 ~ 40.9 sec -> total boot-time: 48.7 sec ratio: 0.191 SEDF: dom0 (5ms/20ms,xon) dom0 cpu-time after boot: 8300470815 ~ 8.3 sec idle -"- 38639540787 ~ 38.6 sec -> total boot-time: 46.9 sec ratio: 0.215 With BVT: dom0 cpu-time after boot: 0x1F20E0DF4 = 8355974644 ~ 8.4 sec idle -"- 0x7D5F045FA = 33654064634 ~ 33.7 sec -> total boot-time: 42.1 sec ratio: 0.248 Kevin Tian''s measurements (on IA64 hardware): SEDF: dom0 (15/20, xoff) Dom0 0x5420288a1b = 361316780571 ~ 361.3 sec IDLE 0x93def9294f = 635101063503 ~ 635.1 sec -> total boot time: 996.4 sec ?! ratio: 0.57 SEDF: dom0(15/20, xon) Dom0 0x1040A91728 = 69804300072 ~ 69.8 sec IDLE 0x19CE88F3E7 = 110839264231 ~ 110.8 sec -> total boot time: 180.6 sec ratio: 0.63 SEDF: dom0(20/20, xoff) Dom0 0x2D61AF8D5D = 194912423261 ~ 194.9 sec IDLE 0x48FD92BF = 1224577727 ~ 1.2 sec -> total boot time: 196.1 sec ratio: 162.42 As you see, for me the fiddling with the parameters of sedf doesn''t make much difference (even to BVT) and the idle-task always has 4-5 times as much CPU time as dom0. In my setup this is due to mounting of NFS devices, which takes quite a while, where dom0 is blocked most of the times. So our times might not be comparable. Did you take your samples at a similar stage during / after boot? I find it quite strange that they have such a massive variance! I''ll look into I/O performance during extra-time execution anyways, probably these artifacts will vanish when addressing this issue! Thanks, Stephan _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Tian, Kevin
2005-Jul-12 04:10 UTC
RE: [Xen-devel] Re: IDLE domain is scheduled more than dom0
>From: xen-devel-bounces@lists.xensource.com >[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Stephan >Diestelhorst >Sent: Monday, July 11, 2005 11:34 PM > >Kevin Tian''s measurements (on IA64 hardware): >SEDF: dom0 (15/20, xoff) >Dom0 0x5420288a1b = 361316780571 ~ 361.3 sec >IDLE 0x93def9294f = 635101063503 ~ 635.1 sec >-> total boot time: 996.4 sec ?! > ratio: 0.57This data was collected until login prompt. Currently still some issue with timer, so that value may not be exact accurate. However I did observe that total boot time is about 5 times slower than BVT or the 3rd case, with IDE probe especially slow. I haven''t figure out exact reason yet, and just wonder whether some different behavior on IA64 may exaggerate that issue...> >SEDF: dom0(15/20, xon) >Dom0 0x1040A91728 = 69804300072 ~ 69.8 sec >IDLE 0x19CE88F3E7 = 110839264231 ~ 110.8 sec >-> total boot time: 180.6 sec > ratio: 0.63Sorry for this inaccurate data, upon which I stopped test at IDE probe due to slow progress. Rough sense is similar to first case, and anyhow the ratio is still not acceptable.> >SEDF: dom0(20/20, xoff) >Dom0 0x2D61AF8D5D = 194912423261 ~ 194.9 sec >IDLE 0x48FD92BF = 1224577727 ~ 1.2 sec >-> total boot time: 196.1 sec > ratio: 162.42This one was also collected for whole boot process until login prompt.> >As you see, for me the fiddling with the parameters of sedf doesn''t make much >difference (even to BVT) and the idle-task always has 4-5 times as much CPU >time as dom0. In my setup this is due to mounting of NFS devices, which takes >quite a while, where dom0 is blocked most of the times. So our times might >not be comparable.So yes, they''re not comparable. In your environment, too many I/O of Dom0 gives up time slice actively, which may shade effect when IDLE is scheduled more unexpectedly. However in my test environment, Dom0 never blocks actively even when doing I/O operation (Current status), which can be considered as a special case to make that corner case more obvious... Thanks a lot, Kevin _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Tian, Kevin
2005-Jul-12 11:38 UTC
RE: [Xen-devel] Re: IDLE domain is scheduled more than dom0
Hi, Stephan, I finally track down the problem to be caused by some bug in XEN/IA64 itself. Due to some code not suitable running with IDLE domain, sometimes the delta for next timer tick was changed to a very large value when current context is IDLE. Then IDLE ran a relative long period without timer interrupt happening. Finally evaluated cpu_time for IDLE became large due to above bug, though only be scheduled several times. Similar bugs were hidden previously on bvt scheduler, due to IDLE never be scheduled after DomN was up. With a simple workaround, now IDLE only occupied about 1% time, with Dom0 to grasp 99%. Then I''ll push out a patch to fix that issue soon. Sorry for blaming your scheduler and it actually looks very nice based on data collected by you. ;-) Thanks a lot for help, Kevin>-----Original Message----- >From: xen-devel-bounces@lists.xensource.com >[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Tian, Kevin >Sent: Tuesday, July 12, 2005 12:10 PM >To: Stephan Diestelhorst >Cc: xen-devel@lists.xensource.com >Subject: RE: [Xen-devel] Re: IDLE domain is scheduled more than dom0 > >>From: xen-devel-bounces@lists.xensource.com >>[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Stephan >>Diestelhorst >>Sent: Monday, July 11, 2005 11:34 PM >> >>Kevin Tian''s measurements (on IA64 hardware): >>SEDF: dom0 (15/20, xoff) >>Dom0 0x5420288a1b = 361316780571 ~ 361.3 sec >>IDLE 0x93def9294f = 635101063503 ~ 635.1 sec >>-> total boot time: 996.4 sec ?! >> ratio: 0.57 > >This data was collected until login prompt. Currently still some issue with timer, >so that value may not be exact accurate. However I did observe that total boot >time is about 5 times slower than BVT or the 3rd case, with IDE probe >especially slow. I haven''t figure out exact reason yet, and just wonder whether >some different behavior on IA64 may exaggerate that issue... > >> >>SEDF: dom0(15/20, xon) >>Dom0 0x1040A91728 = 69804300072 ~ 69.8 sec >>IDLE 0x19CE88F3E7 = 110839264231 ~ 110.8 sec >>-> total boot time: 180.6 sec >> ratio: 0.63 > >Sorry for this inaccurate data, upon which I stopped test at IDE probe due to >slow progress. Rough sense is similar to first case, and anyhow the ratio is still >not acceptable. > >> >>SEDF: dom0(20/20, xoff) >>Dom0 0x2D61AF8D5D = 194912423261 ~ 194.9 sec >>IDLE 0x48FD92BF = 1224577727 ~ 1.2 sec >>-> total boot time: 196.1 sec >> ratio: 162.42 > >This one was also collected for whole boot process until login prompt. > >> >>As you see, for me the fiddling with the parameters of sedf doesn''t make >much >>difference (even to BVT) and the idle-task always has 4-5 times as much >CPU >>time as dom0. In my setup this is due to mounting of NFS devices, which >takes >>quite a while, where dom0 is blocked most of the times. So our times might >>not be comparable. > >So yes, they''re not comparable. In your environment, too many I/O of Dom0 >gives up time slice actively, which may shade effect when IDLE is scheduled >more unexpectedly. However in my test environment, Dom0 never blocks >actively even when doing I/O operation (Current status), which can be >considered as a special case to make that corner case more obvious... > >Thanks a lot, >Kevin > >_______________________________________________ >Xen-devel mailing list >Xen-devel@lists.xensource.com >http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Stephan Diestelhorst
2005-Jul-12 12:24 UTC
Re: [Xen-devel] Re: IDLE domain is scheduled more than dom0
Hi Kevin, that''s good news!> Hi, Stephan, > I finally track down the problem to be caused by some bug in XEN/IA64 > itself. Due to some code not suitable running with IDLE domain, sometimes > the delta for next timer tick was changed to a very large value when > current context is IDLE. Then IDLE ran a relative long period without timer > interrupt happening. Finally evaluated cpu_time for IDLE became large due > to above bug, though only be scheduled several times. Similar bugs were > hidden previously on bvt scheduler, due to IDLE never be scheduled after > DomN was up. > > With a simple workaround, now IDLE only occupied about 1% time, with Dom0 > to grasp 99%. Then I''ll push out a patch to fix that issue soon. > Sorry for blaming your scheduler and it actually looks very nice based on > data collected by you. ;-)Don''t worry, similar things happened to me... With boot-times up to 10 minutes, because I didn''t realize that time-outs were relative, not absolute. Resulted in an nice exponential curve :-) Thanks for your testing! Stephan> > >-----Original Message----- > >From: xen-devel-bounces@lists.xensource.com > >[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Tian, Kevin > >Sent: Tuesday, July 12, 2005 12:10 PM > >To: Stephan Diestelhorst > >Cc: xen-devel@lists.xensource.com > >Subject: RE: [Xen-devel] Re: IDLE domain is scheduled more than dom0 > > > >>From: xen-devel-bounces@lists.xensource.com > >>[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Stephan > >>Diestelhorst > >>Sent: Monday, July 11, 2005 11:34 PM > >> > >>Kevin Tian''s measurements (on IA64 hardware): > >>SEDF: dom0 (15/20, xoff) > >>Dom0 0x5420288a1b = 361316780571 ~ 361.3 sec > >>IDLE 0x93def9294f = 635101063503 ~ 635.1 sec > >>-> total boot time: 996.4 sec ?! > >> ratio: 0.57 > > > >This data was collected until login prompt. Currently still some issue > > with timer, so that value may not be exact accurate. However I did > > observe that total boot time is about 5 times slower than BVT or the 3rd > > case, with IDE probe especially slow. I haven''t figure out exact reason > > yet, and just wonder whether some different behavior on IA64 may > > exaggerate that issue... > > > >>SEDF: dom0(15/20, xon) > >>Dom0 0x1040A91728 = 69804300072 ~ 69.8 sec > >>IDLE 0x19CE88F3E7 = 110839264231 ~ 110.8 sec > >>-> total boot time: 180.6 sec > >> ratio: 0.63 > > > >Sorry for this inaccurate data, upon which I stopped test at IDE probe due > > to slow progress. Rough sense is similar to first case, and anyhow the > > ratio is still not acceptable. > > > >>SEDF: dom0(20/20, xoff) > >>Dom0 0x2D61AF8D5D = 194912423261 ~ 194.9 sec > >>IDLE 0x48FD92BF = 1224577727 ~ 1.2 sec > >>-> total boot time: 196.1 sec > >> ratio: 162.42 > > > >This one was also collected for whole boot process until login prompt. > > > >>As you see, for me the fiddling with the parameters of sedf doesn''t make > > > >much > > > >>difference (even to BVT) and the idle-task always has 4-5 times as much > > > >CPU > > > >>time as dom0. In my setup this is due to mounting of NFS devices, which > > > >takes > > > >>quite a while, where dom0 is blocked most of the times. So our times > >> might not be comparable. > > > >So yes, they''re not comparable. In your environment, too many I/O of Dom0 > >gives up time slice actively, which may shade effect when IDLE is > > scheduled more unexpectedly. However in my test environment, Dom0 never > > blocks actively even when doing I/O operation (Current status), which can > > be considered as a special case to make that corner case more obvious... > > > >Thanks a lot, > >Kevin > > > >_______________________________________________ > >Xen-devel mailing list > >Xen-devel@lists.xensource.com > >http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel