Hi, i am using XEN on a voice mailbox system. One domU sends RTP datagrams every 20ms. Using the sedf scheduler it is easy to setup a domain which meets this criteria. I am disappointed by the announcement that sedf will be removed from future releases. Under credit the time slicing of 30ms and the fairness among the VCPU hurts my system. Multi core cpus and pinning each VCPU to a real CPU can solve this problem, but this will waste resources as real time apps/domUs normally are not CPU intensive. Why will sedf be removed from XEN? How will XEN support soft real-time needs in future? Was not the idea of having different schedulers the solution? regards, Markus _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi, Markus At first, did you see this thread? http://lists.xensource.com/archives/html/xen-devel/2006-11/msg00198.html Latest version of credit scheduler, Runq is split to I/O intensive and CPU intensive. (I/O have high priority.) If your machine has just one I/O intensive jobs are exists, It should be no problem. Anyway, you should try latest xen-unstable at first. Thanks Atsushi SAKAI>Hi, >i am using XEN on a voice mailbox system. >One domU sends RTP datagrams every 20ms. >Using the sedf scheduler it is easy to setup a domain which meets this >criteria. > >I am disappointed by the announcement that sedf will be removed from future >releases. >Under credit the time slicing of 30ms and the fairness among the VCPU hurts >my system. > >Multi core cpus and pinning each VCPU to a real CPU can solve this problem, >but this will waste resources as real time apps/domUs normally are not CPU >intensive. > >Why will sedf be removed from XEN? > >How will XEN support soft real-time needs in future? > >Was not the idea of having different schedulers the solution? > >regards, >Markus>_______________________________________________ >Xen-devel mailing list >Xen-devel@lists.xensource.com >http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi Markus. As Atsushi mentioned, if you''re using the credit scheduler, I/O intensive domUs will preempt CPU intensive ones immediately when they become runnable. They will not wait a full time slice. Well, at least this is true with recent xen-unstable. :-) Also, the "fairness" between domains can be tuned with the per domain "weight". If you assign twice the weight to domU1 as you do to domU2, domU1 will be allowed to use twice as much CPU resources as domU2 and will preempt domU2 while it is behind this target. If domU1 isn''t consuming lots of CPU, its weight shouldn''t matter much though because it will preempt other domains as explained above. Have you actually tried to run your application with the credit scheduler? Based on your description, I theorize it should work and would be interested to look at scheduler traces if you encounter any problems. Keir already addressed the SEDF question. I think soft real time support is a desirable thing. If you actually look at what the SEDF scheduler does though, it''s not always obeying the administrator''s orders or doing a very efficient job with simple workloads. It would be interesting to compare scheduler traces with your SEDF setup and the credit scheduler and see if, how, and why SEDF might be scheduling things better for you. There are certainly simple things we could do in credit to improve soft real time guarantees if needed: - make the system default time slice tuneable. - make per domain time-slice tuneable - introduce strict priority based scheduling (like the current I/O boost but tuneable per domain by the admin) - Use the strict priority mechanism to offer soft real time time-slice and wake-to-run latency best effort guarantees. We can also fix a number of problems with SEDF and continue maintaining it. In general, the ability to run multiple schedulers is a good thing. On very large hosts, I would even envision running multiple instances of the same scheduler -- potentially with different configurations -- on different "soft partitions" of the system. I think it would be very helpful to better understand the workloads we care about by studying scheduler traces. Do you want to generate some for your workload and share them? Traces for both SEDF and credit running on a recent version of unstable would be the best. If you need help generating these, I''d be happy to help. I''m especially interested to address any bugs related to an I/O intensive domain not getting ahold of a CPU under 20ms. With a recent build of xen-untable, I think the credit scheduler should make this happen for you a lot quicker than 20ms (more like < 1 ms)! Cheers, Emmanuel. On Thu, Nov 16, 2006 at 09:13:39AM +0100, Markus Kremer wrote:> Hi, > i am using XEN on a voice mailbox system. > One domU sends RTP datagrams every 20ms. > Using the sedf scheduler it is easy to setup a domain which meets this > criteria. > > I am disappointed by the announcement that sedf will be removed from > future releases. > Under credit the time slicing of 30ms and the fairness among the VCPU > hurts my system. > > Multi core cpus and pinning each VCPU to a real CPU can solve this > problem, but this will waste resources as real time apps/domUs normally > are not CPU intensive. > > Why will sedf be removed from XEN? > > How will XEN support soft real-time needs in future? > > Was not the idea of having different schedulers the solution? > > regards, > Markus> _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi Emmanuel, I am currently running the stable XEN 3.0.3 and previously 3.0.1 with the sedf scheduler. A quick test with the credit scheduler of 3.0.3 showed some long delays. I will try to reproduce the tests with the latest unstable. How do i enable the scheduler traces? thanks, Markus On 11/16/06, Emmanuel Ackaouy <ack@xensource.com> wrote:> > Hi Markus. > > As Atsushi mentioned, if you''re using the credit scheduler, > I/O intensive domUs will preempt CPU intensive ones immediately > when they become runnable. They will not wait a full time slice. > Well, at least this is true with recent xen-unstable. :-) > > Also, the "fairness" between domains can be tuned with the per > domain "weight". If you assign twice the weight to domU1 as you > do to domU2, domU1 will be allowed to use twice as much CPU > resources as domU2 and will preempt domU2 while it is behind > this target. If domU1 isn''t consuming lots of CPU, its weight > shouldn''t matter much though because it will preempt other > domains as explained above. > > Have you actually tried to run your application with the credit > scheduler? Based on your description, I theorize it should work > and would be interested to look at scheduler traces if you > encounter any problems. > > Keir already addressed the SEDF question. I think soft real > time support is a desirable thing. If you actually look at > what the SEDF scheduler does though, it''s not always obeying > the administrator''s orders or doing a very efficient job with > simple workloads. > > It would be interesting to compare scheduler traces with your > SEDF setup and the credit scheduler and see if, how, and why > SEDF might be scheduling things better for you. > > There are certainly simple things we could do in credit to > improve soft real time guarantees if needed: > - make the system default time slice tuneable. > - make per domain time-slice tuneable > - introduce strict priority based scheduling (like the current > I/O boost but tuneable per domain by the admin) > - Use the strict priority mechanism to offer soft real time > time-slice and wake-to-run latency best effort guarantees. > > We can also fix a number of problems with SEDF and continue > maintaining it. In general, the ability to run multiple > schedulers is a good thing. On very large hosts, I would > even envision running multiple instances of the same scheduler > -- potentially with different configurations -- on different > "soft partitions" of the system. > > I think it would be very helpful to better understand the > workloads we care about by studying scheduler traces. Do > you want to generate some for your workload and share them? > Traces for both SEDF and credit running on a recent version > of unstable would be the best. If you need help generating > these, I''d be happy to help. I''m especially interested to > address any bugs related to an I/O intensive domain not > getting ahold of a CPU under 20ms. With a recent build of > xen-untable, I think the credit scheduler should make this > happen for you a lot quicker than 20ms (more like < 1 ms)! > > Cheers, > Emmanuel. > > On Thu, Nov 16, 2006 at 09:13:39AM +0100, Markus Kremer wrote: > > Hi, > > i am using XEN on a voice mailbox system. > > One domU sends RTP datagrams every 20ms. > > Using the sedf scheduler it is easy to setup a domain which meets > this > > criteria. > > > > I am disappointed by the announcement that sedf will be removed from > > future releases. > > Under credit the time slicing of 30ms and the fairness among the VCPU > > hurts my system. > > > > Multi core cpus and pinning each VCPU to a real CPU can solve this > > problem, but this will waste resources as real time apps/domUs > normally > > are not CPU intensive. > > > > Why will sedf be removed from XEN? > > > > How will XEN support soft real-time needs in future? > > > > Was not the idea of having different schedulers the solution? > > > > regards, > > Markus > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xensource.com > > http://lists.xensource.com/xen-devel > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi,
Can I used vcpu-pin to avoid CPU scheduler overhead? Suppose I have four CPUs,
if I used xm vcpu-pin to fix the mapping of each physcial CPU to virtual CPU. So
credit scheduler will not be needed.
Liang
----- Original Message -----
From: Markus Kremer
To: Markus Kremer ; xen-devel
Sent: Thursday, November 16, 2006 7:49 AM
Subject: Re: [Xen-devel] credit vs sedf scheduler
Hi Emmanuel,
I am currently running the stable XEN 3.0.3 and previously 3.0.1 with the sedf
scheduler.
A quick test with the credit scheduler of 3.0.3 showed some long delays.
I will try to reproduce the tests with the latest unstable.
How do i enable the scheduler traces?
thanks,
Markus
On 11/16/06, Emmanuel Ackaouy <ack@xensource.com > wrote:
Hi Markus.
As Atsushi mentioned, if you''re using the credit scheduler,
I/O intensive domUs will preempt CPU intensive ones immediately
when they become runnable. They will not wait a full time slice.
Well, at least this is true with recent xen-unstable. :-)
Also, the "fairness" between domains can be tuned with the per
domain "weight". If you assign twice the weight to domU1 as you
do to domU2, domU1 will be allowed to use twice as much CPU
resources as domU2 and will preempt domU2 while it is behind
this target. If domU1 isn''t consuming lots of CPU, its weight
shouldn''t matter much though because it will preempt other
domains as explained above.
Have you actually tried to run your application with the credit
scheduler? Based on your description, I theorize it should work
and would be interested to look at scheduler traces if you
encounter any problems.
Keir already addressed the SEDF question. I think soft real
time support is a desirable thing. If you actually look at
what the SEDF scheduler does though, it''s not always obeying
the administrator''s orders or doing a very efficient job with
simple workloads.
It would be interesting to compare scheduler traces with your
SEDF setup and the credit scheduler and see if, how, and why
SEDF might be scheduling things better for you.
There are certainly simple things we could do in credit to
improve soft real time guarantees if needed:
- make the system default time slice tuneable.
- make per domain time-slice tuneable
- introduce strict priority based scheduling (like the current
I/O boost but tuneable per domain by the admin)
- Use the strict priority mechanism to offer soft real time
time-slice and wake-to-run latency best effort guarantees.
We can also fix a number of problems with SEDF and continue
maintaining it. In general, the ability to run multiple
schedulers is a good thing. On very large hosts, I would
even envision running multiple instances of the same scheduler
-- potentially with different configurations -- on different
"soft partitions" of the system.
I think it would be very helpful to better understand the
workloads we care about by studying scheduler traces. Do
you want to generate some for your workload and share them?
Traces for both SEDF and credit running on a recent version
of unstable would be the best. If you need help generating
these, I''d be happy to help. I''m especially interested to
address any bugs related to an I/O intensive domain not
getting ahold of a CPU under 20ms. With a recent build of
xen-untable, I think the credit scheduler should make this
happen for you a lot quicker than 20ms (more like < 1 ms)!
Cheers,
Emmanuel.
On Thu, Nov 16, 2006 at 09:13:39AM +0100, Markus Kremer wrote:
> Hi,
> i am using XEN on a voice mailbox system.
> One domU sends RTP datagrams every 20ms.
> Using the sedf scheduler it is easy to setup a domain which meets
this
> criteria.
>
> I am disappointed by the announcement that sedf will be removed from
> future releases.
> Under credit the time slicing of 30ms and the fairness among the
VCPU
> hurts my system.
>
> Multi core cpus and pinning each VCPU to a real CPU can solve this
> problem, but this will waste resources as real time apps/domUs
normally
> are not CPU intensive.
>
> Why will sedf be removed from XEN?
>
> How will XEN support soft real-time needs in future?
>
> Was not the idea of having different schedulers the solution?
>
> regards,
> Markus
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
------------------------------------------------------------------------------
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
On Thu, Nov 16, 2006 at 09:29:30AM -0700, Liang Yang wrote:> Hi, > > Can I used vcpu-pin to avoid CPU scheduler overhead? Suppose I have four > CPUs, if I used xm vcpu-pin to fix the mapping of each physcial CPU to > virtual CPU. So credit scheduler will not be needed. > > LiangPinning all your VCPUs will not stop Xen from executing various scheduler code paths. Why are you worried about the scheduler overhead? It should be negligeable. Emmanuel. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi Emmanuel, Do you mean, if the number of VCPUs is equal to the number of physcial CPUs, Xen will still call credit scheduler to allocate CPU resources amongst V CPUs even though each guest domain is assigned a fixed VCPU/CPU using xm vcpu-pin? The purpose I assign each domain a CPU and fix the mapping of VCPU to CPU is I want each guest domain get equal CPU resources at any time. I don''t want to credit scheduler get involved. Regards, Liang ----- Original Message ----- From: "Emmanuel Ackaouy" <ack@xensource.com> To: "Liang Yang" <multisyncfe991@hotmail.com> Cc: "xen-devel" <xen-devel@lists.xensource.com> Sent: Thursday, November 16, 2006 9:37 AM Subject: Re: [Xen-devel] credit vs sedf scheduler> On Thu, Nov 16, 2006 at 09:29:30AM -0700, Liang Yang wrote: >> Hi, >> >> Can I used vcpu-pin to avoid CPU scheduler overhead? Suppose I have >> four >> CPUs, if I used xm vcpu-pin to fix the mapping of each physcial CPU to >> virtual CPU. So credit scheduler will not be needed. >> >> Liang > > Pinning all your VCPUs will not stop Xen from executing various > scheduler code paths. > > Why are you worried about the scheduler overhead? It should be > negligeable. > > Emmanuel. >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Thu, Nov 16, 2006 at 10:08:40AM -0700, Liang Yang wrote:> Hi Emmanuel, > > Do you mean, if the number of VCPUs is equal to the number of physcial > CPUs, Xen will still call credit scheduler to allocate CPU resources > amongst V CPUs even though each guest domain is assigned a fixed VCPU/CPU > using xm vcpu-pin?What I mean is that: 1- The clock interrupt will still run every 10ms and this will run some scheduler accounting code. 2- After a VCPU completes its time slice, Xen will call into the scheduler code to see if anything else needs to run. Of course, in this case, there will be nothing else to run so the VCPU will continue running. Both of these should be absolutely negligeable. Are you seeing overhead which is not negligeable? If so, this would be a bug and should be fixed.> The purpose I assign each domain a CPU and fix the mapping of VCPU to CPU > is I want each guest domain get equal CPU resources at any time. I don''t > want to credit scheduler get involved.If your goal is to have each domain get equal CPU at any given time, then you don''t need to do anything. The scheduler will automatically spread your VCPUs across the 4 physical CPUs and they will each get an equal amount of physical CPU resources. Also keep in mind that if a 5th domain (dom0 for example) needs CPU resources as well, the credit scheduler will do this fairly and also with negligeable costs whereas your scheme to pin all VCPUs on their own physical CPU will not. Cheers, Emmanuel.> Regards, > > Liang > ----- Original Message ----- > From: "Emmanuel Ackaouy" <ack@xensource.com> > To: "Liang Yang" <multisyncfe991@hotmail.com> > Cc: "xen-devel" <xen-devel@lists.xensource.com> > Sent: Thursday, November 16, 2006 9:37 AM > Subject: Re: [Xen-devel] credit vs sedf scheduler > > > >On Thu, Nov 16, 2006 at 09:29:30AM -0700, Liang Yang wrote: > >> Hi, > >> > >> Can I used vcpu-pin to avoid CPU scheduler overhead? Suppose I have > >>four > >> CPUs, if I used xm vcpu-pin to fix the mapping of each physcial CPU to > >> virtual CPU. So credit scheduler will not be needed. > >> > >> Liang > > > >Pinning all your VCPUs will not stop Xen from executing various > >scheduler code paths. > > > >Why are you worried about the scheduler overhead? It should be > >negligeable. > > > >Emmanuel. > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi Emmanuel, Thanks a lot for your reply and this is clear now. When you are talking about CPU scheduler, another question just came into mind: Xen manual mentioned Xen used kind of depth-first algorithm to determine the CPU sequence on a platform which has two processor and each processor has two cores and each core has two hyper-threads. That means, CPU0 will be the first thread in the first core in the first processor, CPU1 will be the second thread in the first core in the first processor, etc. I have a dual-processor/dual-core platform and it seems this is true by disabling/enabling multi-core in BIOS. However, I was told the CPU sequence is not numbered using this depth-first scheme on the latest Intel quad-core platform. I don''t understand why Xen need to change this. Do you have any idea how to verify the CPU sequence number on a dual-processor/quad-core/hyper-threaded platforms? Best regards, Liang ----- Original Message ----- From: "Emmanuel Ackaouy" <ack@xensource.com> To: "Liang Yang" <yangliang_mr@hotmail.com> Cc: "xen-devel" <xen-devel@lists.xensource.com> Sent: Thursday, November 16, 2006 10:22 AM Subject: Re: [Xen-devel] credit vs sedf scheduler> On Thu, Nov 16, 2006 at 10:08:40AM -0700, Liang Yang wrote: >> Hi Emmanuel, >> >> Do you mean, if the number of VCPUs is equal to the number of physcial >> CPUs, Xen will still call credit scheduler to allocate CPU resources >> amongst V CPUs even though each guest domain is assigned a fixed VCPU/CPU >> using xm vcpu-pin? > > What I mean is that: > > 1- The clock interrupt will still run every 10ms and this will > run some scheduler accounting code. > 2- After a VCPU completes its time slice, Xen will call into > the scheduler code to see if anything else needs to run. Of > course, in this case, there will be nothing else to run so the > VCPU will continue running. > > Both of these should be absolutely negligeable. > > Are you seeing overhead which is not negligeable? If so, this > would be a bug and should be fixed. > >> The purpose I assign each domain a CPU and fix the mapping of VCPU to CPU >> is I want each guest domain get equal CPU resources at any time. I don''t >> want to credit scheduler get involved. > > If your goal is to have each domain get equal CPU at any given > time, then you don''t need to do anything. The scheduler will > automatically spread your VCPUs across the 4 physical CPUs and > they will each get an equal amount of physical CPU resources. > > Also keep in mind that if a 5th domain (dom0 for example) needs > CPU resources as well, the credit scheduler will do this fairly > and also with negligeable costs whereas your scheme to pin all > VCPUs on their own physical CPU will not. > > Cheers, > Emmanuel. > >> Regards, >> >> Liang >> ----- Original Message ----- >> From: "Emmanuel Ackaouy" <ack@xensource.com> >> To: "Liang Yang" <multisyncfe991@hotmail.com> >> Cc: "xen-devel" <xen-devel@lists.xensource.com> >> Sent: Thursday, November 16, 2006 9:37 AM >> Subject: Re: [Xen-devel] credit vs sedf scheduler >> >> >> >On Thu, Nov 16, 2006 at 09:29:30AM -0700, Liang Yang wrote: >> >> Hi, >> >> >> >> Can I used vcpu-pin to avoid CPU scheduler overhead? Suppose I have >> >>four >> >> CPUs, if I used xm vcpu-pin to fix the mapping of each physcial CPU >> >> to >> >> virtual CPU. So credit scheduler will not be needed. >> >> >> >> Liang >> > >> >Pinning all your VCPUs will not stop Xen from executing various >> >scheduler code paths. >> > >> >Why are you worried about the scheduler overhead? It should be >> >negligeable. >> > >> >Emmanuel. >> > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Thu, Nov 16, 2006 at 02:03:08PM -0700, Liang Yang wrote:> Do you have any idea how to verify the CPU sequence number on a > dual-processor/quad-core/hyper-threaded platforms?I recently pushed a some changes to xen-unstable.hg which add better scheduler support for hyperthreaded and multi-core systems: changeset: 12265:cb8eeadd8eae user: Emmanuel Ackaouy <ack@xensource.com> date: Tue Nov 07 10:37:30 2006 +0000 summary: [XEN] When balancing idlers per socket/core, do it one at a time. changeset: 12253:bb6cd7ba259b user: Emmanuel Ackaouy <ack@xensource.com> date: Mon Nov 06 17:32:00 2006 +0000 summary: [XEN] Initial support for multi-core and multi-threaded CPU schedul ing. Part of this change includes some debug code that prints each CPU''s sibling and core map to the console. The sibling map indicates a CPU''s co-threads. The core map indicates a CPU''s peer cores on its local socket. You can print this out from the Xen console. Hit Ctrl-A three times to switch to Xen console and press the "r" key (print debug scheduler runq information). The code for this is by the first printk in csched_dump_pcpu() in xen/common/sched_credit.c . At some point, we should create a hypercall to export the system topology into dom0 user space. Perhaps the NUMA folks will get there first? Emmanuel. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Thu, Nov 16, 2006 at 03:49:41PM +0100, Markus Kremer wrote:> I will try to reproduce the tests with the latest unstable. > How do i enable the scheduler traces?look at xentrace. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> Keir already addressed the SEDF question. I think soft real > time support is a desirable thing. If you actually look at > what the SEDF scheduler does though, it''s not always obeying > the administrator''s orders or doing a very efficient job with > simple workloads. > > It would be interesting to compare scheduler traces with your > SEDF setup and the credit scheduler and see if, how, and why > SEDF might be scheduling things better for you. > > There are certainly simple things we could do in credit to > improve soft real time guarantees if needed: > - make the system default time slice tuneable. > - make per domain time-slice tuneable > - introduce strict priority based scheduling (like the current > I/O boost but tuneable per domain by the admin) > - Use the strict priority mechanism to offer soft real time > time-slice and wake-to-run latency best effort guarantees.vote for soft real time support i have 10-20 domU on single P4+HT (asterisk, freeswitch, openser, openpbx,..) I/O is low --------------------------------------- Marek Cervenka ====================================== _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel