Timo Benk
2007-Feb-01 09:47 UTC
[Xen-devel] Credit Scheduler not working correct (3.0.4-0)
Hi,
on my system, it seems that the credit scheduler does not work as expected. The
problem seems to be related to HyperThreading, if i switch off HT in the BIOS
Setup, the scheduler works as expected.
I have two domains running, both pinned to the same processor:
Domain-0:~ # xm vcpu-list
Name ID VCPU CPU State Time(s) CPU
Affinity
Domain-0 0 0 1 --- 30.2 any cpu
Domain-0 0 1 1 r-- 12.0 any cpu
xendom1 1 0 0 --- 98.6 0
xendom2 2 0 0 r-- 110.3 0
Both domains should consume max. 100 Percent of that processor, but
xendom1 has a weight twice as big as xendom2:
Domain-0:~ # xm sched-credit -d xendom1
{''cap'': 100, ''weight'': 200}
Domain-0:~ # xm sched-credit -d xendom2
{''cap'': 100, ''weight'': 100}
In my understanding of the credit scheduler, xendom1 should get twice as
much CPU-time as xendom2. But:
xentop - 11:05:46 Xen 3.0.4-0
Mem: 2612284k total, 2603852k used, 8432k free CPUs: 2 @ 2992MHz
NAME STATE CPU(sec) CPU(%) [...]
Domain-0 -----r 42 0.6 [...]
xendom1 ------ 122 49.8 [...]
xendom2 -----r 133 50.2 [...]
As you can see, both domains got an equal amount of the CPU.
Am i missing something or do i understand something wrong?
Greetings,
-timo
--
Timo Benk - Jabber ID: fry@downtempo.de - ICQ ID: #414944731
PGP Public Key: http://m28s01.vlinux.de/timo_benk_gpg_key.asc
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Atsushi SAKAI
2007-Feb-01 10:07 UTC
Re: [Xen-devel] Credit Scheduler not working correct (3.0.4-0)
Hi, Timo If you want to divide CPU resources as you expected, each vcpu weight should be same. In your case, you should define xendom1 for 1vcpu and xendom2 for 2vcpu(s). then after you should pin these vcpus to pcpu0. Thanks Atsushi SAKAI Timo Benk <timo.benk@gmx.de> wrote:> Hi, > > on my system, it seems that the credit scheduler does not work as expected. The > problem seems to be related to HyperThreading, if i switch off HT in the BIOS > Setup, the scheduler works as expected. > > I have two domains running, both pinned to the same processor: > > Domain-0:~ # xm vcpu-list > Name ID VCPU CPU State Time(s) CPU > Affinity > Domain-0 0 0 1 --- 30.2 any cpu > Domain-0 0 1 1 r-- 12.0 any cpu > xendom1 1 0 0 --- 98.6 0 > xendom2 2 0 0 r-- 110.3 0 > > Both domains should consume max. 100 Percent of that processor, but > xendom1 has a weight twice as big as xendom2: > > Domain-0:~ # xm sched-credit -d xendom1 > {''cap'': 100, ''weight'': 200} > Domain-0:~ # xm sched-credit -d xendom2 > {''cap'': 100, ''weight'': 100} > > In my understanding of the credit scheduler, xendom1 should get twice as > much CPU-time as xendom2. But: > > xentop - 11:05:46 Xen 3.0.4-0 > Mem: 2612284k total, 2603852k used, 8432k free CPUs: 2 @ 2992MHz > NAME STATE CPU(sec) CPU(%) [...] > Domain-0 -----r 42 0.6 [...] > xendom1 ------ 122 49.8 [...] > xendom2 -----r 133 50.2 [...] > > As you can see, both domains got an equal amount of the CPU. > > Am i missing something or do i understand something wrong? > > Greetings, > -timo > -- > Timo Benk - Jabber ID: fry@downtempo.de - ICQ ID: #414944731 > PGP Public Key: http://m28s01.vlinux.de/timo_benk_gpg_key.asc > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Timo Benk
2007-Feb-01 10:15 UTC
Re: [Xen-devel] Credit Scheduler not working correct (3.0.4-0)
Atsushi SAKAI wrote:> Hi, Timo > > If you want to divide CPU resources as you expected, > each vcpu weight should be same. > In your case, you should define xendom1 for 1vcpu > and xendom2 for 2vcpu(s). then after you should pin these vcpus to pcpu0.Hm, is that a bug or a feature? I thought the crdeit-schedulers primary goal was to avoid this pinning nightmare you have with SEDF? Greetings, -timo> Timo Benk <timo.benk@gmx.de> wrote: > >> Hi, >> >> on my system, it seems that the credit scheduler does not work as expected. The >> problem seems to be related to HyperThreading, if i switch off HT in the BIOS >> Setup, the scheduler works as expected. >> >> I have two domains running, both pinned to the same processor: >> >> Domain-0:~ # xm vcpu-list >> Name ID VCPU CPU State Time(s) CPU >> Affinity >> Domain-0 0 0 1 --- 30.2 any cpu >> Domain-0 0 1 1 r-- 12.0 any cpu >> xendom1 1 0 0 --- 98.6 0 >> xendom2 2 0 0 r-- 110.3 0 >> >> Both domains should consume max. 100 Percent of that processor, but >> xendom1 has a weight twice as big as xendom2: >> >> Domain-0:~ # xm sched-credit -d xendom1 >> {''cap'': 100, ''weight'': 200} >> Domain-0:~ # xm sched-credit -d xendom2 >> {''cap'': 100, ''weight'': 100} >> >> In my understanding of the credit scheduler, xendom1 should get twice as >> much CPU-time as xendom2. But: >> >> xentop - 11:05:46 Xen 3.0.4-0 >> Mem: 2612284k total, 2603852k used, 8432k free CPUs: 2 @ 2992MHz >> NAME STATE CPU(sec) CPU(%) [...] >> Domain-0 -----r 42 0.6 [...] >> xendom1 ------ 122 49.8 [...] >> xendom2 -----r 133 50.2 [...] >> >> As you can see, both domains got an equal amount of the CPU. >> >> Am i missing something or do i understand something wrong? >> >> Greetings, >> -timo >> -- >> Timo Benk - Jabber ID: fry@downtempo.de - ICQ ID: #414944731 >> PGP Public Key: http://m28s01.vlinux.de/timo_benk_gpg_key.asc >> >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel > >-- Timo Benk - Jabber ID: fry@downtempo.de - ICQ ID: #414944731 PGP Public Key: http://m28s01.vlinux.de/timo_benk_gpg_key.asc _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Atsushi SAKAI
2007-Feb-01 10:25 UTC
Re: [Xen-devel] Credit Scheduler not working correct (3.0.4-0)
Hi, Timo Primary goal of Credit Scheduler is loadbalancing multiple CPU resources. So if you pin vcpu, this condition is violated, since vcpu cannot migrate between pcpu. Anyway, Credit scheduler specification is defined by Emmanuel. So I have no comment on it. But, I hope scheduler instruction should be simple.(like vmware weight) Current credit scheduler is very complex when we uses relative weight and vcpu pin with mixing these two parameter) c.f. vmware uses absolute weight(like CPU freqency) Thanks Atsushi SAKAI Timo Benk <timo.benk@gmx.de> wrote:> Atsushi SAKAI wrote: > > Hi, Timo > > > > If you want to divide CPU resources as you expected, > > each vcpu weight should be same. > > In your case, you should define xendom1 for 1vcpu > > and xendom2 for 2vcpu(s). then after you should pin these vcpus to pcpu0. > > Hm, is that a bug or a feature? > > I thought the crdeit-schedulers primary goal was to avoid this pinning nightmare > you have with SEDF? > > Greetings, > -timo > > > Timo Benk <timo.benk@gmx.de> wrote: > > > >> Hi, > >> > >> on my system, it seems that the credit scheduler does not work as expected. The > >> problem seems to be related to HyperThreading, if i switch off HT in the BIOS > >> Setup, the scheduler works as expected. > >> > >> I have two domains running, both pinned to the same processor: > >> > >> Domain-0:~ # xm vcpu-list > >> Name ID VCPU CPU State Time(s) CPU > >> Affinity > >> Domain-0 0 0 1 --- 30.2 any cpu > >> Domain-0 0 1 1 r-- 12.0 any cpu > >> xendom1 1 0 0 --- 98.6 0 > >> xendom2 2 0 0 r-- 110.3 0 > >> > >> Both domains should consume max. 100 Percent of that processor, but > >> xendom1 has a weight twice as big as xendom2: > >> > >> Domain-0:~ # xm sched-credit -d xendom1 > >> {''cap'': 100, ''weight'': 200} > >> Domain-0:~ # xm sched-credit -d xendom2 > >> {''cap'': 100, ''weight'': 100} > >> > >> In my understanding of the credit scheduler, xendom1 should get twice as > >> much CPU-time as xendom2. But: > >> > >> xentop - 11:05:46 Xen 3.0.4-0 > >> Mem: 2612284k total, 2603852k used, 8432k free CPUs: 2 @ 2992MHz > >> NAME STATE CPU(sec) CPU(%) [...] > >> Domain-0 -----r 42 0.6 [...] > >> xendom1 ------ 122 49.8 [...] > >> xendom2 -----r 133 50.2 [...] > >> > >> As you can see, both domains got an equal amount of the CPU. > >> > >> Am i missing something or do i understand something wrong? > >> > >> Greetings, > >> -timo > >> -- > >> Timo Benk - Jabber ID: fry@downtempo.de - ICQ ID: #414944731 > >> PGP Public Key: http://m28s01.vlinux.de/timo_benk_gpg_key.asc > >> > >> > >> _______________________________________________ > >> Xen-devel mailing list > >> Xen-devel@lists.xensource.com > >> http://lists.xensource.com/xen-devel > > > > > > > -- > Timo Benk - Jabber ID: fry@downtempo.de - ICQ ID: #414944731 > PGP Public Key: http://m28s01.vlinux.de/timo_benk_gpg_key.asc >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Emmanuel Ackaouy
2007-Feb-01 10:30 UTC
Re: [Xen-devel] Credit Scheduler not working correct (3.0.4-0)
Ok so If I get this right, what you want to do is have 2 UP guests with different weights. You also want to turn off HT and you do this by pinning both guests on CPU 0. Credit accounting and pinning VCPUs don''t always interact well and I suspect you''re hitting such a problem here. If you have access to the Xen console, can you send output of the ''r'' (runq) key while your domains are each getting 50% of CPU? Just take a few snapshots a few seconds apart. That would confirm the theory. Also, setting cap=100 on a UP guest is pointless. Just leave it as zero. A VCPU isn''t going to get more than 100% of a physical CPU anyway. On Feb 1, 2007, at 10:47, Timo Benk wrote:> Hi, > > on my system, it seems that the credit scheduler does not work as > expected. The > problem seems to be related to HyperThreading, if i switch off HT in > the BIOS > Setup, the scheduler works as expected. > > I have two domains running, both pinned to the same processor: > > Domain-0:~ # xm vcpu-list > Name ID VCPU CPU State Time(s) CPU > Affinity > Domain-0 0 0 1 --- 30.2 any > cpu > Domain-0 0 1 1 r-- 12.0 any > cpu > xendom1 1 0 0 --- 98.6 0 > xendom2 2 0 0 r-- 110.3 0 > > Both domains should consume max. 100 Percent of that processor, but > xendom1 has a weight twice as big as xendom2: > > Domain-0:~ # xm sched-credit -d xendom1 > {''cap'': 100, ''weight'': 200} > Domain-0:~ # xm sched-credit -d xendom2 > {''cap'': 100, ''weight'': 100} > > In my understanding of the credit scheduler, xendom1 should get twice > as > much CPU-time as xendom2. But: > > xentop - 11:05:46 Xen 3.0.4-0 > Mem: 2612284k total, 2603852k used, 8432k free CPUs: 2 @ 2992MHz > NAME STATE CPU(sec) CPU(%) [...] > Domain-0 -----r 42 0.6 [...] > xendom1 ------ 122 49.8 [...] > xendom2 -----r 133 50.2 [...] > > As you can see, both domains got an equal amount of the CPU. > > Am i missing something or do i understand something wrong? > > Greetings, > -timo > -- > Timo Benk - Jabber ID: fry@downtempo.de - ICQ ID: #414944731 > PGP Public Key: http://m28s01.vlinux.de/timo_benk_gpg_key.asc > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Daniele Palumbo
2007-Feb-01 10:58 UTC
Re: [Xen-devel] Credit Scheduler not working correct (3.0.4-0)
On Thursday 01 February 2007 11:30, Emmanuel Ackaouy wrote:> Also, setting cap=100 on a UP guest is pointless. Just leave it > as zero. A VCPU isn''t going to get more than 100% of a physical > CPU anyway.is pointless if you have only 1 vcpu, right? thanks Daniele _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Timo Benk
2007-Feb-01 11:06 UTC
Re: [Xen-devel] Credit Scheduler not working correct (3.0.4-0)
Emmanuel Ackaouy wrote:> If you have access to the Xen console, can you send output of > the ''r'' (runq) key while your domains are each getting 50% of > CPU? Just take a few snapshots a few seconds apart. That would > confirm the theory.Sorry, but i dunno how to do that? Can you give me a hint?> Also, setting cap=100 on a UP guest is pointless. Just leave it > as zero. A VCPU isn''t going to get more than 100% of a physical > CPU anyway.Yepp, you are right. The reason i have done this was ''cause i want to demonstrate the credit-scheduler. If both domains have one vcpu, managed by Xen and i set the following: Domain-0:~ # xm sched-credit -d xendom1 {''cap'': 100, ''weight'': 200} Domain-0:~ # xm sched-credit -d xendom2 {''cap'': 100, ''weight'': 100} Both domain are running at 100 percent cpu, but on different pcpus. That is correct, but does not demonstrate how to use the credit scheduler. Ok, the other advice from Atsushi was to give one domain two vcpus, the other one vcpu, both with the same weight and all vcpus pinned to the same pcpu. That does not help either. Both domains are still running at 50 percent. Even if i set the weight of xendom1, resp. xendom2 to be twice as the value of xendom2, resp. xendom1, i get the same result. Here is what i have done: Domain-0:~ # xm vcpu-list Name ID VCPU CPU State Time(s) CPU Affinity Domain-0 0 0 1 --- 46.1 any cpu Domain-0 0 1 1 r-- 28.5 any cpu xendom1 1 0 0 r-- 558.7 0 xendom2 3 0 0 -b- 21.6 0 xendom2 3 1 0 --- 256.0 0 Domain-0:~ # xm sched-credit -d xendom1 {''cap'': 0, ''weight'': 256} Domain-0:~ # xm sched-credit -d xendom2 {''cap'': 0, ''weight'': 256} 3 domains: 2 running, 0 blocked, 0 paused, 0 crashed, 0 dying, 0 shutdown Mem: 2612284k total, 2603856k used, 8428k free CPUs: 2 @ 2992MHz NAME STATE CPU(sec) CPU(%) MEM(k) MEM(%) MAXMEM(k) MAXMEM(%) VCPUS NETS NETTX(k) NETRX(k) VBDS VBD_OO VBD_RD VBD_WR SSID Domain-0 -----r 85 1.6 2048256 78.4 no limit n/a 2 4 163 139 0 0 0 0 0 xendom1 ------ 566 49.0 262144 10.0 532480 20.4 1 1 5 29 2 0 10 539 0 xendom2 -----r 285 51.0 262144 10.0 262144 10.0 2 1 6 17 2 0 4622 1772 0 Greetings, -timo> On Feb 1, 2007, at 10:47, Timo Benk wrote: >> Hi, >> >> on my system, it seems that the credit scheduler does not work as >> expected. The >> problem seems to be related to HyperThreading, if i switch off HT in >> the BIOS >> Setup, the scheduler works as expected. >> >> I have two domains running, both pinned to the same processor: >> >> Domain-0:~ # xm vcpu-list >> Name ID VCPU CPU State Time(s) CPU >> Affinity >> Domain-0 0 0 1 --- 30.2 any cpu >> Domain-0 0 1 1 r-- 12.0 any cpu >> xendom1 1 0 0 --- 98.6 0 >> xendom2 2 0 0 r-- 110.3 0 >> >> Both domains should consume max. 100 Percent of that processor, but >> xendom1 has a weight twice as big as xendom2: >> >> Domain-0:~ # xm sched-credit -d xendom1 >> {''cap'': 100, ''weight'': 200} >> Domain-0:~ # xm sched-credit -d xendom2 >> {''cap'': 100, ''weight'': 100} >> >> In my understanding of the credit scheduler, xendom1 should get twice as >> much CPU-time as xendom2. But: >> >> xentop - 11:05:46 Xen 3.0.4-0 >> Mem: 2612284k total, 2603852k used, 8432k free CPUs: 2 @ 2992MHz >> NAME STATE CPU(sec) CPU(%) [...] >> Domain-0 -----r 42 0.6 [...] >> xendom1 ------ 122 49.8 [...] >> xendom2 -----r 133 50.2 [...] >> >> As you can see, both domains got an equal amount of the CPU. >> >> Am i missing something or do i understand something wrong? >> >> Greetings, >> -timo >> -- >> Timo Benk - Jabber ID: fry@downtempo.de - ICQ ID: #414944731 >> PGP Public Key: http://m28s01.vlinux.de/timo_benk_gpg_key.asc >> >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel >> >-- Timo Benk - Jabber ID: fry@downtempo.de - ICQ ID: #414944731 PGP Public Key: http://m28s01.vlinux.de/timo_benk_gpg_key.asc _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Atsushi SAKAI
2007-Feb-01 11:32 UTC
Re: [Xen-devel] Credit Scheduler not working correct (3.0.4-0)
Hi, Emmanuel I wrote down my guess based on previous study for Timo''s cases. (I ask you this behavior is specification or not previously.) http://lists.xensource.com/archives/html/xen-devel/2006-10/msg00365.html This case is similar to case 2) (when pinned-vcpu credit sum is over 1pcpu capacity) Let''s calculate on this. Dom1 for 100Weight Dom2 for 200Weight But CPU resources is 200(2CPU x 100) So these credits are Dom1:67 and Dom2:133 But These vcpus are pinned to pcpu0 pcpu0 has only 100. but credit total is 200. This makes a problem. These 2vcpu has same priority in credit scheduler, since it cannot not consume its credit. So these vcpus are scheduled as round robin. This makes equal consumption of pcpu0. Is this wrong guess? Thanks Atsushi SAKAI Emmanuel Ackaouy <ack@xensource.com> wrote:> Ok so If I get this right, what you want to do is have 2 UP guests with > different weights. You also want to turn off HT and you do this by > pinning both guests on CPU 0. > > Credit accounting and pinning VCPUs don''t always interact well > and I suspect you''re hitting such a problem here. > > If you have access to the Xen console, can you send output of > the ''r'' (runq) key while your domains are each getting 50% of > CPU? Just take a few snapshots a few seconds apart. That would > confirm the theory. > > Also, setting cap=100 on a UP guest is pointless. Just leave it > as zero. A VCPU isn''t going to get more than 100% of a physical > CPU anyway. > > > On Feb 1, 2007, at 10:47, Timo Benk wrote: > > Hi, > > > > on my system, it seems that the credit scheduler does not work as > > expected. The > > problem seems to be related to HyperThreading, if i switch off HT in > > the BIOS > > Setup, the scheduler works as expected. > > > > I have two domains running, both pinned to the same processor: > > > > Domain-0:~ # xm vcpu-list > > Name ID VCPU CPU State Time(s) CPU > > Affinity > > Domain-0 0 0 1 --- 30.2 any > > cpu > > Domain-0 0 1 1 r-- 12.0 any > > cpu > > xendom1 1 0 0 --- 98.6 0 > > xendom2 2 0 0 r-- 110.3 0 > > > > Both domains should consume max. 100 Percent of that processor, but > > xendom1 has a weight twice as big as xendom2: > > > > Domain-0:~ # xm sched-credit -d xendom1 > > {''cap'': 100, ''weight'': 200} > > Domain-0:~ # xm sched-credit -d xendom2 > > {''cap'': 100, ''weight'': 100} > > > > In my understanding of the credit scheduler, xendom1 should get twice > > as > > much CPU-time as xendom2. But: > > > > xentop - 11:05:46 Xen 3.0.4-0 > > Mem: 2612284k total, 2603852k used, 8432k free CPUs: 2 @ 2992MHz > > NAME STATE CPU(sec) CPU(%) [...] > > Domain-0 -----r 42 0.6 [...] > > xendom1 ------ 122 49.8 [...] > > xendom2 -----r 133 50.2 [...] > > > > As you can see, both domains got an equal amount of the CPU. > > > > Am i missing something or do i understand something wrong? > > > > Greetings, > > -timo > > -- > > Timo Benk - Jabber ID: fry@downtempo.de - ICQ ID: #414944731 > > PGP Public Key: http://m28s01.vlinux.de/timo_benk_gpg_key.asc > > > > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xensource.com > > http://lists.xensource.com/xen-devel > > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Emmanuel Ackaouy
2007-Feb-01 12:16 UTC
Re: [Xen-devel] Credit Scheduler not working correct (3.0.4-0)
On Feb 1, 2007, at 12:32, Atsushi SAKAI wrote:> Hi, Emmanuel > > I wrote down my guess based on previous study for Timo''s cases. > (I ask you this behavior is specification or not previously.) > http://lists.xensource.com/archives/html/xen-devel/2006-10/ > msg00365.html > This case is similar to case 2) > (when pinned-vcpu credit sum is over 1pcpu capacity) > > > Let''s calculate on this. > Dom1 for 100Weight > Dom2 for 200Weight > But CPU resources is 200(2CPU x 100) > So these credits are Dom1:67 and Dom2:133 > But These vcpus are pinned to pcpu0 > pcpu0 has only 100. but credit total is 200. > This makes a problem. > > These 2vcpu has same priority in credit scheduler, > since it cannot not consume its credit. > So these vcpus are scheduled as round robin. > This makes equal consumption of pcpu0. > > Is this wrong guess?You''ve described the problem. Arbitrary pinning makes system wide weighted fair share complex. Within that problem set there are simple cases where we could easily do better than now though. Using cpumasks to take one physical CPU out of the system is one such case. Arguably, we should have a notion of system partitions to do this though but in the mean time, we can throw some code together to not totally bail on weighted fair share when some simpler uses of pinning are used. I''ll probably have time to look at this next week. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Emmanuel Ackaouy
2007-Feb-06 13:17 UTC
Re: [Xen-devel] Credit Scheduler not working correct (3.0.4-0)
On Feb 1, 2007, at 11:58, Daniele Palumbo wrote:> On Thursday 01 February 2007 11:30, Emmanuel Ackaouy wrote: > Also, setting cap=100 on a UP guest is pointless. Just leave it >> as zero. A VCPU isn''t going to get more than 100% of a physical >> CPU anyway. > > is pointless if you have only 1 vcpu, right?Yes, because 100 means 1 physical CPU. 100 * number of VCPUs is the upper bound anyway. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel