Hi, I did a small experiment regarding Xen''s virtual cpu mapping. It seems sometimes manual rebalance (vcpu-pin) is required. Is there any plan of adding auto CPU rebalancing to the Xen scheduler? (or Xend tool?) -- On a two-way server, I measured a CPU-loop benchmark with three one-way VMs. 1. create VM1 # xm vcpu-list Name ID VCPU CPU State Time(s) CPU Affinity dom0 0 0 0 -b- 139.6 any cpu dom0 0 1 1 r-- 99.7 any cpu VM1 14 0 1 --- 0.5 any cpu 2. create VM2 # xm vcpu-list Name ID VCPU CPU State Time(s) CPU Affinity dom0 0 0 0 -b- 141.4 any cpu dom0 0 1 1 r-- 101.4 any cpu VM1 14 0 1 -b- 0.9 any cpu VM2 15 0 0 r-- 0.5 any cpu 3. create VM3 # xm vcpu-list Name ID VCPU CPU State Time(s) CPU Affinity dom0 0 0 0 r-- 142.6 any cpu dom0 0 1 1 r-- 103.9 any cpu VM3 16 0 1 --- 0.3 any cpu VM1 14 0 1 -b- 0.9 any cpu VM2 15 0 0 -b- 0.9 any cpu 4. shutdown VM2 # xm shutdown VM2 # xm list Name ID Mem(MiB) VCPUs State Time(s) dom0 0 512 2 r----- 251.9 VM3 16 64 1 -b---- 0.8 VM1 14 64 1 -b---- 0.9 5. benchmark CPU-loop sh on VM1 and VM3 # xm vcpu-list Name ID VCPU CPU State Time(s) CPU Affinity dom0 0 0 0 r-- 146.4 any cpu dom0 0 1 1 -b- 109.2 any cpu VM3 16 0 1 -b- 0.9 any cpu VM1 14 0 1 -b- 1.0 any cpu # xm console VM1 root@tiny ~ # ./test.sh DATE: 12/28/05 TIME:04:11:14 DATE: 12/28/05 TIME:04:14:54 root@tiny ~ # exec time is 3m40s # xm console VM3 root@tiny ~ # ./test.sh DATE: 12/28/05 TIME:04:11:15 DATE: 12/28/05 TIME:04:14:55 root@tiny ~ # exec time is 3m40s 6. manual rebalance # xm vcpu-pin VM1 0 0 # xm vcpu-list Name ID VCPU CPU State Time(s) CPU Affinity dom0 0 0 0 -b- 148.5 any cpu dom0 0 1 1 r-- 113.4 any cpu VM3 16 0 1 -b- 110.1 any cpu VM1 14 0 0 -b- 110.5 0 7. same benchmark as step5, and result is 2 times better. # xm console VM1 root@tiny ~ # ./test.sh DATE: 12/28/05 TIME:04:19:24 DATE: 12/28/05 TIME:04:21:00 root@tiny ~ # exec time is 1m36s # xm console VM3 root@tiny ~ # ./test.sh root@tiny ~ # ./test.sh DATE: 12/28/05 TIME:04:19:25 DATE: 12/28/05 TIME:04:20:59 root@tiny ~ # exec time is 1m36s Best Regards, Matsumoto _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Thu, Dec 29, 2005 at 11:33:26AM +0900, Matsumoto wrote:> I did a small experiment regarding Xen''s > virtual cpu mapping. > It seems sometimes manual rebalance (vcpu-pin) is required.Unfortunately, that is the expected behaviour as of today.> Is there any plan of adding auto CPU > rebalancing to the Xen scheduler? (or Xend tool?)Yes. We''re working on it. Emmanuel. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 12/29/05, Emmanuel Ackaouy <ack@xensource.com> wrote:> On Thu, Dec 29, 2005 at 11:33:26AM +0900, Matsumoto wrote: > > I did a small experiment regarding Xen''s > > virtual cpu mapping. > > It seems sometimes manual rebalance (vcpu-pin) is required. > > Unfortunately, that is the expected behaviour as of today. > > > Is there any plan of adding auto CPU > > rebalancing to the Xen scheduler? (or Xend tool?) > > Yes. We''re working on it. > > Emmanuel.Can I ask what work is being done? Which scheduler? Also is any scheduler going to include any algorthm to handle gang scheduling of multiplie vcpus? _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> Also is any scheduler going to include any algorthm to handle > gang scheduling of multiplie vcpus?It remains to be seen whether gang scheduling is actually necessary. I''d like to see how well some combination of "bad pre-emption" avoidance/mitigation coupled with biasing the scheduler to run vcpu''s of domains that already have vcpu''s running. Strict gang scheduling is likely to lead to a lot of wasted cycles, and I expect we can do better. The first step is to define a set of workloads we want to optimise for, and collect some traces. Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 29 Dec 2005, at 22:05, Ian Pratt wrote:> It remains to be seen whether gang scheduling is actually necessary. > I''d > like to see how well some combination of "bad pre-emption" > avoidance/mitigation coupled with biasing the scheduler to run vcpu''s > of > domains that already have vcpu''s running. > > Strict gang scheduling is likely to lead to a lot of wasted cycles, and > I expect we can do better. The first step is to define a set of > workloads we want to optimise for, and collect some traces.We need a gang-scheduling option for hyperthreads. Unless they are devoted to service domains, it doesn''t make sense to run different domUs simultaneously on the same core. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel