While doing some performance analysis of Xen I have noticed some interesting behavior of the credit scheduler that I would like to see discussed. In my most basic test I have a 4 socket dual core Intel box (Paxville) with a uniprocessor dom0 and 7 uniprocessor domUs. Each of the domUs is pinned on its own core with the first core of the system left for dom0. When running the credit scheduler the dom0 VCPU will bounce around the system, sometimes landing on the same thread as one of the domUs or sometimes on one of the sibling hyperthreads (this appears to happen a majority of the time it moves). This is less than ideal when considering cache warmth and the sharing of CPU resources when the first core of the system is always available in this configuration. Does the credit scheduler have any awareness of cache warmth or CPU siblings when balancing? I have also seen similar behavior when running tests in the domUs such that each has its VCPU running at 100% utilization so I believe this behavior to be fairly uniform. In my testing I am looking for uniform behavior so I have just been setting sched=sedf but I would like to move to credit scheduler since it is the new default. However, until I sort out this behavior I cannot. Attached is a snapshot of "xm vcpu-list" taken every 5 seconds for 5 measurements to show the dom0 VCPU migration pattern on a system with idle domains. -- Karl Rister IBM Linux Performance Team kmr@us.ibm.com _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> In my most basic test I have a 4 socket dual core Intel box (Paxville)with> a > uniprocessor dom0 and 7 uniprocessor domUs. Each of the domUs ispinned on> its own core with the first core of the system left for dom0. Whenrunning> the credit scheduler the dom0 VCPU will bounce around the system,sometimes> landing on the same thread as one of the domUs or sometimes on one ofthe> sibling hyperthreads (this appears to happen a majority of the time it > moves). This is less than ideal when considering cache warmth and the > sharing of CPU resources when the first core of the system is always > available in this configuration. Does the credit scheduler have any > awareness of cache warmth or CPU siblings when balancing? > > I have also seen similar behavior when running tests in the domUs suchthat> each has its VCPU running at 100% utilization so I believe thisbehavior to> be fairly uniform.The sometimes suboptimal use of hyperthreading is well understood and is on the todo list. It hasn''t been a priority as the current generation of Intel Core/Core2 and Opteron CPUs don''t have HT. Apart from the hyperthreading sibling case, a dom0 vcpu should only ever migrate to another CPU that has been idle for a complete rebalancing period. Hence the rate of movement should be very low and thus the overhead of priming the cache tiny. It''s conceivable one could construct an argument that it''s not worth the special case code in the scheduler to fix... BTW: The PhD thesis of our very own James Bulpin has lots of useful data on cache warming times and how to optimize for HT systems: http://www.cl.cam.ac.uk/TechReports/UCAM-CL-TR-619.pdf Allowing dom0 vcpus to explicitly be pinned would certainly be a good thing though, and is slated for post 3.0.3 -- see the thread on this topic earlier today. Actually, in the interim it would be easy to add a xen command line parameter to pin dom0 vcpus... Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 28/8/06 11:28 pm, "Karl Rister" <kmr@us.ibm.com> wrote:> This is less than ideal when considering cache warmth and the > sharing of CPU resources when the first core of the system is always > available in this configuration. Does the credit scheduler have any > awareness of cache warmth or CPU siblings when balancing?It doesn''t right now, but in your setup dom0 has its ''home core'' all to itself, so it''s odd that it''s bouncing around. Other cores should only steal the domain if it is runnable but not running -- that should almost never be the case if the domain is not contending for CPU. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
I''ve seen something similar... I have a cpu with two HT and two domains (single-vcpu dom0 and an HVM domain), and I''m using the VMEXIT/VMENTER tracing. The most natural thing would be for each to get its own logical cpu, but under the credit scheduler, the HVM domain is seeing the VMENTER on a different logical cpu than it took the VMEXIT on. This makes the RDTSC values taken not comparable. -George On 8/29/06, Keir Fraser <Keir.Fraser@cl.cam.ac.uk> wrote:> > > > On 28/8/06 11:28 pm, "Karl Rister" <kmr@us.ibm.com> wrote: > > > This is less than ideal when considering cache warmth and the > > sharing of CPU resources when the first core of the system is always > > available in this configuration. Does the credit scheduler have any > > awareness of cache warmth or CPU siblings when balancing? > > It doesn''t right now, but in your setup dom0 has its ''home core'' all to > itself, so it''s odd that it''s bouncing around. Other cores should only steal > the domain if it is runnable but not running -- that should almost never be > the case if the domain is not contending for CPU. > > -- Keir > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Mon, Aug 28, 2006 at 05:28:08PM -0500, Karl Rister wrote:> While doing some performance analysis of Xen I have noticed some interesting > behavior of the credit scheduler that I would like to see discussed. > > In my most basic test I have a 4 socket dual core Intel box (Paxville) with a > uniprocessor dom0 and 7 uniprocessor domUs. Each of the domUs is pinned on > its own core with the first core of the system left for dom0. When running > the credit scheduler the dom0 VCPU will bounce around the system, sometimes > landing on the same thread as one of the domUs or sometimes on one of the > sibling hyperthreads (this appears to happen a majority of the time it > moves). This is less than ideal when considering cache warmth and the > sharing of CPU resources when the first core of the system is always > available in this configuration. Does the credit scheduler have any > awareness of cache warmth or CPU siblings when balancing?A related issue I noticed the other day when doing xen testing off of xen-unstable was that the irqbalancer process ended up consuming 100% of dom0 cpu when running an SMP Dom0 on the credit scheduler, and a bunch of idle domUs. It got the the point that ssh wasn''t feasable, and I had to go to the console and shut off irqbalancer to return things to normal. I''d follow up further, but I''m about to go "off grid" for the next 2.5 weeks. I relayed information to other folks on the IBM team to see if anyone else can reproduce it. -Sean -- Sean Dague IBM Linux Technology Center email: japh@us.ibm.com Open Hypervisor Team alt: sldague@us.ibm.com _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> A related issue I noticed the other day when doing xen testing off of > xen-unstable was that the irqbalancer process ended up consuming 100%of> dom0 cpu when running an SMP Dom0 on the credit scheduler, and a bunchof> idle domUs. It got the the point that ssh wasn''t feasable, and I hadto go> to the console and shut off irqbalancer to return things to normal. > > I''d follow up further, but I''m about to go "off grid" for the next 2.5 > weeks. I relayed information to other folks on the IBM team to see if > anyone else can reproduce it.No good can come of running irqbalancer in dom0, but it certainly shouldn''t end up burning CPU. Would be good to understand what''s going on here. Thanks, Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Mon, Aug 28, 2006 at 05:28:08PM -0500, Karl Rister wrote:> In my testing I am looking for uniform behavior so I have just been setting > sched=sedf but I would like to move to credit scheduler since it is the new > default. However, until I sort out this behavior I cannot.One thing you could do is restrict dom0''s VCPU so that it doesn''t move around to a hyperthread on a core that you have assigned to a domU. So figure out what CPU dom0''s VCPU is on and restrict its cpumask so that it stays on that core. Then move any domU out of that core. _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel