Hi, all: I am studying the credit2 scheduler and found it would compensate VCPU migration with 50 microseconds. /* How much to "compensate" a vcpu for L2 migration */ #define CSCHED_MIGRATE_COMPENSATION MICROSECS(50) Why 50us? Is this number based on some experiments? If so, could any one kindly indicate how to perform the overhead measurement? Thanks very much! Sisu -- Sisu Xi, PhD Candidate http://www.cse.wustl.edu/~xis/ Department of Computer Science and Engineering Campus Box 1045 Washington University in St. Louis One Brookings Drive St. Louis, MO 63130 _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On ven, 2013-03-29 at 16:16 -0500, Sisu Xi wrote:> Hi, all: >Hi again Sisu,> I am studying the credit2 scheduler and found it would compensate VCPU > migration with 50 microseconds. > /* How much to "compensate" a vcpu for L2 migration */ > #define CSCHED_MIGRATE_COMPENSATION MICROSECS(50) > > > Why 50us? Is this number based on some experiments? >This predates my joining of the Xen community, but yes, ISTR hearing people talking about it coming from a mixture of common sense and experience/measurements. BTW, you may want to try, when sending e-mail to this list, adding in Cc some of the people that are more involved with the code/feature you''re asking about. You can look at the MAINTAINERS file, in the source tree, to have an idea of who to put there. For instance, for scheduling, the absolute authority is George (which I added), and in fact MAINTAINERS says: SCHEDULING M: George Dunlap <george.dunlap@eu.citrix.com> S: Supported F: xen/common/sched* (however, feel free to Cc me as well... I do not master all the Xen scheduler code yet, but am always curious about it. :-) ) Regards, Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Hi, Dario: Thanks for your reply and tips for sending emails! :) I am also performing some overhead measurement for the scheduler. If a VCPU is migrated from one core to another, the overhead is around 2 microseconds on my machine, which is much less than what is set in Credit2 (50 microseconds). Thanks again for your help! Sisu On Mon, Apr 8, 2013 at 8:46 PM, Dario Faggioli <dario.faggioli@citrix.com>wrote:> On ven, 2013-03-29 at 16:16 -0500, Sisu Xi wrote: > > Hi, all: > > > Hi again Sisu, > > > I am studying the credit2 scheduler and found it would compensate VCPU > > migration with 50 microseconds. > > /* How much to "compensate" a vcpu for L2 migration */ > > #define CSCHED_MIGRATE_COMPENSATION MICROSECS(50) > > > > > > Why 50us? Is this number based on some experiments? > > > This predates my joining of the Xen community, but yes, ISTR hearing > people talking about it coming from a mixture of common sense and > experience/measurements. > > BTW, you may want to try, when sending e-mail to this list, adding in Cc > some of the people that are more involved with the code/feature you''re > asking about. You can look at the MAINTAINERS file, in the source tree, > to have an idea of who to put there. For instance, for scheduling, the > absolute authority is George (which I added), and in fact MAINTAINERS > says: > > SCHEDULING > M: George Dunlap <george.dunlap@eu.citrix.com> > S: Supported > F: xen/common/sched* > > (however, feel free to Cc me as well... I do not master all the Xen > scheduler code yet, but am always curious about it. :-) ) > > Regards, > Dario > > -- > <<This happens because I choose it to happen!>> (Raistlin Majere) > ----------------------------------------------------------------- > Dario Faggioli, Ph.D, http://about.me/dario.faggioli > Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) > >-- Sisu Xi, PhD Candidate http://www.cse.wustl.edu/~xis/ Department of Computer Science and Engineering Campus Box 1045 Washington University in St. Louis One Brookings Drive St. Louis, MO 63130 _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On mer, 2013-04-10 at 06:44 +0100, Sisu Xi wrote:> Hi, Dario: >Hi! :-)> I am also performing some overhead measurement for the scheduler. If a > VCPU is migrated from one core to another, the overhead is around 2 > microseconds on my machine, which is much less than what is set in > Credit2 (50 microseconds). >Ok, and, finally, how did you decide to measure this? I mean, what is it that you are doing for coming up with that 2us value? When I was doing research on task scheduling, in "plain" OS, not hypervisors, I remember having setup something like this: you take a task (or a set of them), performing a known computation on a certain CPU, and you measure how long it takes. After that, you force a migration to another CPU, and keep monitoring the duration of the computation. Well, the first samples that you get right after the migration, gives you an idea of the overhead caused by the migration itself. Of course, you can/should try different kind of migrations, i.e., across cores (i.e., with a lot of shared caches), across sockets, etc. It is an indirect method, but the nice thing is you can use real programs and inspect the effect of migration on real workloads (e.g., a video player playing the same video). Of course, we have one more "layer of indirection" here (the vCPUs), but I guess we can think of something... So, again, I''m curious, how are you measuring it? Regards, Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
(BTW, please don''t top-post on xen-devel: respond in-line to the thing you want to respond to.) On 10/04/13 06:44, Sisu Xi wrote:> Hi, Dario: > > Thanks for your reply and tips for sending emails! :) > > I am also performing some overhead measurement for the scheduler. If a > VCPU is migrated from one core to another, the overhead is around 2 > microseconds on my machine, which is much less than what is set in > Credit2 (50 microseconds).When you say "overhead", I assume you mean that''s how long the whole migration takes? The point of the compensation isn''t so much for the actual migration itself, but for the lower performance the vcpu will get after the migration due to having cold caches. -George
On mer, 2013-04-10 at 09:21 +0100, George Dunlap wrote> On 10/04/13 06:44, Sisu Xi wrote: > > I am also performing some overhead measurement for the scheduler. If a > > VCPU is migrated from one core to another, the overhead is around 2 > > microseconds on my machine, which is much less than what is set in > > Credit2 (50 microseconds). > > When you say "overhead", I assume you mean that''s how long the whole > migration takes? > > The point of the compensation isn''t so much for the actual migration > itself, but for the lower performance the vcpu will get after the > migration due to having cold caches. >Which is, BTW, right the effect that we were trying to measure (although, again, it was Linux, not Xen at that time), with the experiments I was describing in my e-mail... Might have been obvious, but I think it''s worth making it even more that (thanks George :-) ), since I agree with George that _this_ is what we should be concerned about, when it comes to migration. Regards, Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Hi, Dario and George: Thanks for your reply. I am doing the measurement within Xen. I agree that a more precise description would be " the time spent performing scheduling and context switch ". and it does NOT consider cache. (I think the cache effect should be measured in the application level. not in hypervisor.) Basically I add code in trace.h, schedule.c, and domain.c (for context_switch). I measure three time here: 1. The time spent in making scheduling decisions. defined as scheduling latency. It includes grabbing the spinlock, call the scheduler dependent do_scheduler() function, and release the spinlock; 2. If the scheduler make a decision to perform a context switch (prev !next), it will call context_switch() in domain.c. I measure the time spent, right before the context_saved(prev); 3. The time spent in context_saved(prev), which is to insert the current VCPU back to runq. If a global queue is used and protected by a spinlock, this would include the time to grab the lock and so on. I ran the experiments on a i7 6 core machine, hyper-thread disabled, speedstep disabled. It has one socket. each core has dedicated L1 and L2 cache, and all of them share L3 cache. Host is CentOS 6.3 with Linux 3.4.35, Xen is 4.1.4. Domain 0 boots with one VCPU and pinned to core 0. one guest VM boots with 5 VCPUs, and the cpumask set for the remaining 5 VCPUs. I am using a gedf scheduler which I am working on. it is essentially a sedf scheduler sharing one global queue. In the guest VM, I just ran 5 cpu busy loops to make all the VCPUs busy. The data is get via xentrace with -M option, and I measured it for 10 seconds. The numbers are scheduler dependent, but the absolute number spent in context_switch() should be scheduler independent. And in my measurement, the worst case happens when we do context switch from a IDLE VCPU to a busy VCPU (which is scheduled previously on a different core), and perform the tlb_flush. It is around 2 microseconds on my machine. I am attaching the cdf plots of these three measurements. Any feedback is welcome. Thanks. Sisu On Wed, Apr 10, 2013 at 3:25 AM, Dario Faggioli <dario.faggioli@citrix.com>wrote:> On mer, 2013-04-10 at 09:21 +0100, George Dunlap wrote > > On 10/04/13 06:44, Sisu Xi wrote: > > > I am also performing some overhead measurement for the scheduler. If a > > > VCPU is migrated from one core to another, the overhead is around 2 > > > microseconds on my machine, which is much less than what is set in > > > Credit2 (50 microseconds). > > > > When you say "overhead", I assume you mean that''s how long the whole > > migration takes? > > > > The point of the compensation isn''t so much for the actual migration > > itself, but for the lower performance the vcpu will get after the > > migration due to having cold caches. > > > Which is, BTW, right the effect that we were trying to measure > (although, again, it was Linux, not Xen at that time), with the > experiments I was describing in my e-mail... > > Might have been obvious, but I think it''s worth making it even more that > (thanks George :-) ), since I agree with George that _this_ is what we > should be concerned about, when it comes to migration. > > Regards, > Dario > > -- > <<This happens because I choose it to happen!>> (Raistlin Majere) > ----------------------------------------------------------------- > Dario Faggioli, Ph.D, http://about.me/dario.faggioli > Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) > >-- Sisu Xi, PhD Candidate http://www.cse.wustl.edu/~xis/ Department of Computer Science and Engineering Campus Box 1045 Washington University in St. Louis One Brookings Drive St. Louis, MO 63130 _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel