Yang, Xiaowei
2009-Mar-09 08:18 UTC
[Xen-devel] [PATCH] Add migration_cost option to scheduler
The idea is borrowed from Linux kernel: if the vCPU is just scheduled out and put to run-queue, it''s likely cache-hot on its current pCPU, and it may be scheduled in in a short period of time; however, if vCPU is migrated to another pCPU, it need to re-warm the cache - that''s the meaning of migration cost. The patch introduces an option migration_cost to depress too aggressive vCPU migration (actually we really see migration frequency is very high most of the time.), while in the meantime keeping load balance works in certain degree. Linux kernel uses 0.5ms by default. Considering the cost may be higher (e.g. VMCS impact) than in native, migration_cost=1ms is chosen for our tests, which are performed on a 4x 6-core Dunnington platform. In 24-VM case, there is ~2% stable performance gain for enterprise workloads like SPECjbb and sysbench. If HVM is with stubdom, the gain is more: 4% for the same workloads. The best value may vary on different platforms based on different cache hierarchy and with different workloads. Due to resource limit, we haven''t test many combinations. And we plans to try more in future. Welcome to evaluate and give feedback on what''s suitable / not suitable for you. Signed-off-by: Xiaowei Yang <xiaowei.yang@intel.com> Thanks, Xiaowei _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
George Dunlap
2009-Mar-09 12:55 UTC
Re: [Xen-devel] [PATCH] Add migration_cost option to scheduler
Hmm, I think this patch may not be exactly what we want. It looks like it checks for how long a vcpu has been in its current stat, not how recently it has been running. So if a vcpu sleeps for a long time on a cpu that''s running other workloads, then wakes up (blocked->runnable), the cache is by no means "hot". But since it has only been in the "runnable" state for a few hundred cycles, it won''t be migrated, even though there''s little cost. However, if the pcpu was idle since the last time this vcpu ran (i.e., if we''re just catching the vcpu in the process of waking up), the cache *is* still hot. Hmm.... -George 2009/3/9 Yang, Xiaowei <xiaowei.yang@intel.com>:> The idea is borrowed from Linux kernel: if the vCPU is just scheduled out > and put to run-queue, it''s likely cache-hot on its current pCPU, and it may > be scheduled in in a short period of time; however, if vCPU is migrated to > another pCPU, it need to re-warm the cache - that''s the meaning of migration > cost. > > The patch introduces an option migration_cost to depress too aggressive vCPU > migration (actually we really see migration frequency is very high most of > the time.), while in the meantime keeping load balance works in certain > degree. > > Linux kernel uses 0.5ms by default. Considering the cost may be higher (e.g. > VMCS impact) than in native, migration_cost=1ms is chosen for our tests, > which are performed on a 4x 6-core Dunnington platform. In 24-VM case, there > is ~2% stable performance gain for enterprise workloads like SPECjbb and > sysbench. If HVM is with stubdom, the gain is more: 4% for the same > workloads. > > The best value may vary on different platforms based on different cache > hierarchy and with different workloads. Due to resource limit, we haven''t > test many combinations. And we plans to try more in future. Welcome to > evaluate and give feedback on what''s suitable / not suitable for you. > > Signed-off-by: Xiaowei Yang <xiaowei.yang@intel.com> > > > Thanks, > Xiaowei > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Tian, Kevin
2009-Mar-09 13:08 UTC
RE: [Xen-devel] [PATCH] Add migration_cost option to scheduler
>From: George Dunlap >Sent: Monday, March 09, 2009 8:56 PM > >Hmm, I think this patch may not be exactly what we want. It looks >like it checks for how long a vcpu has been in its current stat, not >how recently it has been running. So if a vcpu sleeps for a long time >on a cpu that''s running other workloads, then wakes up >(blocked->runnable), the cache is by no means "hot". But since it has >only been in the "runnable" state for a few hundred cycles, it won''t >be migrated, even though there''s little cost.Then to add a per-vcpu last_running_timestamp which is recorded when vcpu is scheduled out, could hit the purpose here?> >However, if the pcpu was idle since the last time this vcpu ran (i.e., >if we''re just catching the vcpu in the process of waking up), the >cache *is* still hot. Hmm.... >If peer pcpu is idle, shouldn''t it be skipped by load balancer running on another pcpu, which is out of the cache-hot logic? Thanks, Kevin _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Yang, Xiaowei
2009-Mar-10 03:36 UTC
Re: [Xen-devel] [PATCH] Add migration_cost option to scheduler
Tian, Kevin wrote:>> From: George Dunlap >> Sent: Monday, March 09, 2009 8:56 PM >> >> Hmm, I think this patch may not be exactly what we want. It looks >> like it checks for how long a vcpu has been in its current stat, not >> how recently it has been running. So if a vcpu sleeps for a long time >> on a cpu that''s running other workloads, then wakes up >> (blocked->runnable), the cache is by no means "hot". But since it has >> only been in the "runnable" state for a few hundred cycles, it won''t >> be migrated, even though there''s little cost. > > Then to add a per-vcpu last_running_timestamp which is recorded when > vcpu is scheduled out, could hit the purpose here?Yes, it''s more reasonable. I can make a patch. Thanks, xiaowei> >> However, if the pcpu was idle since the last time this vcpu ran (i.e., >> if we''re just catching the vcpu in the process of waking up), the >> cache *is* still hot. Hmm.... >> > > If peer pcpu is idle, shouldn''t it be skipped by load balancer running on > another pcpu, which is out of the cache-hot logic? > > Thanks, > Kevin_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Yang, Xiaowei
2009-Mar-10 10:56 UTC
Re: [Xen-devel] [PATCH] Add migration_cost option to scheduler
Yang, Xiaowei wrote:> Tian, Kevin wrote: >>> From: George Dunlap >>> Sent: Monday, March 09, 2009 8:56 PM >>> >>> Hmm, I think this patch may not be exactly what we want. It looks >>> like it checks for how long a vcpu has been in its current stat, not >>> how recently it has been running. So if a vcpu sleeps for a long time >>> on a cpu that''s running other workloads, then wakes up >>> (blocked->runnable), the cache is by no means "hot". But since it has >>> only been in the "runnable" state for a few hundred cycles, it won''t >>> be migrated, even though there''s little cost. >> Then to add a per-vcpu last_running_timestamp which is recorded when >> vcpu is scheduled out, could hit the purpose here? > Yes, it''s more reasonable. I can make a patch. >The patch is attached. Had a quick test using the previous config and saw no obvious different result with it. May be explained by the fact that the benchmarks has no many IO operations - db and logfile are put to ramdisk for sysbench test. Anyway it''s closer to what we really want. Thanks both of you! Signed-off-by: Xiaowei Yang <xiaowei.yang@intel.com> Thanks, xiaowei _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel