My question is related to an earlier discussion on the list ( http://lists.xensource.com/archives/html/xen-devel/2007-03/msg01395.html). Is there any plan for implementing gang scheduling (of all vcpus of a domain) in xen? I see that gang scheduling is mentioned in almost all the latest talks about xen (posted on xen.org). It seems to me that gang scheduling would be desirable, specially if there are synchronization points (lock and semaphores) in guest OS and applications. I understand that gang scheduling can lead to wasted cycles, but that might be acceptable in certain situations. In fact, wasted cycles can probably be avoided by doing opportunisitic gang scheduling (i.e. gang schedule, unless there would be wasted cycles) and by carefully choosing the number of vcpus. -- Nauman Rafique Purdue University _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hello, Nauman Rafique, le Mon 21 Apr 2008 13:52:21 -0400, a écrit :> In fact, wasted cycles can probably be avoided by doing opportunisitic > gang scheduling (i.e. gang schedule, unless there would be wasted > cycles)How do you detect that there would be wasted cycles? Samuel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 21/04/08 18:56 +0100, Samuel Thibault wrote:> Hello, > > Nauman Rafique, le Mon 21 Apr 2008 13:52:21 -0400, a écrit : > > In fact, wasted cycles can probably be avoided by doing opportunisitic > > gang scheduling (i.e. gang schedule, unless there would be wasted > > cycles) > > How do you detect that there would be wasted cycles?The only way is for a very self-aware guest to use a paravirtual feature to give a hint to the scheduler. Which is also the way to solve the original problem: the paravirtualized guest can provide a hint to the scheduler that it is holding a contended lock. Alternatively, the scheduler can notify the guest that it is about to be preempted by the hypervisor and now would be a good time to sleep before gaining a contended spinlock. In either case lock contention within a multi-vcpu guest is probably not as bad a problem as the various solutions cause. If a workload is suffering so much, give it multiple vcpus that are mapped 1:1 to physical cpus. Mike -- Mike D. Day IBM LTC Cell: 919 412-3900 Sametime: ncmike@us.ibm.com AIM: ncmikeday Yahoo: ultra.runner PGP key: http://www.ncultra.org/ncmike/pubkey.asc _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Mike D. Day, le Mon 21 Apr 2008 16:32:18 -0400, a écrit :> On 21/04/08 18:56 +0100, Samuel Thibault wrote: > > Nauman Rafique, le Mon 21 Apr 2008 13:52:21 -0400, a écrit : > > > In fact, wasted cycles can probably be avoided by doing opportunisitic > > > gang scheduling (i.e. gang schedule, unless there would be wasted > > > cycles) > > > > How do you detect that there would be wasted cycles? > > The only way is for a very self-aware guest to use a paravirtual > feature to give a hint to the scheduler.Agreed :)> Alternatively, the scheduler can notify the guest that it is about to > be preempted by the hypervisor and now would be a good time to sleep > before gaining a contended spinlock.Mmm, "about to be" means "on the next tick" then. That may be a long time. Samuel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
If we know that a particular VM is running a compute bound workload with synchronization, we can mark that VM to be gang scheduled. That would work better than pinning to physical cpus, since scheduler can still load balance across all physical cpus. Nevertheless, as Mike mentioned, idle cycles can be accounted for by providing hints from guest OS. Hints can be rolled into the multicall done at context switch. It seems to me that it would be desirable to have gang scheduling available as an one of the scheduler options. -- Nauman Rafique Purdue University On Mon, Apr 21, 2008 at 5:34 PM, Samuel Thibault < samuel.thibault@eu.citrix.com> wrote:> Mike D. Day, le Mon 21 Apr 2008 16:32:18 -0400, a écrit : > > On 21/04/08 18:56 +0100, Samuel Thibault wrote: > > > Nauman Rafique, le Mon 21 Apr 2008 13:52:21 -0400, a écrit : > > > > In fact, wasted cycles can probably be avoided by doing > opportunisitic > > > > gang scheduling (i.e. gang schedule, unless there would be wasted > > > > cycles) > > > > > > How do you detect that there would be wasted cycles? > > > > The only way is for a very self-aware guest to use a paravirtual > > feature to give a hint to the scheduler. > > Agreed :) > > > Alternatively, the scheduler can notify the guest that it is about to > > be preempted by the hypervisor and now would be a good time to sleep > > before gaining a contended spinlock. > > Mmm, "about to be" means "on the next tick" then. That may be a long > time. > > Samuel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel