Hi, list, A VMM provides a VMCS for each VM. How does the VMM assign system resources for each VM? For example, in a multi-core environment, how can I enable the VMM to run, say, on a two-core intel''s processor while I am able to force a VM to execute only on a specific core upon initializing the Guest OS ? that is to say, how I can assure the VM to believe that there is only one physical core running on this machine? What should I do? Is it possible to assign a core when VMM starts a new VM after the VMM is booted ? Thanks a lot! _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
I think that''s possible, you can check the cpupool related command in the xl manpage. But I have a similar question -- In CPU like i7, 8 logical cores are not fully equivalent. They are hyper-threading over CMP. The two HT cores from a single physical core are contending for pipeline resource. In linux kernel, there is specific scheduler optimization for this case. My question is that how XEN hypervisor handle this? Will such HT info be available to the VM? I guess there may be some different depend on whether you bind VCPU to host core or not. Also, is there any different for dom0 && domU in this aspect? Thanks, Timothy On Wed, Dec 19, 2012 at 2:20 PM, ZHANG Zhi <zhizhang@smu.edu.sg> wrote:> Hi, list, > > A VMM provides a VMCS for each VM. How does the VMM assign system resources > for each VM? For example, in a multi-core environment, how can I enable the > VMM to run, say, on a two-core intel’s processor while I am able to force a > VM to execute only on a specific core upon initializing the Guest OS ? that > is to say, how I can assure the VM to believe that there is only one > physical core running on this machine? > > What should I do? Is it possible to assign a core when VMM starts a new VM > after the VMM is booted ? > > Thanks a lot! > > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel >
Well, I can''t agree. HW provide features and SW use it. Otherwise, HW feature is just a vain. PS: please keep the list CCed so that people are aware of the discussion. Probably they have something to share on this topic. On Wed, Jan 9, 2013 at 1:37 PM, ZHANG Zhi <zhizhang@smu.edu.sg> wrote:> I think it''s the hardware''s business, namely the Intel or AMD, not the hyperviosr''s business. Xen does not have to consider such an issue. > > Re: > I think that''s possible, you can check the cpupool related command in the xl manpage. > > But I have a similar question -- In CPU like i7, 8 logical cores are not fully equivalent. > They are hyper-threading over CMP. The two HT cores from a single physical core are contending for pipeline resource. In linux kernel, there is specific scheduler optimization for this case. > My question is that how XEN hypervisor handle this? Will such HT info be available to the VM? > I guess there may be some different depend on whether you bind VCPU to host core or not. > > Also, is there any different for dom0 && domU in this aspect? > > Thanks, > Timothy > > On Wed, Dec 19, 2012 at 2:20 PM, ZHANG Zhi <zhizhang@smu.edu.sg> wrote: >> Hi, list, >> >> A VMM provides a VMCS for each VM. How does the VMM assign system >> resources for each VM? For example, in a multi-core environment, how >> can I enable the VMM to run, say, on a two-core intel''s processor >> while I am able to force a VM to execute only on a specific core upon >> initializing the Guest OS ? that is to say, how I can assure the VM to >> believe that there is only one physical core running on this machine? >> >> What should I do? Is it possible to assign a core when VMM starts a >> new VM after the VMM is booted ? >> >> Thanks a lot! >> >> >> >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xen.org >> http://lists.xen.org/xen-devel >>
Yeah, I think you''re right. Maybe it also depends on the virtualization techniques. If it''s hardware assisted virtualization, guest OS may detect the HT info, because it can also scan the BIOS memory, right? re: Re: [Xen-devel] multi-core VMM Well, I can''t agree. HW provide features and SW use it. Otherwise, HW feature is just a vain. PS: please keep the list CCed so that people are aware of the discussion. Probably they have something to share on this topic. On Wed, Jan 9, 2013 at 1:37 PM, ZHANG Zhi <zhizhang@smu.edu.sg> wrote:> I think it''s the hardware''s business, namely the Intel or AMD, not the hyperviosr''s business. Xen does not have to consider such an issue. > > Re: > I think that''s possible, you can check the cpupool related command in the xl manpage. > > But I have a similar question -- In CPU like i7, 8 logical cores are not fully equivalent. > They are hyper-threading over CMP. The two HT cores from a single physical core are contending for pipeline resource. In linux kernel, there is specific scheduler optimization for this case. > My question is that how XEN hypervisor handle this? Will such HT info be available to the VM? > I guess there may be some different depend on whether you bind VCPU to host core or not. > > Also, is there any different for dom0 && domU in this aspect? > > Thanks, > Timothy > > On Wed, Dec 19, 2012 at 2:20 PM, ZHANG Zhi <zhizhang@smu.edu.sg> wrote: >> Hi, list, >> >> A VMM provides a VMCS for each VM. How does the VMM assign system >> resources for each VM? For example, in a multi-core environment, how >> can I enable the VMM to run, say, on a two-core intel''s processor >> while I am able to force a VM to execute only on a specific core upon >> initializing the Guest OS ? that is to say, how I can assure the VM >> to believe that there is only one physical core running on this machine? >> >> What should I do? Is it possible to assign a core when VMM starts a >> new VM after the VMM is booted ? >> >> Thanks a lot! >> >> >> >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xen.org >> http://lists.xen.org/xen-devel >>
I''m not quite sure. Exposing HT info to guest is only part of the story (but absolutely required) HT is a tech to share compute resources provided by a single physical core so as to achieve better utilization. OS should schedule two complementary instead of competing threads to one physical core to achieve better performance. Good example includes INT thread + FP thread, calc thread + mem bounded thread etc. If two threads contents for same resources in a physical core, the overall throughput for one physical core may even drop as compare to the case of only one thread. I''m not sure how OS scheduler obtain such info, but maybe there are instruction statistics for this purpose. So it appears to me that only when the OS have full control over the core, can it schedule best for HT. (but this really depends on what kind of statistics is available. The statement above is what I can guess best.) In virtualized env, the above assumption can not be easily achieved. People may be able to manipulate the CPU affinity through cpuset etc. On the other hand, without human invention, I''m not sure if the xen scheduler is doing anything special to handle HT. On Thu, Jan 10, 2013 at 9:00 PM, ZHANG Zhi <zhizhang@smu.edu.sg> wrote:> Yeah, I think you''re right. > Maybe it also depends on the virtualization techniques. If it''s hardware assisted virtualization, guest OS may detect the HT info, because it can also scan the BIOS memory, right? > > > re: Re: [Xen-devel] multi-core VMM > > Well, I can''t agree. > HW provide features and SW use it. Otherwise, HW feature is just a vain. > > PS: please keep the list CCed so that people are aware of the discussion. Probably they have something to share on this topic. > > On Wed, Jan 9, 2013 at 1:37 PM, ZHANG Zhi <zhizhang@smu.edu.sg> wrote: >> I think it''s the hardware''s business, namely the Intel or AMD, not the hyperviosr''s business. Xen does not have to consider such an issue. >> >> Re: >> I think that''s possible, you can check the cpupool related command in the xl manpage. >> >> But I have a similar question -- In CPU like i7, 8 logical cores are not fully equivalent. >> They are hyper-threading over CMP. The two HT cores from a single physical core are contending for pipeline resource. In linux kernel, there is specific scheduler optimization for this case. >> My question is that how XEN hypervisor handle this? Will such HT info be available to the VM? >> I guess there may be some different depend on whether you bind VCPU to host core or not. >> >> Also, is there any different for dom0 && domU in this aspect? >> >> Thanks, >> Timothy >> >> On Wed, Dec 19, 2012 at 2:20 PM, ZHANG Zhi <zhizhang@smu.edu.sg> wrote: >>> Hi, list, >>> >>> A VMM provides a VMCS for each VM. How does the VMM assign system >>> resources for each VM? For example, in a multi-core environment, how >>> can I enable the VMM to run, say, on a two-core intel''s processor >>> while I am able to force a VM to execute only on a specific core upon >>> initializing the Guest OS ? that is to say, how I can assure the VM >>> to believe that there is only one physical core running on this machine? >>> >>> What should I do? Is it possible to assign a core when VMM starts a >>> new VM after the VMM is booted ? >>> >>> Thanks a lot! >>> >>> >>> >>> >>> _______________________________________________ >>> Xen-devel mailing list >>> Xen-devel@lists.xen.org >>> http://lists.xen.org/xen-devel >>>
Hi Stefano, We are discussing the topic of HT (hyper-threading) over virtualization. So far it is just out of curiosity. Do you know who is the expert that can share some comments? Thanks, Timothy On Thu, Jan 10, 2013 at 11:24 PM, G.R. <firemeteor@users.sourceforge.net> wrote:> I''m not quite sure. Exposing HT info to guest is only part of the > story (but absolutely required) > > HT is a tech to share compute resources provided by a single physical > core so as to achieve better utilization. > OS should schedule two complementary instead of competing threads to > one physical core to achieve better performance. > Good example includes INT thread + FP thread, calc thread + mem > bounded thread etc. > If two threads contents for same resources in a physical core, the > overall throughput for one physical core may even drop as compare to > the case of only one thread. > I''m not sure how OS scheduler obtain such info, but maybe there are > instruction statistics for this purpose. > > So it appears to me that only when the OS have full control over the > core, can it schedule best for HT. > (but this really depends on what kind of statistics is available. The > statement above is what I can guess best.) > > In virtualized env, the above assumption can not be easily achieved. > People may be able to manipulate the CPU affinity through cpuset etc. > On the other hand, without human invention, I''m not sure if the xen > scheduler is doing anything special to handle HT. > > On Thu, Jan 10, 2013 at 9:00 PM, ZHANG Zhi <zhizhang@smu.edu.sg> wrote: >> Yeah, I think you''re right. >> Maybe it also depends on the virtualization techniques. If it''s hardware assisted virtualization, guest OS may detect the HT info, because it can also scan the BIOS memory, right? >> >> >> re: Re: [Xen-devel] multi-core VMM >> >> Well, I can''t agree. >> HW provide features and SW use it. Otherwise, HW feature is just a vain. >> >> PS: please keep the list CCed so that people are aware of the discussion. Probably they have something to share on this topic. >> >> On Wed, Jan 9, 2013 at 1:37 PM, ZHANG Zhi <zhizhang@smu.edu.sg> wrote: >>> I think it''s the hardware''s business, namely the Intel or AMD, not the hyperviosr''s business. Xen does not have to consider such an issue. >>> >>> Re: >>> I think that''s possible, you can check the cpupool related command in the xl manpage. >>> >>> But I have a similar question -- In CPU like i7, 8 logical cores are not fully equivalent. >>> They are hyper-threading over CMP. The two HT cores from a single physical core are contending for pipeline resource. In linux kernel, there is specific scheduler optimization for this case. >>> My question is that how XEN hypervisor handle this? Will such HT info be available to the VM? >>> I guess there may be some different depend on whether you bind VCPU to host core or not. >>> >>> Also, is there any different for dom0 && domU in this aspect? >>> >>> Thanks, >>> Timothy >>> >>> On Wed, Dec 19, 2012 at 2:20 PM, ZHANG Zhi <zhizhang@smu.edu.sg> wrote: >>>> Hi, list, >>>> >>>> A VMM provides a VMCS for each VM. How does the VMM assign system >>>> resources for each VM? For example, in a multi-core environment, how >>>> can I enable the VMM to run, say, on a two-core intel''s processor >>>> while I am able to force a VM to execute only on a specific core upon >>>> initializing the Guest OS ? that is to say, how I can assure the VM >>>> to believe that there is only one physical core running on this machine? >>>> >>>> What should I do? Is it possible to assign a core when VMM starts a >>>> new VM after the VMM is booted ? >>>> >>>> Thanks a lot! >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> Xen-devel mailing list >>>> Xen-devel@lists.xen.org >>>> http://lists.xen.org/xen-devel >>>>
Before SandyBridge, it was not a good idea to have two mem bounded threads run on a single core: there was only ONE memory read port. On SandyBridge, IvvyBridge and Haswell, there are more memory ports and L3 cache is split between cores. Each core has a "CBo" (cache box) indirectly attached to 2.5MB of L3 cache. A consistent hash function across cores is calculated on cacheline address to obtain the target CBo. So there are no "locality" benefit from this chunking. I can''t see a way for schedulers to decide the policy, that would be an application developer controlled thing. So I would rather leverage HT information to actually make sure some threads are NOT on the same core rather than trying to group them. This also means that VCPU to CPU mapping needs to be controlled tightly... François-Frédéric -----Message d''origine----- De : xen-devel-bounces@lists.xen.org [mailto:xen-devel-bounces@lists.xen.org] De la part de G.R. Envoyé : jeudi 10 janvier 2013 16:25 À : ZHANG Zhi Cc : xen-devel@lists.xen.org Objet : Re: [Xen-devel] multi-core VMM I''m not quite sure. Exposing HT info to guest is only part of the story (but absolutely required) HT is a tech to share compute resources provided by a single physical core so as to achieve better utilization. OS should schedule two complementary instead of competing threads to one physical core to achieve better performance. Good example includes INT thread + FP thread, calc thread + mem bounded thread etc. If two threads contents for same resources in a physical core, the overall throughput for one physical core may even drop as compare to the case of only one thread. I''m not sure how OS scheduler obtain such info, but maybe there are instruction statistics for this purpose. So it appears to me that only when the OS have full control over the core, can it schedule best for HT. (but this really depends on what kind of statistics is available. The statement above is what I can guess best.) In virtualized env, the above assumption can not be easily achieved. People may be able to manipulate the CPU affinity through cpuset etc. On the other hand, without human invention, I''m not sure if the xen scheduler is doing anything special to handle HT. On Thu, Jan 10, 2013 at 9:00 PM, ZHANG Zhi <zhizhang@smu.edu.sg> wrote:> Yeah, I think you''re right. > Maybe it also depends on the virtualization techniques. If it''shardware assisted virtualization, guest OS may detect the HT info, because it can also scan the BIOS memory, right?> > > re: Re: [Xen-devel] multi-core VMM > > Well, I can''t agree. > HW provide features and SW use it. Otherwise, HW feature is just a vain. > > PS: please keep the list CCed so that people are aware of the discussion.Probably they have something to share on this topic.> > On Wed, Jan 9, 2013 at 1:37 PM, ZHANG Zhi <zhizhang@smu.edu.sg> wrote: >> I think it''s the hardware''s business, namely the Intel or AMD, not thehyperviosr''s business. Xen does not have to consider such an issue.>> >> Re: >> I think that''s possible, you can check the cpupool related command in thexl manpage.>> >> But I have a similar question -- In CPU like i7, 8 logical cores are notfully equivalent.>> They are hyper-threading over CMP. The two HT cores from a singlephysical core are contending for pipeline resource. In linux kernel, there is specific scheduler optimization for this case.>> My question is that how XEN hypervisor handle this? Will such HT info beavailable to the VM?>> I guess there may be some different depend on whether you bind VCPU tohost core or not.>> >> Also, is there any different for dom0 && domU in this aspect? >> >> Thanks, >> Timothy >> >> On Wed, Dec 19, 2012 at 2:20 PM, ZHANG Zhi <zhizhang@smu.edu.sg> wrote: >>> Hi, list, >>> >>> A VMM provides a VMCS for each VM. How does the VMM assign system >>> resources for each VM? For example, in a multi-core environment, how >>> can I enable the VMM to run, say, on a two-core intel''s processor >>> while I am able to force a VM to execute only on a specific core >>> upon initializing the Guest OS ? that is to say, how I can assure >>> the VM to believe that there is only one physical core running on thismachine?>>> >>> What should I do? Is it possible to assign a core when VMM starts a >>> new VM after the VMM is booted ? >>> >>> Thanks a lot! >>> >>> >>> >>> >>> _______________________________________________ >>> Xen-devel mailing list >>> Xen-devel@lists.xen.org >>> http://lists.xen.org/xen-devel >>>_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Hmm, I know that there is a ''SMT scheduler support'' in the linux kernel config. But I never tried to check out how it works. Having application developer to control is not an effective way. And I never heard of such API... Maybe I just need to check out the linux solution... I still tend to believe there are HW support for application behavior statistics. On Fri, Jan 11, 2013 at 12:14 AM, François-Frédéric Ozog <ff@ozog.com> wrote:> Before SandyBridge, it was not a good idea to have two mem bounded threads > run on a single core: there was only ONE memory read port. On SandyBridge, > IvvyBridge and Haswell, there are more memory ports and L3 cache is split > between cores. Each core has a "CBo" (cache box) indirectly attached to > 2.5MB of L3 cache. A consistent hash function across cores is calculated on > cacheline address to obtain the target CBo. So there are no "locality" > benefit from this chunking. > > I can''t see a way for schedulers to decide the policy, that would be an > application developer controlled thing. > So I would rather leverage HT information to actually make sure some threads > are NOT on the same core rather than trying to group them. > This also means that VCPU to CPU mapping needs to be controlled tightly... > > François-Frédéric > > -----Message d''origine----- > De : xen-devel-bounces@lists.xen.org > [mailto:xen-devel-bounces@lists.xen.org] De la part de G.R. > Envoyé : jeudi 10 janvier 2013 16:25 > À : ZHANG Zhi > Cc : xen-devel@lists.xen.org > Objet : Re: [Xen-devel] multi-core VMM > > I''m not quite sure. Exposing HT info to guest is only part of the story (but > absolutely required) > > HT is a tech to share compute resources provided by a single physical core > so as to achieve better utilization. > OS should schedule two complementary instead of competing threads to one > physical core to achieve better performance. > Good example includes INT thread + FP thread, calc thread + mem bounded > thread etc. > If two threads contents for same resources in a physical core, the overall > throughput for one physical core may even drop as compare to the case of > only one thread. > I''m not sure how OS scheduler obtain such info, but maybe there are > instruction statistics for this purpose. > > So it appears to me that only when the OS have full control over the core, > can it schedule best for HT. > (but this really depends on what kind of statistics is available. The > statement above is what I can guess best.) > > In virtualized env, the above assumption can not be easily achieved. > People may be able to manipulate the CPU affinity through cpuset etc. > On the other hand, without human invention, I''m not sure if the xen > scheduler is doing anything special to handle HT. > > On Thu, Jan 10, 2013 at 9:00 PM, ZHANG Zhi <zhizhang@smu.edu.sg> wrote: >> Yeah, I think you''re right. >> Maybe it also depends on the virtualization techniques. If it''s > hardware assisted virtualization, guest OS may detect the HT info, because > it can also scan the BIOS memory, right? >> >> >> re: Re: [Xen-devel] multi-core VMM >> >> Well, I can''t agree. >> HW provide features and SW use it. Otherwise, HW feature is just a vain. >> >> PS: please keep the list CCed so that people are aware of the discussion. > Probably they have something to share on this topic. >> >> On Wed, Jan 9, 2013 at 1:37 PM, ZHANG Zhi <zhizhang@smu.edu.sg> wrote: >>> I think it''s the hardware''s business, namely the Intel or AMD, not the > hyperviosr''s business. Xen does not have to consider such an issue. >>> >>> Re: >>> I think that''s possible, you can check the cpupool related command in the > xl manpage. >>> >>> But I have a similar question -- In CPU like i7, 8 logical cores are not > fully equivalent. >>> They are hyper-threading over CMP. The two HT cores from a single > physical core are contending for pipeline resource. In linux kernel, there > is specific scheduler optimization for this case. >>> My question is that how XEN hypervisor handle this? Will such HT info be > available to the VM? >>> I guess there may be some different depend on whether you bind VCPU to > host core or not. >>> >>> Also, is there any different for dom0 && domU in this aspect? >>> >>> Thanks, >>> Timothy >>> >>> On Wed, Dec 19, 2012 at 2:20 PM, ZHANG Zhi <zhizhang@smu.edu.sg> wrote: >>>> Hi, list, >>>> >>>> A VMM provides a VMCS for each VM. How does the VMM assign system >>>> resources for each VM? For example, in a multi-core environment, how >>>> can I enable the VMM to run, say, on a two-core intel''s processor >>>> while I am able to force a VM to execute only on a specific core >>>> upon initializing the Guest OS ? that is to say, how I can assure >>>> the VM to believe that there is only one physical core running on this > machine? >>>> >>>> What should I do? Is it possible to assign a core when VMM starts a >>>> new VM after the VMM is booted ? >>>> >>>> Thanks a lot! >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> Xen-devel mailing list >>>> Xen-devel@lists.xen.org >>>> http://lists.xen.org/xen-devel >>>> > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel
On Fri, 2013-01-11 at 00:44 +0800, G.R. wrote:> Hmm, I know that there is a ''SMT scheduler support'' in the linux kernel config. > But I never tried to check out how it works. >Ok, sorry for joining late this thread. In my defence, it was quite hard to follow, with wrong reply-to-s, missing messages, etc. :-) That being said, I can contribute with what I remember about Linux SMT and what I know about Xen SMT support. Well, although they''re very different if you look at the actual code, both approaches takes into account the differences between a full-fledged core/processor/whatever you want to call it, and an hyperthread (HT). Simplifying the thing quite a bit, we could say that both at least try not to run a task/vCPU on the sibling of a busy hyperthread, in case there are idle cores[1]. So, at least in Xen''s credit scheduler, if you have 3 core with 2 HT each, 2 running vCPU and a waking-up one, the scheduler will try to send the new vCPU to the idle core. That''s how it works here... Hope this helps sorting out your doubts (feel free to ask further if not). Notice that all this is well buried in the internals of the Xen (credit) scheduler, and it is _not_ exported to almost anyone. From outside the scheduler, all the ''ways'' are seen like a pCPU. Yes, `xl info -n'' (on a Xen host) will tell you which pCPUs belong to which socket, which socket(s?) to which node, etc., and you certainly can infer what pCPUs are hyperdreads, cores or full processors, but that does not change the fact that your 2 sockets, 4 cores-per-socket, 2 threads-per-cores system will be seen as a 16 pCPUs Xen host (and the same for Linux on baremetal, try `cat /proc/cpuinfo'' there and you''ll see 16 entries).> Having application developer to control is not an effective way. > And I never heard of such API... >Me neither, although that does not mean it can''t exist, at least for Linux on baremetal. Exporting such king of information to guests (here, I''m mostly interested in this, given where we are, rather than in a Linux baremetal HT API) is something that could probably be done, but not without having clear what the situation is, what limitations we would be introducing and whether the benefits surpasses the costs and pay for the efforts. :-) First of all, the vast majority of systems are still (or at least so I think) homogeneous: with hyperthread enabled, every pCPU is a thread! Pick, for instance, the system I was talking about before (2 sockets, 4 cores, 2 threads): you either keep HT enabled, and you have 16 threads, or disable it, and you have 8 "full-cores". So, in this case, we can say that either all the information is already exported, or that there is not much information to export in the first place! :-) However, let''s assume you can put your hands on an heterogeneous system, where some cores have hyperthreading enabled and others some don''t. For instance, you can sort of create one by off-lining some of the pCPUs (which, as said, are hyperthreads), assuming that leaving only one of the 2 HTs on-line on a certain core would allow us to call it a "full-core". IOW, let''s say that, on core 0, you have both its HTs on-line, so pCPU 0 and 1 are HTs, while on core 1 you off-line one of its 2 HTs (say pCPU 3), so that pCPU 2 can be considered a full core. Now, if you run a 2 vCPUs guest on such host, as I was saying above, Xen will almost always run one of the guest''s vCPU on either pCPU 0 or 1, and the other one on pCPU 3 (where the almost come from the fact that this depends on the load, since, even if the guest is the only one in the system, there always also are Dom0''s vCPUs!). Now you can think that telling the guest that one of its vCPU is a thread and one is a full-core might be nice, and it is definitely true that this would trigger the guest OS''s logic for dealing with SMT (and, as said, Linux has some). However, among its two vCPUs, which one you''d advertise to the guest as a thread and which one as a core? It''s heuristics, so it''s not impossible that you end up executing _both_ the guest''s vCPUs on the same core (i.e., on pCPUs 0 and 1), if for instance the Dom0 vCPUs are very busy, or if you start a new guest at some point, in which case you''ll be giving the guest OS the wrong information, potentially worsening the performances. The only way to avoid the above that I can think of is vCPU-to-pCPU pinning. In fact, if you statically pin, at guest creation time, vCPU 0 to pCPU 0 and vCPU 1 to pCPU 2 (thanks to the fact that you disabled, by off-lining it, pCPU 3, the sibling HT of pCPU 2), you''ll never see vCPU 1 running on a busy HT. However, this not only means that you have to pin the guest''s vCPUs at the time you create the guest itself, but also that you can''t change this during the whole guest lifetime, so, no vCPU re-pinning, no mangling with cpupools, probably not even suspension and live migration (unless you''re sure you''re going to resume the guest at a time and on a host with the exact same characteristics). So, with all this limitations, is it still worth? Don''t get me wrong, I''m not at all saying it never is, what I''m saying is it would be a solution for a very small subset of use cases, so we better concentrate on something else for now... But, hey, if something comes out that implements this (of course, if it doesn''t cause regressions on other workloads, if it doesn''t complicate the code too much, ...), I''d at least look at the patches with interest! :-P> Maybe I just need to check out the linux solution... > I still tend to believe there are HW support for application behavior > statistics. >I''m not sure I understood what you mean with "HW support for application behaviour statistics". There certainly are some information you can try to infer from things like hardware counters, etc., and since SMT scheduling (like all scheduling? :-D) is mostly heuristics, we sure could use them to try to do better. On the down side, if talking for example about hardware performance counters, most of them are hardware/CPU/chipset specific and dependant, so it''s very hard to take advantage of them in something that has to run on a wide range of CPU and chipsets, from different manufacturers and different time periods! Of course you can construct _in_software_ almost every statistic you like, but that will definitely come at some price, especially if it has to be accurate enough to be useful for scheduling decisions, which are, by their nature, a quite high frequency activity. Again, I hope this helps. Thanks and Regards, Dario [1] Unless there are power management constraints. Xen, for instance, as a parameter that you can use to tell the scheduler (well, at least the credit scheduler) to do right the opposite, i.e., consolidate work instead of spreading it -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://retis.sssup.it/people/faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On Thu, 2013-01-10 at 23:24 +0800, G.R. wrote:> On the other hand, without human invention, I''m not sure if the xen > scheduler is doing anything special to handle HT. >As I tried to explain in the other e-mail, it is doing something special. You could claim that it is doing something quite basic, which I agree, but it is the only sane thing you can do at that level that results general enough to maximize the benefits (or, if you prefer, minimize the loss) for the most of the workloads. :-) There has been a previous discussion on the list touching this subject. Here it is, in case you''re interested: http://lists.xen.org/archives/html/xen-devel/2012-11/msg00729.html Regards, Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://retis.sssup.it/people/faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Regarding scheduling stuff, it would be George Dunlap and Dario Faggioli. On Thu, 10 Jan 2013, G.R. wrote:> Hi Stefano, > We are discussing the topic of HT (hyper-threading) over virtualization. > So far it is just out of curiosity. Do you know who is the expert that > can share some comments? > > Thanks, > Timothy > > On Thu, Jan 10, 2013 at 11:24 PM, G.R. <firemeteor@users.sourceforge.net> wrote: > > I''m not quite sure. Exposing HT info to guest is only part of the > > story (but absolutely required) > > > > HT is a tech to share compute resources provided by a single physical > > core so as to achieve better utilization. > > OS should schedule two complementary instead of competing threads to > > one physical core to achieve better performance. > > Good example includes INT thread + FP thread, calc thread + mem > > bounded thread etc. > > If two threads contents for same resources in a physical core, the > > overall throughput for one physical core may even drop as compare to > > the case of only one thread. > > I''m not sure how OS scheduler obtain such info, but maybe there are > > instruction statistics for this purpose. > > > > So it appears to me that only when the OS have full control over the > > core, can it schedule best for HT. > > (but this really depends on what kind of statistics is available. The > > statement above is what I can guess best.) > > > > In virtualized env, the above assumption can not be easily achieved. > > People may be able to manipulate the CPU affinity through cpuset etc. > > On the other hand, without human invention, I''m not sure if the xen > > scheduler is doing anything special to handle HT. > > > > On Thu, Jan 10, 2013 at 9:00 PM, ZHANG Zhi <zhizhang@smu.edu.sg> wrote: > >> Yeah, I think you''re right. > >> Maybe it also depends on the virtualization techniques. If it''s hardware assisted virtualization, guest OS may detect the HT info, because it can also scan the BIOS memory, right? > >> > >> > >> re: Re: [Xen-devel] multi-core VMM > >> > >> Well, I can''t agree. > >> HW provide features and SW use it. Otherwise, HW feature is just a vain. > >> > >> PS: please keep the list CCed so that people are aware of the discussion. Probably they have something to share on this topic. > >> > >> On Wed, Jan 9, 2013 at 1:37 PM, ZHANG Zhi <zhizhang@smu.edu.sg> wrote: > >>> I think it''s the hardware''s business, namely the Intel or AMD, not the hyperviosr''s business. Xen does not have to consider such an issue. > >>> > >>> Re: > >>> I think that''s possible, you can check the cpupool related command in the xl manpage. > >>> > >>> But I have a similar question -- In CPU like i7, 8 logical cores are not fully equivalent. > >>> They are hyper-threading over CMP. The two HT cores from a single physical core are contending for pipeline resource. In linux kernel, there is specific scheduler optimization for this case. > >>> My question is that how XEN hypervisor handle this? Will such HT info be available to the VM? > >>> I guess there may be some different depend on whether you bind VCPU to host core or not. > >>> > >>> Also, is there any different for dom0 && domU in this aspect? > >>> > >>> Thanks, > >>> Timothy > >>> > >>> On Wed, Dec 19, 2012 at 2:20 PM, ZHANG Zhi <zhizhang@smu.edu.sg> wrote: > >>>> Hi, list, > >>>> > >>>> A VMM provides a VMCS for each VM. How does the VMM assign system > >>>> resources for each VM? For example, in a multi-core environment, how > >>>> can I enable the VMM to run, say, on a two-core intel''s processor > >>>> while I am able to force a VM to execute only on a specific core upon > >>>> initializing the Guest OS ? that is to say, how I can assure the VM > >>>> to believe that there is only one physical core running on this machine? > >>>> > >>>> What should I do? Is it possible to assign a core when VMM starts a > >>>> new VM after the VMM is booted ? > >>>> > >>>> Thanks a lot! > >>>> > >>>> > >>>> > >>>> > >>>> _______________________________________________ > >>>> Xen-devel mailing list > >>>> Xen-devel@lists.xen.org > >>>> http://lists.xen.org/xen-devel > >>>> >
On Fri, 2013-01-11 at 13:08 +0000, Stefano Stabellini wrote:> Regarding scheduling stuff, it would be George Dunlap and Dario > Faggioli. >Thanks Stefano. :-) Dario Faggioli already gave his view, ans is always keen on discussing these issues. :-D George is on vacation until Monday, IIRC. Apologies again if the thread slipped below my "radar" for some time. Regards, Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://retis.sssup.it/people/faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Apparently Analagous Threads
- [PATCH] xen: numa-sched: leave node-affinity alone if not in "auto" mode
- [PATCH v2] xen: sched_credit: filter node-affinity mask against online cpus
- [PATCH 0 of 3] xen: sched_credit: fix tickling and add some tracing
- [PATCH v2 0/3] nested vmx bug fixes
- [PATCH 00 of 10 v3] Automatic NUMA placement for xl