Hi, I sent the following problem on xen-user ML without an answer. I hope I''ll get one in this ML. My application is written in std C++ and it makes a matrix multiplication: so it uses only CPU and memory (no I/O, no network). I''m quite surprise that with CAP = 100% I got my results in about 600 seconds and with CAP = 50% I got my results in about 1800 seconds (around 3 times longer). For this kind of application I was expecting to get results in about 1200 seconds (2 times longer) for the second scenario with respect to the first one. Of course, the HW and SW are exactly the same for the 2 experiments. Am I wrong or the CAP mechanism is not working well? Thanks, Massimo
On Tue, May 21, 2013 at 10:02 AM, Massimo Canonico <mex@di.unipmn.it> wrote:> Hi, > I sent the following problem on xen-user ML without an answer. I hope I''ll > get one in this ML. > > My application is written in std C++ and it makes a matrix multiplication: > so it uses only CPU and memory (no I/O, no network). > > > I''m quite surprise that with CAP = 100% I got my results in about 600 > seconds and with CAP = 50% I got my results in about 1800 seconds (around 3 > times longer). > > For this kind of application I was expecting to get results in about 1200 > seconds (2 times longer) for the second scenario with respect to the first > one. > > Of course, the HW and SW are exactly the same for the 2 experiments. > > Am I wrong or the CAP mechanism is not working well?How many vcpus do you have? I''m not sure the CAP stuff actually does anything sensible with multiple cpus -- e.g., a VM with 2 vcpus maxed out will typically actually get 200%, so will a CAP force both of them to get only 25% each? The other possibility if you''ve got multiple vcpus is that you''re running into something I call "concurrency hazard": if one vcpu grabs a spinlock and then is paused in order to enforce the cap, and another vcpu tries to grab that spinlock, it will just spin until the first vcpu manages to wake up and release it again. -George
On Tue, May 21, 2013 at 11:41 AM, George Dunlap <George.Dunlap@eu.citrix.com> wrote:> On Tue, May 21, 2013 at 10:02 AM, Massimo Canonico <mex@di.unipmn.it> wrote: >> Hi, >> I sent the following problem on xen-user ML without an answer. I hope I''ll >> get one in this ML. >> >> My application is written in std C++ and it makes a matrix multiplication: >> so it uses only CPU and memory (no I/O, no network). >> >> >> I''m quite surprise that with CAP = 100% I got my results in about 600 >> seconds and with CAP = 50% I got my results in about 1800 seconds (around 3 >> times longer). >> >> For this kind of application I was expecting to get results in about 1200 >> seconds (2 times longer) for the second scenario with respect to the first >> one. >> >> Of course, the HW and SW are exactly the same for the 2 experiments. >> >> Am I wrong or the CAP mechanism is not working well? > > How many vcpus do you have? > > I''m not sure the CAP stuff actually does anything sensible with > multiple cpus -- e.g., a VM with 2 vcpus maxed out will typically > actually get 200%, so will a CAP force both of them to get only 25% > each? > > The other possibility if you''ve got multiple vcpus is that you''re > running into something I call "concurrency hazard": if one vcpu grabs > a spinlock and then is paused in order to enforce the cap, and another > vcpu tries to grab that spinlock, it will just spin until the first > vcpu manages to wake up and release it again.Sorry, meant to add: if you have more than one vcpu and you want to limit the cpu footprint, you''re much better off halving the number of vcpus rather than giving each vcpu only 50% of a cpu. I gave a presentation on why here, if you''re interested: http://www.citrix.com/tv/#videos/2930 -George
On 05/21/2013 12:48 PM, George Dunlap wrote:> On Tue, May 21, 2013 at 11:41 AM, George Dunlap > <George.Dunlap@eu.citrix.com> wrote: >> On Tue, May 21, 2013 at 10:02 AM, Massimo Canonico <mex@di.unipmn.it> wrote: >>> Hi, >>> I sent the following problem on xen-user ML without an answer. I hope I''ll >>> get one in this ML. >>> >>> My application is written in std C++ and it makes a matrix multiplication: >>> so it uses only CPU and memory (no I/O, no network). >>> >>> >>> I''m quite surprise that with CAP = 100% I got my results in about 600 >>> seconds and with CAP = 50% I got my results in about 1800 seconds (around 3 >>> times longer). >>> >>> For this kind of application I was expecting to get results in about 1200 >>> seconds (2 times longer) for the second scenario with respect to the first >>> one. >>> >>> Of course, the HW and SW are exactly the same for the 2 experiments. >>> >>> Am I wrong or the CAP mechanism is not working well? >> How many vcpus do you have? >> >> I''m not sure the CAP stuff actually does anything sensible with >> multiple cpus -- e.g., a VM with 2 vcpus maxed out will typically >> actually get 200%, so will a CAP force both of them to get only 25% >> each? >> >> The other possibility if you''ve got multiple vcpus is that you''re >> running into something I call "concurrency hazard": if one vcpu grabs >> a spinlock and then is paused in order to enforce the cap, and another >> vcpu tries to grab that spinlock, it will just spin until the first >> vcpu manages to wake up and release it again. > Sorry, meant to add: if you have more than one vcpu and you want to > limit the cpu footprint, you''re much better off halving the number of > vcpus rather than giving each vcpu only 50% of a cpu. > > I gave a presentation on why here, if you''re interested: > > http://www.citrix.com/tv/#videos/2930 > > -GeorgeHi George, thanks for your answer. Our physical machine has two processors and two cores for each processors. The virtual machine where our application run has just one VCPU that is pinned to a single core. Instead, dom0 was not pinned to any particular core. So I think that my problem could not be related to the presence of 4 vcpus. Any idea? M
On mar, 2013-05-21 at 13:54 +0200, Massimo Canonico wrote:> Hi George, > thanks for your answer. > > Our physical machine has two processors and two cores for each processors. > > The virtual machine where our application run has just one VCPU that is > pinned to a single core. > Instead, dom0 was not pinned to any particular core. >Ok, and what are you doing in Dom0? I guess almost nothing, is that the case? Also, when inspecting the vCPU utilization, do you see the VM vCPU busy up to 100% without cap (or with cap set to 100) and, OTOH, up to 50% with cap set to 50?> So I think that my problem could not be related to the presence of 4 vcpus. >Well, looks like a scheduling issue, or, in any case, one where something is interacting with the scheduling. Can you perhaps boot Dom0 so that it uses only 2 or 3 cores (or arrange for that later, e.g., with cpupools) and pin the vCPU of the VM on the 4th one? I''m not sure what could be going on, but that sounds a reasonable way to rule out as much room for interference as possible... Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Hi Dario, inline my answers On 05/21/2013 03:06 PM, Dario Faggioli wrote:> On mar, 2013-05-21 at 13:54 +0200, Massimo Canonico wrote: >> Hi George, >> thanks for your answer. >> >> Our physical machine has two processors and two cores for each processors. >> >> The virtual machine where our application run has just one VCPU that is >> pinned to a single core. >> Instead, dom0 was not pinned to any particular core. >> > Ok, and what are you doing in Dom0? I guess almost nothing, is that the > case?yes> > Also, when inspecting the vCPU utilization, do you see the VM vCPU busy > up to 100% without cap (or with cap set to 100) and, OTOH, up to 50% > with cap set to 50?I have monitored the CPU usage with xentop during the experiment and I got what you said: my application always use the total amonut of CPU available.> Well, looks like a scheduling issue, or, in any case, one where > something is interacting with the scheduling. Can you perhaps boot Dom0 > so that it uses only 2 or 3 cores (or arrange for that later, e.g., with > cpupools) and pin the vCPU of the VM on the 4th one?In my experiments, I pin the vCPU of the VM on one core and other cores are pinned to Dom0. [root@csitest ~]# xl vcpu-list Name ID VCPU CPU State Time(s) CPU Affinity Domain-0 0 0 0 --- 383.3 0 Domain-0 0 1 0 -b- 358.1 0 Domain-0 0 2 0 -b- 224.3 0 Domain-0 0 3 0 r-- 256.8 0 rubis-web 1 0 2 -b- 9250.9 2 (rubis-web is, of course, the VM where my application run) Is this configuration what you meant? Thanks, Massimo> > I''m not sure what could be going on, but that sounds a reasonable way to > rule out as much room for interference as possible... > > Dario >
On mar, 2013-05-21 at 15:28 +0100, Massimo Canonico wrote:> On 05/21/2013 03:06 PM, Dario Faggioli wrote: > > Also, when inspecting the vCPU utilization, do you see the VM vCPU busy > > up to 100% without cap (or with cap set to 100) and, OTOH, up to 50% > > with cap set to 50? > I have monitored the CPU usage with xentop during the experiment and > I got what you said: my application always use the total amonut of CPU > available. >Ok.> > Well, looks like a scheduling issue, or, in any case, one where > > something is interacting with the scheduling. Can you perhaps boot Dom0 > > so that it uses only 2 or 3 cores (or arrange for that later, e.g., with > > cpupools) and pin the vCPU of the VM on the 4th one? > In my experiments, I pin the vCPU of the VM on one core and other cores > are pinned to Dom0. > [root@csitest ~]# xl vcpu-list > Name ID VCPU CPU State Time(s) CPU > Affinity > Domain-0 0 0 0 --- 383.3 0 > Domain-0 0 1 0 -b- 358.1 0 > Domain-0 0 2 0 -b- 224.3 0 > Domain-0 0 3 0 r-- 256.8 0 > rubis-web 1 0 2 -b- 9250.9 2 > > (rubis-web is, of course, the VM where my application run) > > Is this configuration what you meant? >Sort of. However, since (you said) you have 4 cores, what I was thinking was more a situation where you have the 4 Dom0 vCPUs pinned to cores 0-2, and the VM vCPU pinned to core 3. You should be able to achieve that by doing right this: # xl vcpu-pin 0 all 0-2 # xl vcpu-pin rubis-web all 3 Does it make sense? Oh, and also, you have 4 _actual_cores_, right? Or is there any hyperthreading involved? Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Hi dario,> Sort of. However, since (you said) you have 4 cores, what I was thinking > was more a situation where you have the 4 Dom0 vCPUs pinned to cores > 0-2, and the VM vCPU pinned to core 3. > > You should be able to achieve that by doing right this: > > # xl vcpu-pin 0 all 0-2 > # xl vcpu-pin rubis-web all 3I have set pin as you suggested: [root@csitest ~]# xl vcpu-pin 0 all 0-2 [root@csitest ~]# xl vcpu-pin rubis-web all 3 [root@csitest ~]# xl vcpu-list Name ID VCPU CPU State Time(s) CPU Affinity Domain-0 0 0 0 -b- 384.5 0-2 Domain-0 0 1 2 -b- 358.9 0-2 Domain-0 0 2 2 -b- 224.6 0-2 Domain-0 0 3 1 r-- 257.0 0-2 rubis-web 1 0 3 -b- 9251.6 3> Does it make sense? Oh, and also, you have 4 _actual_cores_, right? Or > is there any hyperthreading involved?No hyperthreading are involved. At the end of this email I''ve pasted what /proc/cpuinfo says about cores (they are actual cores). The experiment just finished said again that the service time is 3 times longer when CAP=50% with respect to the scenario with CAP=100%. If you want me do more experiment, just let me know, I''ll be happy to collaborate. I can also provide you my code: it''s a very simple C++ code with a makefile that make you ready to run it in few seconds. Cheers, Massimo [root@csitest ~]# cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 15 model name : Intel(R) Xeon(R) CPU 5160 @ 3.00GHz stepping : 6 microcode : 0xd2 cpu MHz : 3000.186 cache size : 4096 KB fpu : yes fpu_exception : yes cpuid level : 10 wp : yes flags : fpu de tsc msr pae mce cx8 apic sep mca cmov pat clflush acpi mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good nopl pni monitor est ssse3 cx16 hypervisor lahf_lm dtherm bogomips : 6000.37 clflush size : 64 cache_alignment : 64 address sizes : 36 bits physical, 48 bits virtual power management: processor : 1 vendor_id : GenuineIntel cpu family : 6 model : 15 model name : Intel(R) Xeon(R) CPU 5160 @ 3.00GHz stepping : 6 microcode : 0xd2 cpu MHz : 3000.186 cache size : 4096 KB fpu : yes fpu_exception : yes cpuid level : 10 wp : yes flags : fpu de tsc msr pae mce cx8 apic sep mca cmov pat clflush acpi mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good nopl pni monitor est ssse3 cx16 hypervisor lahf_lm dtherm bogomips : 6000.37 clflush size : 64 cache_alignment : 64 address sizes : 36 bits physical, 48 bits virtual power management: processor : 2 vendor_id : GenuineIntel cpu family : 6 model : 15 model name : Intel(R) Xeon(R) CPU 5160 @ 3.00GHz stepping : 6 microcode : 0xd2 cpu MHz : 3000.186 cache size : 4096 KB fpu : yes fpu_exception : yes cpuid level : 10 wp : yes flags : fpu de tsc msr pae mce cx8 apic sep mca cmov pat clflush acpi mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good nopl pni monitor est ssse3 cx16 hypervisor lahf_lm dtherm bogomips : 6000.37 clflush size : 64 cache_alignment : 64 address sizes : 36 bits physical, 48 bits virtual power management: processor : 3 vendor_id : GenuineIntel cpu family : 6 model : 15 model name : Intel(R) Xeon(R) CPU 5160 @ 3.00GHz stepping : 6 microcode : 0xd2 cpu MHz : 3000.186 cache size : 4096 KB fpu : yes fpu_exception : yes cpuid level : 10 wp : yes flags : fpu de tsc msr pae mce cx8 apic sep mca cmov pat clflush acpi mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good nopl pni monitor est ssse3 cx16 hypervisor lahf_lm dtherm bogomips : 6000.37 clflush size : 64 cache_alignment : 64 address sizes : 36 bits physical, 48 bits virtual power management:
I have run other tests, changing the run duration but it seems that CAP mechanism has some problem. I always get response time too much longer (3 times instead of 2). Any idea about how to fix the problem? Did someone try to make a similar experiment? Thanks, M On 05/21/2013 04:47 PM, Dario Faggioli wrote:> On mar, 2013-05-21 at 15:28 +0100, Massimo Canonico wrote: >> On 05/21/2013 03:06 PM, Dario Faggioli wrote: >>> Also, when inspecting the vCPU utilization, do you see the VM vCPU busy >>> up to 100% without cap (or with cap set to 100) and, OTOH, up to 50% >>> with cap set to 50? >> I have monitored the CPU usage with xentop during the experiment and >> I got what you said: my application always use the total amonut of CPU >> available. >> > Ok. > >>> Well, looks like a scheduling issue, or, in any case, one where >>> something is interacting with the scheduling. Can you perhaps boot Dom0 >>> so that it uses only 2 or 3 cores (or arrange for that later, e.g., with >>> cpupools) and pin the vCPU of the VM on the 4th one? >> In my experiments, I pin the vCPU of the VM on one core and other cores >> are pinned to Dom0. >> [root@csitest ~]# xl vcpu-list >> Name ID VCPU CPU State Time(s) CPU >> Affinity >> Domain-0 0 0 0 --- 383.3 0 >> Domain-0 0 1 0 -b- 358.1 0 >> Domain-0 0 2 0 -b- 224.3 0 >> Domain-0 0 3 0 r-- 256.8 0 >> rubis-web 1 0 2 -b- 9250.9 2 >> >> (rubis-web is, of course, the VM where my application run) >> >> Is this configuration what you meant? >> > Sort of. However, since (you said) you have 4 cores, what I was thinking > was more a situation where you have the 4 Dom0 vCPUs pinned to cores > 0-2, and the VM vCPU pinned to core 3. > > You should be able to achieve that by doing right this: > > # xl vcpu-pin 0 all 0-2 > # xl vcpu-pin rubis-web all 3 > > Does it make sense? Oh, and also, you have 4 _actual_cores_, right? Or > is there any hyperthreading involved? > > Dario >
On mer, 2013-05-22 at 16:42 +0200, Massimo Canonico wrote:> I have run other tests, changing the run duration but > it seems that CAP mechanism has some problem. > > I always get response time too much longer (3 times instead of 2). >I see.> Any idea about how to fix the problem? >Not really, no... Although, I admit, it''s a bit since I looked at how cap was implemented in details and, even worse, I really never had the chance to use or play with it.> Did someone try to make a similar experiment? >Well, you said you can share info and code for this right? If you''re keen to give me a pointer to the source and the instructions on how you run it, I guess I can give it a try. Let me know, Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Hi Dario, first of thanks for your help.>> Did someone try to make a similar experiment? >> > Well, you said you can share info and code for this right? If you''re > keen to give me a pointer to the source and the instructions on how you > run it, I guess I can give it a try.In [1] you can find the source code. The steps to do in order to run my application follow: tar xvfz code.tgz cd code make cliSer/server cd cliSer/ ./server The application make 20 times a matrix mutiplication. At the end, it shows the time elapsed. If you need any info, please do not esitate. M [1] https://docs.google.com/file/d/0B1wfqsFZVqJlbWxrbGhMYnJmS1k/edit?usp=sharing> Let me know, > Dario >
Hi Dario and hi all, is there any news about this issue? I do not want switch to another VMM hypervisor but I need this cap mechanism working. Thanks, Massimo On 05/24/2013 09:48 AM, Massimo Canonico wrote:> Hi Dario, > first of thanks for your help. >>> Did someone try to make a similar experiment? >>> >> Well, you said you can share info and code for this right? If you''re >> keen to give me a pointer to the source and the instructions on how you >> run it, I guess I can give it a try. > In [1] you can find the source code. > The steps to do in order to run my application follow: > tar xvfz code.tgz > cd code > make cliSer/server > cd cliSer/ > ./server > > The application make 20 times a matrix mutiplication. At the end, it > shows the time elapsed. > > If you need any info, please do not esitate. > > M > > [1] > https://docs.google.com/file/d/0B1wfqsFZVqJlbWxrbGhMYnJmS1k/edit?usp=sharing > > > > >> Let me know, >> Dario >>
On mar, 2013-06-04 at 16:03 +0200, Massimo Canonico wrote:> Hi Dario and hi all, >Hi,> is there any news about this issue? >Not really I''m afraid.> I do not want switch to another VMM hypervisor but I need this cap > mechanism working. >I see. Well, I definitely want to give your scripts a try, even if it were just for personal curiosity. I hoped to be able to do that last week, let me see if I can find a slot for running your tests by today or tomorrow, ok? Regards, Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On mar, 2013-05-21 at 11:02 +0200, Massimo Canonico wrote:> Hi, >Hi again,> I sent the following problem on xen-user ML without an answer. I hope > I''ll get one in this ML. > > My application is written in std C++ and it makes a matrix > multiplication: so it uses only CPU and memory (no I/O, no network). > > I''m quite surprise that with CAP = 100% I got my results in about 600 > seconds and with CAP = 50% I got my results in about 1800 seconds > (around 3 times longer). > > For this kind of application I was expecting to get results in about > 1200 seconds (2 times longer) for the second scenario with respect to > the first one. > > Of course, the HW and SW are exactly the same for the 2 experiments. > > Am I wrong or the CAP mechanism is not working well? >Ok, I found a minute to run your code myself on my test box. It''s quite a large one, but since the VM has only 1 vcpu, that shouldn''t really make much difference. I configured vcpu-pinning in such a way that there should be no room for interference of any kind, i.e., dedicating a core to the VM, and making sure even his fellow thread is not busy (which matters in an hyperthreaded system): # xl vcpu-list Name ID VCPU CPU State Time(s) CPU Affinity Domain-0 0 0 7 -b- 38.7 0-7 Domain-0 0 1 3 -b- 2.3 0-7 Domain-0 0 2 2 -b- 3.3 0-7 Domain-0 0 3 6 -b- 6.8 0-7 Domain-0 0 4 4 -b- 3.2 0-7 Domain-0 0 5 2 -b- 3.6 0-7 Domain-0 0 6 4 -b- 2.1 0-7 Domain-0 0 7 1 -b- 1.8 0-7 Domain-0 0 8 0 -b- 2.2 0-7 Domain-0 0 9 7 -b- 1.7 0-7 Domain-0 0 10 1 -b- 1.8 0-7 Domain-0 0 11 5 r-- 10.4 0-7 Domain-0 0 12 1 -b- 3.5 0-7 Domain-0 0 13 2 -b- 3.5 0-7 Domain-0 0 14 3 -b- 2.7 0-7 Domain-0 0 15 0 -b- 1.9 0-7 vm1 1 0 11 -b- 677.0 11 The numbers I''m getting are, I think, much more consistent with the expectations: * no cap: Client served in 299.024 Client served in 298.783 Client served in 298.445 * cap 50%: Client served in 643.668 Client served in 643.372 Client served in 644.342 Which means time roughly doubles. I tried without pinning as well, and I''m getting pretty much the same values. At this point, I''m not sure what could be going on on your side. If you want to try producing some traces, we can help inspect them, looking for something weird. You can find some information about how to produce and better interpret traces in this blog post: http://blog.xen.org/index.php/2012/09/27/tracing-with-xentrace-and-xenalyze/ Perhaps you can share your VM config file and Dom0 configuration (basically, Xen and Linux boot command lines), to check whether there is something strange there. Also, you might have said this already (in which case I forgot), what versions of Xen and Linux are we talking about? I really am out of good ideas... George, any clue? Regards, Dario _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Hi Dario, and thanks for these test. I forgot to ask you which xen version has beed used for your experiments. Cheers, Massimo On 06/05/2013 07:11 PM, Dario Faggioli wrote:> On mer, 2013-06-05 at 18:50 +0200, Dario Faggioli wrote: >> The numbers I''m getting are, I think, much more consistent with the >> expectations: >> >> * no cap: >> Client served in 299.024 >> Client served in 298.783 >> Client served in 298.445 >> * cap 50%: >> Client served in 643.668 >> Client served in 643.372 >> Client served in 644.342 >> >> Which means time roughly doubles. >> > Oh, I''ve just gave it a go with cap=25%, which produced this "Client > served in 1353.14"... So still slightly more, but close enough to 4 > times the times it get when at 100%... > > Dario >
On mer, 2013-06-05 at 18:50 +0200, Dario Faggioli wrote:> The numbers I''m getting are, I think, much more consistent with the > expectations: > > * no cap: > Client served in 299.024 > Client served in 298.783 > Client served in 298.445 > * cap 50%: > Client served in 643.668 > Client served in 643.372 > Client served in 644.342 > > Which means time roughly doubles. >Oh, I''ve just gave it a go with cap=25%, which produced this "Client served in 1353.14"... So still slightly more, but close enough to 4 times the times it get when at 100%... Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On mer, 2013-06-05 at 19:05 +0200, Massimo Canonico wrote:> Hi Dario, > and thanks for these test. > > I forgot to ask you which xen version has beed used for your experiments. >Yeah, me too! :-) I''m using xen-unstable, pulled yesterday (commit id e430510e5cbbfcdc1077739292def633e70fedea), compiled and installed on a Debian unstable system. Dom0 kernel is a bit old, as it''s a 3.6. What about you? Regards, Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On 06/06/2013 10:37 AM, Dario Faggioli wrote:> On mer, 2013-06-05 at 19:05 +0200, Massimo Canonico wrote: >> Hi Dario, >> and thanks for these test. >> >> I forgot to ask you which xen version has beed used for your experiments. >> > Yeah, me too! :-) > > I''m using xen-unstable, pulled yesterday (commit id > e430510e5cbbfcdc1077739292def633e70fedea), compiled and installed on a > Debian unstable system. Dom0 kernel is a bit old, as it''s a 3.6. > > What about you?xen 4.2.2 kernel dom0: 3.8.11-200.fc18.x86_64 M> > Regards, > Dario >
On 05/06/13 17:50, Dario Faggioli wrote:> On mar, 2013-05-21 at 11:02 +0200, Massimo Canonico wrote: >> Hi, >> > Hi again, > >> I sent the following problem on xen-user ML without an answer. I hope >> I''ll get one in this ML. >> >> My application is written in std C++ and it makes a matrix >> multiplication: so it uses only CPU and memory (no I/O, no network). >> >> I''m quite surprise that with CAP = 100% I got my results in about 600 >> seconds and with CAP = 50% I got my results in about 1800 seconds >> (around 3 times longer). >> >> For this kind of application I was expecting to get results in about >> 1200 seconds (2 times longer) for the second scenario with respect to >> the first one. >> >> Of course, the HW and SW are exactly the same for the 2 experiments. >> >> Am I wrong or the CAP mechanism is not working well? >> > Ok, I found a minute to run your code myself on my test box. It''s quite > a large one, but since the VM has only 1 vcpu, that shouldn''t really > make much difference. > > I configured vcpu-pinning in such a way that there should be no room for > interference of any kind, i.e., dedicating a core to the VM, and making > sure even his fellow thread is not busy (which matters in an > hyperthreaded system): > > # xl vcpu-list > Name ID VCPU CPU State Time(s) CPU > Affinity > Domain-0 0 0 7 -b- 38.7 0-7 > Domain-0 0 1 3 -b- 2.3 0-7 > Domain-0 0 2 2 -b- 3.3 0-7 > Domain-0 0 3 6 -b- 6.8 0-7 > Domain-0 0 4 4 -b- 3.2 0-7 > Domain-0 0 5 2 -b- 3.6 0-7 > Domain-0 0 6 4 -b- 2.1 0-7 > Domain-0 0 7 1 -b- 1.8 0-7 > Domain-0 0 8 0 -b- 2.2 0-7 > Domain-0 0 9 7 -b- 1.7 0-7 > Domain-0 0 10 1 -b- 1.8 0-7 > Domain-0 0 11 5 r-- 10.4 0-7 > Domain-0 0 12 1 -b- 3.5 0-7 > Domain-0 0 13 2 -b- 3.5 0-7 > Domain-0 0 14 3 -b- 2.7 0-7 > Domain-0 0 15 0 -b- 1.9 0-7 > vm1 1 0 11 -b- 677.0 11 > > The numbers I''m getting are, I think, much more consistent with the > expectations: > > * no cap: > Client served in 299.024 > Client served in 298.783 > Client served in 298.445 > * cap 50%: > Client served in 643.668 > Client served in 643.372 > Client served in 644.342 > > Which means time roughly doubles. > > I tried without pinning as well, and I''m getting pretty much the same > values. > > At this point, I''m not sure what could be going on on your side. If you > want to try producing some traces, we can help inspect them, looking for > something weird. You can find some information about how to produce and > better interpret traces in this blog post: > > http://blog.xen.org/index.php/2012/09/27/tracing-with-xentrace-and-xenalyze/ > > Perhaps you can share your VM config file and Dom0 configuration > (basically, Xen and Linux boot command lines), to check whether there is > something strange there. Also, you might have said this already (in > which case I forgot), what versions of Xen and Linux are we talking > about? > > I really am out of good ideas... George, any clue?Well for one, from the scheduler''s perspective, the promise isn''t that you''ll get 50% of the *performance*, but 50% of the *cpu time*. I haven''t been following the thread terribly closely, but I don''t remember seeing any xentop or xentrace reports. The first question is, other than performance, do you have any reason to believe that the VM is not getting 50% of the cpu time? At some point while your test is running, could you execute the following command in dom0: xentrace -D -e 0x21000 -T 10 /tmp/test.trace This will take a 10-second trace of just the scheduling events, placing the result in /tmp/test.trace Then download and build xenalyze from the hg repo here: http://xenbits.xen.org/ext/xenalyze and run he following command: xenalyze -s /tmp/test.trace > /tmp/test.summary And post the results here? Thanks, -George
On gio, 2013-06-06 at 10:02 +0100, George Dunlap wrote:> Well for one, from the scheduler''s perspective, the promise isn''t that > you''ll get 50% of the *performance*, but 50% of the *cpu time*. I > haven''t been following the thread terribly closely, but I don''t remember > seeing any xentop or xentrace reports. >Yep, I suggested monitoring the vcpu usage, and Massimo reported a correct 50% utilization for the vcpu in question in xentop.> The first question is, other > than performance, do you have any reason to believe that the VM is not > getting 50% of the cpu time? >It correctly gets 50% for me, and for Massimo too, as per what he said in one other e-mail in the thread. I agree that it''s vcpu utilization and non guest performance we''re in control of, but for a purely CPU bound workload, I guess one can expect some sort of relationship between the two things, can''t him? Besides, that''s right what I get in my tests with Massimo''s code. :-)> At some point while your test is running, could you execute the > following command in dom0: > > xentrace -D -e 0x21000 -T 10 /tmp/test.trace > > This will take a 10-second trace of just the scheduling events, placing > the result in /tmp/test.trace > > Then download and build xenalyze from the hg repo here: > > http://xenbits.xen.org/ext/xenalyze > > and run he following command: > > xenalyze -s /tmp/test.trace > /tmp/test.summary > > And post the results here? >Yes, I suggested trying to produce some traces too. However, having seen that I get numbers consistent with the expectations, I now wonder what can interact with the scheduler and make Massimo''s results different... Regards, Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On gio, 2013-06-06 at 10:57 +0200, Massimo Canonico wrote:> On 06/06/2013 10:37 AM, Dario Faggioli wrote: > > On mer, 2013-06-05 at 19:05 +0200, Massimo Canonico wrote: > >> Hi Dario, > >> and thanks for these test. > >> > >> I forgot to ask you which xen version has beed used for your experiments. > >> > > Yeah, me too! :-) > > > > I''m using xen-unstable, pulled yesterday (commit id > > e430510e5cbbfcdc1077739292def633e70fedea), compiled and installed on a > > Debian unstable system. Dom0 kernel is a bit old, as it''s a 3.6. > > > > What about you? > xen 4.2.2 > kernel dom0: 3.8.11-200.fc18.x86_64 >I see. There shouldn''t be much differences, from a credit scheduler point of view (and even more so wrt the caps) between 4.2 and unstable. Anyway, just to rule out all the possible doubts, can you build and install unstable on your box? What about the guest? PV or HVM? What''s running in there? In my case it was a PV guest, also running a Debian unstable, and using the same kernel dom0 does. Regards, Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On 06.06.2013 10:57, Massimo Canonico wrote:> > On 06/06/2013 10:37 AM, Dario Faggioli wrote: >> On mer, 2013-06-05 at 19:05 +0200, Massimo Canonico wrote: >>> Hi Dario, >>> and thanks for these test. >>> >>> I forgot to ask you which xen version has beed used for your experiments. >>> >> Yeah, me too! :-) >> >> I''m using xen-unstable, pulled yesterday (commit id >> e430510e5cbbfcdc1077739292def633e70fedea), compiled and installed on a >> Debian unstable system. Dom0 kernel is a bit old, as it''s a 3.6. >> >> What about you? > xen 4.2.2 > kernel dom0: 3.8.11-200.fc18.x86_64Just had an idea: is there any other load on the system during your test (other domains, dom0 load)? If not, it could be that the power management is reducing the cpu speed during idle (when the cap applies). This could lead to reduced performance overall. You can test this by setting the xen hypervisor boot option cpufreq=none and run your test again (with and without cap). Juergen -- Juergen Gross Principal Developer Operating Systems PBG PDG ES&S SWE OS6 Telephone: +49 (0) 89 3222 2967 Fujitsu Technology Solutions e-mail: juergen.gross@ts.fujitsu.com Domagkstr. 28 Internet: ts.fujitsu.com D-80807 Muenchen Company details: ts.fujitsu.com/imprint.html
On 06/06/13 11:39, Juergen Gross wrote:> On 06.06.2013 10:57, Massimo Canonico wrote: >> >> On 06/06/2013 10:37 AM, Dario Faggioli wrote: >>> On mer, 2013-06-05 at 19:05 +0200, Massimo Canonico wrote: >>>> Hi Dario, >>>> and thanks for these test. >>>> >>>> I forgot to ask you which xen version has beed used for your >>>> experiments. >>>> >>> Yeah, me too! :-) >>> >>> I''m using xen-unstable, pulled yesterday (commit id >>> e430510e5cbbfcdc1077739292def633e70fedea), compiled and installed on a >>> Debian unstable system. Dom0 kernel is a bit old, as it''s a 3.6. >>> >>> What about you? >> xen 4.2.2 >> kernel dom0: 3.8.11-200.fc18.x86_64 > > Just had an idea: is there any other load on the system during your > test (other > domains, dom0 load)? If not, it could be that the power management is > reducing > the cpu speed during idle (when the cap applies). This could lead to > reduced > performance overall. > > You can test this by setting the xen hypervisor boot option > > cpufreq=none > > and run your test again (with and without cap).Ah, genius Juergen! That would make total sense. -George
On 06/06/2013 12:44 PM, George Dunlap wrote:> On 06/06/13 11:39, Juergen Gross wrote: >> On 06.06.2013 10:57, Massimo Canonico wrote: >>> >>> On 06/06/2013 10:37 AM, Dario Faggioli wrote: >>>> On mer, 2013-06-05 at 19:05 +0200, Massimo Canonico wrote: >>>>> Hi Dario, >>>>> and thanks for these test. >>>>> >>>>> I forgot to ask you which xen version has beed used for your >>>>> experiments. >>>>> >>>> Yeah, me too! :-) >>>> >>>> I''m using xen-unstable, pulled yesterday (commit id >>>> e430510e5cbbfcdc1077739292def633e70fedea), compiled and installed on a >>>> Debian unstable system. Dom0 kernel is a bit old, as it''s a 3.6. >>>> >>>> What about you? >>> xen 4.2.2 >>> kernel dom0: 3.8.11-200.fc18.x86_64 >> >> Just had an idea: is there any other load on the system during your >> test (other >> domains, dom0 load)? If not, it could be that the power management is >> reducing >> the cpu speed during idle (when the cap applies). This could lead to >> reduced >> performance overall. >> >> You can test this by setting the xen hypervisor boot option >> >> cpufreq=none >> >> and run your test again (with and without cap). > > Ah, genius Juergen! That would make total sense. > > -George >Unfortunately, this did not change much. I set "cpufreq=none" in the boot line and restart my experiment. With no cap I got 298.029 with cap=50% I got 910.272 (average values of 3 experiments for each cap setting) dom0 load during the experiment is less than 1% (that says xentop) Thanks, M
On 06/06/13 13:52, Massimo Canonico wrote:> > On 06/06/2013 12:44 PM, George Dunlap wrote: >> On 06/06/13 11:39, Juergen Gross wrote: >>> On 06.06.2013 10:57, Massimo Canonico wrote: >>>> >>>> On 06/06/2013 10:37 AM, Dario Faggioli wrote: >>>>> On mer, 2013-06-05 at 19:05 +0200, Massimo Canonico wrote: >>>>>> Hi Dario, >>>>>> and thanks for these test. >>>>>> >>>>>> I forgot to ask you which xen version has beed used for your >>>>>> experiments. >>>>>> >>>>> Yeah, me too! :-) >>>>> >>>>> I''m using xen-unstable, pulled yesterday (commit id >>>>> e430510e5cbbfcdc1077739292def633e70fedea), compiled and installed >>>>> on a >>>>> Debian unstable system. Dom0 kernel is a bit old, as it''s a 3.6. >>>>> >>>>> What about you? >>>> xen 4.2.2 >>>> kernel dom0: 3.8.11-200.fc18.x86_64 >>> >>> Just had an idea: is there any other load on the system during your >>> test (other >>> domains, dom0 load)? If not, it could be that the power management >>> is reducing >>> the cpu speed during idle (when the cap applies). This could lead to >>> reduced >>> performance overall. >>> >>> You can test this by setting the xen hypervisor boot option >>> >>> cpufreq=none >>> >>> and run your test again (with and without cap). >> >> Ah, genius Juergen! That would make total sense. >> >> -George >> > Unfortunately, this did not change much. I set "cpufreq=none" in the > boot line and restart my experiment. > With no cap I got 298.029 > with cap=50% I got 910.272 > (average values of 3 experiments for each cap setting) > > dom0 load during the experiment is less than 1% (that says xentop)Have you checked your BIOS for performance settings? -George
On 06.06.2013 14:52, Massimo Canonico wrote:> > On 06/06/2013 12:44 PM, George Dunlap wrote: >> On 06/06/13 11:39, Juergen Gross wrote: >>> On 06.06.2013 10:57, Massimo Canonico wrote: >>>> >>>> On 06/06/2013 10:37 AM, Dario Faggioli wrote: >>>>> On mer, 2013-06-05 at 19:05 +0200, Massimo Canonico wrote: >>>>>> Hi Dario, >>>>>> and thanks for these test. >>>>>> >>>>>> I forgot to ask you which xen version has beed used for your experiments. >>>>>> >>>>> Yeah, me too! :-) >>>>> >>>>> I''m using xen-unstable, pulled yesterday (commit id >>>>> e430510e5cbbfcdc1077739292def633e70fedea), compiled and installed on a >>>>> Debian unstable system. Dom0 kernel is a bit old, as it''s a 3.6. >>>>> >>>>> What about you? >>>> xen 4.2.2 >>>> kernel dom0: 3.8.11-200.fc18.x86_64 >>> >>> Just had an idea: is there any other load on the system during your test (other >>> domains, dom0 load)? If not, it could be that the power management is reducing >>> the cpu speed during idle (when the cap applies). This could lead to reduced >>> performance overall. >>> >>> You can test this by setting the xen hypervisor boot option >>> >>> cpufreq=none >>> >>> and run your test again (with and without cap). >> >> Ah, genius Juergen! That would make total sense. >> >> -George >> > Unfortunately, this did not change much. I set "cpufreq=none" in the boot lineYou added the boot parameter for the hypervisor, not dom0? And (please forgive my paranoia) you rebooted the complete system after that?> and restart my experiment. > With no cap I got 298.029 > with cap=50% I got 910.272 > (average values of 3 experiments for each cap setting) > > dom0 load during the experiment is less than 1% (that says xentop)What was the load reported by xentop for your domu? Could you try: xl vcpu-list; sleep 10; xl vcpu-list when the test is running and post the output? Juergen -- Juergen Gross Principal Developer Operating Systems PBG PDG ES&S SWE OS6 Telephone: +49 (0) 89 3222 2967 Fujitsu Technology Solutions e-mail: juergen.gross@ts.fujitsu.com Domagkstr. 28 Internet: ts.fujitsu.com D-80807 Muenchen Company details: ts.fujitsu.com/imprint.html
On gio, 2013-06-06 at 14:52 +0200, Massimo Canonico wrote:> On 06/06/2013 12:44 PM, George Dunlap wrote: > >> You can test this by setting the xen hypervisor boot option > >> > >> cpufreq=none > >> > >> and run your test again (with and without cap). > > > > Ah, genius Juergen! That would make total sense. > > > > -George > > > Unfortunately, this did not change much. I set "cpufreq=none" in the > boot line and restart my experiment. > With no cap I got 298.029 > with cap=50% I got 910.272 > (average values of 3 experiments for each cap setting) >Mmm... 298.029 is a lot faster than what you reported before, though... Are this the exact same experiments? In fact, in the first e-mail of this thread you said that "with CAP 100% I got my results in about 600 seconds". So, have you changed something or the cpufred=none did at least have some effect? Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On 06/06/2013 03:03 PM, Dario Faggioli wrote:> On gio, 2013-06-06 at 14:52 +0200, Massimo Canonico wrote: >> On 06/06/2013 12:44 PM, George Dunlap wrote: >>>> You can test this by setting the xen hypervisor boot option >>>> >>>> cpufreq=none >>>> >>>> and run your test again (with and without cap). >>> Ah, genius Juergen! That would make total sense. >>> >>> -George >>> >> Unfortunately, this did not change much. I set "cpufreq=none" in the >> boot line and restart my experiment. >> With no cap I got 298.029 >> with cap=50% I got 910.272 >> (average values of 3 experiments for each cap setting) >> > Mmm... 298.029 is a lot faster than what you reported before, though... > Are this the exact same experiments? > > In fact, in the first e-mail of this thread you said that "with CAP > 100% I got my results in about 600 seconds". So, have you changed > something or the cpufred=none did at least have some effect? > > DarioHi Dario, I only changed the workload from "40 times matrix multiplications" to "20 times matrix multiplications" to get the results faster, since I do not think that this makes any difference in perfomance comparisin between cap/no cap. M
On 06/06/2013 03:02 PM, Juergen Gross wrote:> On 06.06.2013 14:52, Massimo Canonico wrote: >> >> On 06/06/2013 12:44 PM, George Dunlap wrote: >>> On 06/06/13 11:39, Juergen Gross wrote: >>>> On 06.06.2013 10:57, Massimo Canonico wrote: >>>>> >>>>> On 06/06/2013 10:37 AM, Dario Faggioli wrote: >>>>>> On mer, 2013-06-05 at 19:05 +0200, Massimo Canonico wrote: >>>>>>> Hi Dario, >>>>>>> and thanks for these test. >>>>>>> >>>>>>> I forgot to ask you which xen version has beed used for your >>>>>>> experiments. >>>>>>> >>>>>> Yeah, me too! :-) >>>>>> >>>>>> I''m using xen-unstable, pulled yesterday (commit id >>>>>> e430510e5cbbfcdc1077739292def633e70fedea), compiled and installed >>>>>> on a >>>>>> Debian unstable system. Dom0 kernel is a bit old, as it''s a 3.6. >>>>>> >>>>>> What about you? >>>>> xen 4.2.2 >>>>> kernel dom0: 3.8.11-200.fc18.x86_64 >>>> >>>> Just had an idea: is there any other load on the system during your >>>> test (other >>>> domains, dom0 load)? If not, it could be that the power management >>>> is reducing >>>> the cpu speed during idle (when the cap applies). This could lead >>>> to reduced >>>> performance overall. >>>> >>>> You can test this by setting the xen hypervisor boot option >>>> >>>> cpufreq=none >>>> >>>> and run your test again (with and without cap). >>> >>> Ah, genius Juergen! That would make total sense. >>> >>> -George >>> >> Unfortunately, this did not change much. I set "cpufreq=none" in the >> boot line > > You added the boot parameter for the hypervisor, not dom0?Fedora, after few seconds, asks you which kernel do you want to use. You can add some parameter in the command line who launches the kernel. So, I add "cpufreq=none" in the command line.> And (please forgive > my paranoia) you rebooted the complete system after that?Your paranoia is also mine, I always reboot my machines. Thanks for asking.>> and restart my experiment. >> With no cap I got 298.029 >> with cap=50% I got 910.272 >> (average values of 3 experiments for each cap setting) >> >> dom0 load during the experiment is less than 1% (that says xentop) > > What was the load reported by xentop for your domu?My virtual machines is called rubis-web and during "cap=50%" experiment I can see this values in xentop for this specific domain.> > Could you try: > > xl vcpu-list; sleep 10; xl vcpu-list > > when the test is running and post the output? > > > Juergen >here we go: [root@csitest ~]# xl vcpu-list; sleep 10; xl vcpu-list Name ID VCPU CPU State Time(s) CPU Affinity Domain-0 0 0 0 --- 23.0 0 Domain-0 0 1 0 --- 10.5 0 Domain-0 0 2 0 --- 8.5 0 Domain-0 0 3 0 r-- 6.8 0 rubis-web 1 0 2 r-- 2968.5 2 Name ID VCPU CPU State Time(s) CPU Affinity Domain-0 0 0 0 --- 23.0 0 Domain-0 0 1 0 r-- 10.6 0 Domain-0 0 2 0 --- 8.5 0 Domain-0 0 3 0 --- 6.8 0 rubis-web 1 0 2 r-- 2973.5 2 Concerning the George''s question:> Have you checked your BIOS for performance settings?I''m not sure what you mean for "BIOS perfomance settings". To my best knowledge, in the BIOS I have to be sure that the "hw virtualization" is enabled. Thanks for your time, M
On 06/06/13 14:55, Massimo Canonico wrote:> > On 06/06/2013 03:02 PM, Juergen Gross wrote: >> On 06.06.2013 14:52, Massimo Canonico wrote: >>> >>> On 06/06/2013 12:44 PM, George Dunlap wrote: >>>> On 06/06/13 11:39, Juergen Gross wrote: >>>>> On 06.06.2013 10:57, Massimo Canonico wrote: >>>>>> >>>>>> On 06/06/2013 10:37 AM, Dario Faggioli wrote: >>>>>>> On mer, 2013-06-05 at 19:05 +0200, Massimo Canonico wrote: >>>>>>>> Hi Dario, >>>>>>>> and thanks for these test. >>>>>>>> >>>>>>>> I forgot to ask you which xen version has beed used for your >>>>>>>> experiments. >>>>>>>> >>>>>>> Yeah, me too! :-) >>>>>>> >>>>>>> I''m using xen-unstable, pulled yesterday (commit id >>>>>>> e430510e5cbbfcdc1077739292def633e70fedea), compiled and >>>>>>> installed on a >>>>>>> Debian unstable system. Dom0 kernel is a bit old, as it''s a 3.6. >>>>>>> >>>>>>> What about you? >>>>>> xen 4.2.2 >>>>>> kernel dom0: 3.8.11-200.fc18.x86_64 >>>>> >>>>> Just had an idea: is there any other load on the system during >>>>> your test (other >>>>> domains, dom0 load)? If not, it could be that the power management >>>>> is reducing >>>>> the cpu speed during idle (when the cap applies). This could lead >>>>> to reduced >>>>> performance overall. >>>>> >>>>> You can test this by setting the xen hypervisor boot option >>>>> >>>>> cpufreq=none >>>>> >>>>> and run your test again (with and without cap). >>>> >>>> Ah, genius Juergen! That would make total sense. >>>> >>>> -George >>>> >>> Unfortunately, this did not change much. I set "cpufreq=none" in the >>> boot line >> >> You added the boot parameter for the hypervisor, not dom0? > Fedora, after few seconds, asks you which kernel do you want to use. > You can add some parameter in the command line who launches the > kernel. So, I add "cpufreq=none" in the command line. >> And (please forgive >> my paranoia) you rebooted the complete system after that? > Your paranoia is also mine, I always reboot my machines. Thanks for > asking. >>> and restart my experiment. >>> With no cap I got 298.029 >>> with cap=50% I got 910.272 >>> (average values of 3 experiments for each cap setting) >>> >>> dom0 load during the experiment is less than 1% (that says xentop) >> >> What was the load reported by xentop for your domu? > My virtual machines is called rubis-web and during "cap=50%" > experiment I can see this values in xentop for this specific domain. >> >> Could you try: >> >> xl vcpu-list; sleep 10; xl vcpu-list >> >> when the test is running and post the output? >> >> >> Juergen >> > here we go: > > [root@csitest ~]# xl vcpu-list; sleep 10; xl vcpu-list > Name ID VCPU CPU State Time(s) CPU > Affinity > Domain-0 0 0 0 --- 23.0 0 > Domain-0 0 1 0 --- 10.5 0 > Domain-0 0 2 0 --- 8.5 0 > Domain-0 0 3 0 r-- 6.8 0 > rubis-web 1 0 2 r-- 2968.5 2 > Name ID VCPU CPU State Time(s) CPU > Affinity > Domain-0 0 0 0 --- 23.0 0 > Domain-0 0 1 0 r-- 10.6 0 > Domain-0 0 2 0 --- 8.5 0 > Domain-0 0 3 0 --- 6.8 0 > rubis-web 1 0 2 r-- 2973.5 2 > > Concerning the George''s question: >> Have you checked your BIOS for performance settings? > I''m not sure what you mean for "BIOS perfomance settings". To my best > knowledge, in the BIOS I have to be sure that the "hw virtualization" > is enabled.Some BIOSes have settings that will automatically mess around with the performance settings of the processor. This could be automatically slowing the core down because it''s only 50% busy. This would be different on each BIOS. Just take a quick look for anything that seems to say something about performance, and set it to "max" or "performance" mode (as opposed to say, power-saving or balanced mode). You might also try playing around with the "turbo" mode if it''s available -- maybe the "turbo" mode takes a bit of time to get going, and running at 50% never kicks it in. If you disable "turbo" mode, you may find that with no cap you get 450s instead of 300s. -George
On gio, 2013-06-06 at 15:55 +0200, Massimo Canonico wrote:> On 06/06/2013 03:02 PM, Juergen Gross wrote: > > You added the boot parameter for the hypervisor, not dom0? > Fedora, after few seconds, asks you which kernel do you want to use. You > can add some parameter in the command line who launches the kernel. So, > I add "cpufreq=none" in the command line. >IIRC, that won''t do, as you''re most likely modifying Dom0 command line, you need to be sure you''re adding that bit to the Xen boot command line. I think you can do that dynamically, by editing the grub2 parameters on-line, as you''re saying you did, but you have to make sure you affect Xen and not Linux. IIRC Otherwise, you should go to /etc/default/grub (I think), look for something like GRUB_CMDLINE_XEN (I think) add the "cpufreq=none" string there and regenerate grub.cfg by running: # grub2-mkconfig -o /boot/grub2/grub.cfg After that, I recommend double checking that /boot/grub2/grub.cfg has it, and has it in the right place. Feel free to post it here, if you''re not sure about that...> > Could you try: > > > > xl vcpu-list; sleep 10; xl vcpu-list > > > > when the test is running and post the output? > > > > > > Juergen > > > here we go: > > [root@csitest ~]# xl vcpu-list; sleep 10; xl vcpu-list > Name ID VCPU CPU State Time(s) CPU > Affinity > Domain-0 0 0 0 --- 23.0 0 > Domain-0 0 1 0 --- 10.5 0 > Domain-0 0 2 0 --- 8.5 0 > Domain-0 0 3 0 r-- 6.8 0 > rubis-web 1 0 2 r-- 2968.5 2 > Name ID VCPU CPU State Time(s) CPU > Affinity > Domain-0 0 0 0 --- 23.0 0 > Domain-0 0 1 0 r-- 10.6 0 > Domain-0 0 2 0 --- 8.5 0 > Domain-0 0 3 0 --- 6.8 0 > rubis-web 1 0 2 r-- 2973.5 2 >So, rubis-web''s vcpu ran for 5 secs, which sounds right. I see why Jurgen asked you to do that, but not how he plans to squeeze anything more than that from these numbers... Waiting for him to enlighten us again! ;-P> Concerning the George''s question: > > Have you checked your BIOS for performance settings? > I''m not sure what you mean for "BIOS perfomance settings". To my best > knowledge, in the BIOS I have to be sure that the "hw virtualization" is > enabled. >I think he was referring to something related to poerformance vs. power management, i.e., cpufreq/speedstep tweaks, and stuff like that, if ou have any in the BIOS... Do you? Regards, Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On 06/06/2013 04:12 PM, Dario Faggioli wrote:> On gio, 2013-06-06 at 15:55 +0200, Massimo Canonico wrote: >> On 06/06/2013 03:02 PM, Juergen Gross wrote: >>> You added the boot parameter for the hypervisor, not dom0? >> Fedora, after few seconds, asks you which kernel do you want to use. You >> can add some parameter in the command line who launches the kernel. So, >> I add "cpufreq=none" in the command line. >> > IIRC, that won''t do, as you''re most likely modifying Dom0 command line, > you need to be sure you''re adding that bit to the Xen boot command > line. > > I think you can do that dynamically, by editing the grub2 parameters > on-line, as you''re saying you did, but you have to make sure you affect > Xen and not Linux. IIRC > > Otherwise, you should go to /etc/default/grub (I think), look for > something like GRUB_CMDLINE_XEN (I think) add the "cpufreq=none" string > there and regenerate grub.cfg by running: > > # grub2-mkconfig -o /boot/grub2/grub.cfg > > After that, I recommend double checking that /boot/grub2/grub.cfg has > it, and has it in the right place. > > Feel free to post it here, if you''re not sure about that... > >>> Could you try: >>> >>> xl vcpu-list; sleep 10; xl vcpu-list >>> >>> when the test is running and post the output? >>> >>> >>> Juergen >>> >> here we go: >> >> [root@csitest ~]# xl vcpu-list; sleep 10; xl vcpu-list >> Name ID VCPU CPU State Time(s) CPU >> Affinity >> Domain-0 0 0 0 --- 23.0 0 >> Domain-0 0 1 0 --- 10.5 0 >> Domain-0 0 2 0 --- 8.5 0 >> Domain-0 0 3 0 r-- 6.8 0 >> rubis-web 1 0 2 r-- 2968.5 2 >> Name ID VCPU CPU State Time(s) CPU >> Affinity >> Domain-0 0 0 0 --- 23.0 0 >> Domain-0 0 1 0 r-- 10.6 0 >> Domain-0 0 2 0 --- 8.5 0 >> Domain-0 0 3 0 --- 6.8 0 >> rubis-web 1 0 2 r-- 2973.5 2 >> > So, rubis-web''s vcpu ran for 5 secs, which sounds right. I see why > Jurgen asked you to do that, but not how he plans to squeeze anything > more than that from these numbers... Waiting for him to enlighten us > again! ;-P > >> Concerning the George''s question: >>> Have you checked your BIOS for performance settings? >> I''m not sure what you mean for "BIOS perfomance settings". To my best >> knowledge, in the BIOS I have to be sure that the "hw virtualization" is >> enabled. >> > I think he was referring to something related to poerformance vs. power > management, i.e., cpufreq/speedstep tweaks, and stuff like that, if ou > have any in the BIOS... Do you? > > Regards, > Dario >Actually it seems that the problem is related to a BIOS setting. In my HP server, there were a parameter called "HP power regulator" that I set to "HP Static High-performance power" (before it was set to "dynamic performance power") and now I get what it is expected: cap 50% make my application twice longer than when cap is 100%. Thanks to all of you guys. I think that this should be reported in the doc where the cap mechanism is explained. Cheer, Massimo
On gio, 2013-06-06 at 17:27 +0200, Massimo Canonico wrote:> Actually it seems that the problem is related to a BIOS setting. In my > HP server, there were a parameter called "HP power regulator" that I set > to "HP Static High-performance power" (before it was set to "dynamic > performance power") and now I get what it is expected: cap 50% make my > application twice longer than when cap is 100%. >Cool! :-)> Thanks to all of you guys. I think that this should be reported in the > doc where the cap mechanism is explained. >Well, yes, I guess we can put there something like the following: "Beware of bugous BIOSes, providing obscure options with fancy names, promising to solve world hunger and all the other nasty problems out there if you check them, while all they do is just messing up with your OS power management strategies (for your ease and convenience)" :-D :-D Jokes apart, we may want to add a line there, perhaps on the Wiki, but I honestly don''t think this is by any means the cap mechanism''s. :-) Glad you have it working... I guess that means you can stay on Xen, right? :-P Regards, Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On gio, 2013-06-06 at 17:42 +0200, Dario Faggioli wrote:> Jokes apart, we may want to add a line there, perhaps on the Wiki, but I > honestly don''t think this is by any means the cap mechanism''s. :-)^fault What I meant is: I don''t think this can be considered cap''s fault. Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
On 06/06/13 16:54, Dario Faggioli wrote:> On gio, 2013-06-06 at 17:42 +0200, Dario Faggioli wrote: >> Jokes apart, we may want to add a line there, perhaps on the Wiki, but I >> honestly don''t think this is by any means the cap mechanism''s. :-) > ^fault > > What I meant is: I don''t think this can be considered cap''s fault.It''s certainly not cap''s fault, but it wouldn''t hurt to just put a comment saying something like, "NB that many systems have features that will scale down the computing power of a cpu that is not 100% utilized. This can be in the operating system, but can also sometimes be below the operating system in the BIOS. If you set a cap such that individual cores are running at less than 100%, this may have an impact on the performance of your workload over and above the impact of the cap. For example, a workload that may take 300s with a cap of 100% may take 900s with a cap of 50%, because in addition to being run only 50% of the time, the cpu is running in a lower state and so the 50% you get is less powerful. If this is a problem, look at performance and cpufreq options in our operating system and your BIOS." -George
On 06.06.2013 16:12, Dario Faggioli wrote:> On gio, 2013-06-06 at 15:55 +0200, Massimo Canonico wrote: >> On 06/06/2013 03:02 PM, Juergen Gross wrote: >>> Could you try: >>> >>> xl vcpu-list; sleep 10; xl vcpu-list >>> >>> when the test is running and post the output? >>> >>> >>> Juergen >>> >> here we go: >> >> [root@csitest ~]# xl vcpu-list; sleep 10; xl vcpu-list >> Name ID VCPU CPU State Time(s) CPU >> Affinity >> Domain-0 0 0 0 --- 23.0 0 >> Domain-0 0 1 0 --- 10.5 0 >> Domain-0 0 2 0 --- 8.5 0 >> Domain-0 0 3 0 r-- 6.8 0 >> rubis-web 1 0 2 r-- 2968.5 2 >> Name ID VCPU CPU State Time(s) CPU >> Affinity >> Domain-0 0 0 0 --- 23.0 0 >> Domain-0 0 1 0 r-- 10.6 0 >> Domain-0 0 2 0 --- 8.5 0 >> Domain-0 0 3 0 --- 6.8 0 >> rubis-web 1 0 2 r-- 2973.5 2 >> > So, rubis-web''s vcpu ran for 5 secs, which sounds right. I see why > Jurgen asked you to do that, but not how he plans to squeeze anything > more than that from these numbers... Waiting for him to enlighten us > again! ;-PJust for the records: I only wanted to be sure the problem isn''t in the xen scheduler. And this seems to be proved by the numbers. Juergen -- Juergen Gross Principal Developer Operating Systems PBG PDG ES&S SWE OS6 Telephone: +49 (0) 89 3222 2967 Fujitsu Technology Solutions e-mail: juergen.gross@ts.fujitsu.com Domagkstr. 28 Internet: ts.fujitsu.com D-80807 Muenchen Company details: ts.fujitsu.com/imprint.html
On 06/06/2013 06:27 PM, George Dunlap wrote:> On 06/06/13 16:54, Dario Faggioli wrote: >> On gio, 2013-06-06 at 17:42 +0200, Dario Faggioli wrote: >>> Jokes apart, we may want to add a line there, perhaps on the Wiki, >>> but I >>> honestly don''t think this is by any means the cap mechanism''s. :-) >> ^fault >> >> What I meant is: I don''t think this can be considered cap''s fault. > > It''s certainly not cap''s fault, but it wouldn''t hurt to just put a > comment saying something like, > > "NB that many systems have features that will scale down the computing > power of a cpu that is not 100% utilized. This can be in the > operating system, but can also sometimes be below the operating system > in the BIOS. If you set a cap such that individual cores are running > at less than 100%, this may have an impact on the performance of your > workload over and above the impact of the cap. For example, a workload > that may take 300s with a cap of 100% may take 900s with a cap of 50%, > because in addition to being run only 50% of the time, the cpu is > running in a lower state and so the 50% you get is less powerful. If > this is a problem, look at performance and cpufreq options in our > operating system and your BIOS." > > -GeorgeThanks George, this is exactly what I meant and I think it is quite important to put this "warning" in evidence. M