Andrew J Younge
2009-Apr-06 18:29 UTC
[Xen-devel] Issues using xenpm in 3.3.1 on Intel Nehalem cpu
Hello, I am having some issues with using the new Xen Power management features in Xen 3.3.1 and I''m need of some help. If this question should be directed elsewhere, please let me know. My goal is to be able to dynamically turn the CPU frequency up or down for each core in dom0, depending on the VM load. I am hoping to do this either with an automatic governor (ondemand) or write my own custom scripts to control this dynamically. To my knowledge, the new Core i7 Nehalem processors are able to dynamically control each core''s frequency and voltage independent of another, which is why I chose the CPU. In a normal non-xen environment (vanilla Ubuntu 8.10) I can see this working with various governors using the cpufreq-set and cpufreq-info tools that in turn rely on the acpi-cpufreq module. When Xen announced it had the same support in 3.3.1, I was very excited to be able to recreate this effect with virtual machine loads in Xen. My current setup is a Intel Core i7 2.66GHz (with hyperthreading) in a X58 motherboard with 6Gb DDR3 ram and a 300GB raptor SATA HD. I have installed Ubuntu 8.10 server x86_64 as the default OS and then compiled and installed Xen 3.3.1 from source [1],[2]. Furthermore, I am using the 2.6.27.5 xen kernel [3] for Dom0 as I had issues booting with the 2.6.18-8-xen kernel that came with the source. I have enabled my dom0 to boot with cpufreq=xen cpuidle parameters in my grub menu.conf [4] to enable use of the xenpm tool. While xenpm seems to be working on my system by reading all C states and P states (with hyperthreading enabled too), the controls do nothing. For instance, if I try to set any governor (ex: "xenpm set-scaling-governor performance"), it seems to have no effect. No matter what command I give, the default behavior is the same. If the system is idle, then it will show all cores as running at the lowest P-state of 1600MHz, which is correct. However when I create load that only would use 1 core (burnP5 &), the all cores report to run at 2668Mhz (highest setting). This is not what I want, obviously. Like I said before, the acpi-cpufreq module only scales up 1 core in such a situation. If I "xm create" a DomU (ex: Debian Lenny) and put load on that machine (burnP5), the result is no different, all P-states report at 2668Mhz. I don''t know if its actually turning all the CPUs up to 2.6Ghz, or xenpm just reporting at 2.6Ghz but doing the right thing under the hood. After a bit of google searching, I found a previous mailing list thread [5] that describes a similar issue and provides a patch fix. However it looks like this patch was issued before the release of 3.3.1 so I don''t think it applies to me. I did take a look at the code, and the patch doesn''t match up with the current code structure that I have in 3.3.1, so the patch wouldn''t install even if I tried. It also looks that there were still problems being reported after the patch that are similar to mine (all cores reporting the same freq when maybe they aren''t at the same freq). To be honest some of the details discussed in the later part of this article are beyond my current level of understanding, so if I am missing something please let me know. I have attached the output of "xm dmesg" after booting up in Dom0 and running "xenpm" once as it may be of some use. Any help with this issue would be greatly appreciated. I am really looking forward to have dynamic control of each core operating frequency in Xen as I believe some valuable research can come from this. I am not sure if there is an issue with Xen 3.3.1 and the new Core i7 system I have, or just my level of understanding. I would be happy to test any patches or experimental versions if necessary. My lab has also ordered a dual-socket quad-core Xeon 5520 system (also Nehalem based) that I plan to use in a similar way, so if there is testing that can be done on this machine too I would also be volunteer its use. Thank you in advance for any help you can provide me. Andrew References: [1] Xen 3.3.1 source code http://bits.xensource.com/oss-xen/release/3.3.1/xen-3.3.1.tar.gz [2] Instructions on installing Xen 3.3.1 on Debian http://www.howtoforge.com/virtualization-with-xen-3.3.1-on-debian-etch [3] Instructions to compile 2.6.27 dom0 kernel http://www.howtoforge.com/installing-xen-3.3-with-kernel-2.6.27-on-ubuntu-8.10-x86_64 [4] Xenpm manual http://wiki.xensource.com/xenwiki/xenpm [5] C-state and P-state conrol in Xen http://markmail.org/message/2ivrwafrue2krdox#query:depends%20on%20!PROCESSOR_EXTERNAL_CONTROL+page:1+mid:digdksyggm7ofbdp+state:results<http://markmail.org/message/2ivrwafrue2krdox#query:depends%20on%20%21PROCESSOR_EXTERNAL_CONTROL+page:1+mid:digdksyggm7ofbdp+state:results> Andrew J Younge Rochester Institute of Technology 102 Lomb Memorial Drive Rochester, NY 14623 ajy4490@rit.edu 1.585.259.5170 _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Tian, Kevin
2009-Apr-07 08:09 UTC
RE: [Xen-devel] Issues using xenpm in 3.3.1 on Intel Nehalem cpu
could you try latest Xen-unstable at which xenpm wiki is actually made? Xen3.3.1 only have one governor (ondemand) supported, and also I can't tell exactly which issues exist on that branch since the code has been improved a lot after that (such as the one you pointed out). Thanks Kevin ________________________________ From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Andrew J Younge Sent: 2009年4月7日 2:30 To: xen-devel@lists.xensource.com; Yu, Ke; Tian, Kevin Cc: Lizhe Wang; Gregor von Laszewski Subject: [Xen-devel] Issues using xenpm in 3.3.1 on Intel Nehalem cpu Hello, I am having some issues with using the new Xen Power management features in Xen 3.3.1 and I'm need of some help. If this question should be directed elsewhere, please let me know. My goal is to be able to dynamically turn the CPU frequency up or down for each core in dom0, depending on the VM load. I am hoping to do this either with an automatic governor (ondemand) or write my own custom scripts to control this dynamically. To my knowledge, the new Core i7 Nehalem processors are able to dynamically control each core's frequency and voltage independent of another, which is why I chose the CPU. In a normal non-xen environment (vanilla Ubuntu 8.10) I can see this working with various governors using the cpufreq-set and cpufreq-info tools that in turn rely on the acpi-cpufreq module. When Xen announced it had the same support in 3.3.1, I was very excited to be able to recreate this effect with virtual machine loads in Xen. My current setup is a Intel Core i7 2.66GHz (with hyperthreading) in a X58 motherboard with 6Gb DDR3 ram and a 300GB raptor SATA HD. I have installed Ubuntu 8.10 server x86_64 as the default OS and then compiled and installed Xen 3.3.1 from source [1],[2]. Furthermore, I am using the 2.6.27.5 xen kernel [3] for Dom0 as I had issues booting with the 2.6.18-8-xen kernel that came with the source. I have enabled my dom0 to boot with cpufreq=xen cpuidle parameters in my grub menu.conf [4] to enable use of the xenpm tool. While xenpm seems to be working on my system by reading all C states and P states (with hyperthreading enabled too), the controls do nothing. For instance, if I try to set any governor (ex: "xenpm set-scaling-governor performance"), it seems to have no effect. No matter what command I give, the default behavior is the same. If the system is idle, then it will show all cores as running at the lowest P-state of 1600MHz, which is correct. However when I create load that only would use 1 core (burnP5 &), the all cores report to run at 2668Mhz (highest setting). This is not what I want, obviously. Like I said before, the acpi-cpufreq module only scales up 1 core in such a situation. If I "xm create" a DomU (ex: Debian Lenny) and put load on that machine (burnP5), the result is no different, all P-states report at 2668Mhz. I don't know if its actually turning all the CPUs up to 2.6Ghz, or xenpm just reporting at 2.6Ghz but doing the right thing under the hood. After a bit of google searching, I found a previous mailing list thread [5] that describes a similar issue and provides a patch fix. However it looks like this patch was issued before the release of 3.3.1 so I don't think it applies to me. I did take a look at the code, and the patch doesn't match up with the current code structure that I have in 3.3.1, so the patch wouldn't install even if I tried. It also looks that there were still problems being reported after the patch that are similar to mine (all cores reporting the same freq when maybe they aren't at the same freq). To be honest some of the details discussed in the later part of this article are beyond my current level of understanding, so if I am missing something please let me know. I have attached the output of "xm dmesg" after booting up in Dom0 and running "xenpm" once as it may be of some use. Any help with this issue would be greatly appreciated. I am really looking forward to have dynamic control of each core operating frequency in Xen as I believe some valuable research can come from this. I am not sure if there is an issue with Xen 3.3.1 and the new Core i7 system I have, or just my level of understanding. I would be happy to test any patches or experimental versions if necessary. My lab has also ordered a dual-socket quad-core Xeon 5520 system (also Nehalem based) that I plan to use in a similar way, so if there is testing that can be done on this machine too I would also be volunteer its use. Thank you in advance for any help you can provide me. Andrew References: [1] Xen 3.3.1 source code http://bits.xensource.com/oss-xen/release/3.3.1/xen-3.3.1.tar.gz [2] Instructions on installing Xen 3.3.1 on Debian http://www.howtoforge.com/virtualization-with-xen-3.3.1-on-debian-etch [3] Instructions to compile 2.6.27 dom0 kernel http://www.howtoforge.com/installing-xen-3.3-with-kernel-2.6.27-on-ubuntu-8.10-x86_64 [4] Xenpm manual http://wiki.xensource.com/xenwiki/xenpm [5] C-state and P-state conrol in Xen http://markmail.org/message/2ivrwafrue2krdox#query:depends%20on%20!PROCESSOR_EXTERNAL_CONTROL+page:1+mid:digdksyggm7ofbdp+state:results<http://markmail.org/message/2ivrwafrue2krdox#query:depends%20on%20%21PROCESSOR_EXTERNAL_CONTROL+page:1+mid:digdksyggm7ofbdp+state:results> Andrew J Younge Rochester Institute of Technology 102 Lomb Memorial Drive Rochester, NY 14623 ajy4490@rit.edu<mailto:ajy4490@rit.edu> 1.585.259.5170 _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel