Right now, CPU frequency scaling in Xen does not work (right) for a
combination of reasons:
1) dom0 might not run on all physical CPUs
2) dom0 might not even have enough VCPUs to fill all physical CPUs
3) dom0 has no idea how busy the other guests are, so it does not
know whether to scale the CPU frequencies up or down
I can see two ways to fix this:
1) move cpu frequency scaling into the hypervisor, since the hypervisor
knows how busy all the guests are
2) make sure every physical CPU core has a dom0 VCPU pinned to it, and
have dom0 poll the hypervisor to find out how busy each CPU is,
scaling the frequencies of the CPUs as required
These approaches both have their drawbacks. One requires porting over
a series of drivers from Linux to the hypervisor, maintaining two sets
into the future. The second could mess up some of the credit
scheduler''s balancing, and requires Linux modifications to poll the
hypervisor for the CPU load...
Also, it would mean the end of running a 1 vcpu small dom0, which some
people seem to like.
Which of these approaches would you prefer?
Is there a 3rd option?
--
Who do you trust?
The people with all the right answers?
Or the people with the right questions?
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
On 25/10/06 7:04 pm, "Rik van Riel" <riel@redhat.com> wrote:> These approaches both have their drawbacks. One requires porting over > a series of drivers from Linux to the hypervisor, maintaining two sets > into the future. The second could mess up some of the credit > scheduler''s balancing, and requires Linux modifications to poll the > hypervisor for the CPU load... > > Also, it would mean the end of running a 1 vcpu small dom0, which some > people seem to like. > > Which of these approaches would you prefer?Frequency changes in dom0, please. The ''small dom0'' issue is easily handled -- create a driver domain to handle the frequency management. Or move the potentially resource-intensive processes in dom0 out to a domU (amounts to the same thing, really). It ought to take little enough resource that having a VCPU per PCPU should not affect scheduling of more user-visible VMs. We want to have the flexibility to implement cpufreq policy in a user-space process anyway, so the feedback loop requires an interface to be punched through between Xen and a cpufreq-managing guest. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser wrote:> Frequency changes in dom0, please. The ''small dom0'' issue is easily handled > -- create a driver domain to handle the frequency management. Or move the > potentially resource-intensive processes in dom0 out to a domUor have a cpufreq "driver domain"... OK, I''ll take a look at how we can do these things. I''m guessing the way to start would be having either userspace cpufreq code polling the hypervisor for CPU load, or have a drivers/cpufreq/cpufreq_ondemand-xen.c that gets the CPU load information from the hypervisor. I''m guessing something like xentop already gets the right info from the hypervisor in a relatively efficient way, or some of the code in libvirt. I''ll dig around to see what code to hook into... -- Who do you trust? The people with all the right answers? Or the people with the right questions? _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> -----Original Message----- > From: xen-devel-bounces@lists.xensource.com > [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of > Rik van Riel > Sent: 25 October 2006 22:13 > To: Keir Fraser > Cc: xen-devel@lists.xensource.com > Subject: Re: [Xen-devel] RFC: cpu frequency scaling > > Keir Fraser wrote: > > > Frequency changes in dom0, please. The ''small dom0'' issue > is easily handled > > -- create a driver domain to handle the frequency > management. Or move the > > potentially resource-intensive processes in dom0 out to a domU > > or have a cpufreq "driver domain"... > > OK, I''ll take a look at how we can do these things. > > I''m guessing the way to start would be having either userspace > cpufreq code polling the hypervisor for CPU load, or have a > drivers/cpufreq/cpufreq_ondemand-xen.c that gets the CPU load > information from the hypervisor. > > I''m guessing something like xentop already gets the right info > from the hypervisor in a relatively efficient way, or some of > the code in libvirt. > > I''ll dig around to see what code to hook into...Consider that dual core CPU''s (at least from AMD) will have to have the same ("choose highest") CPU-speed both cores. This means that you need to know which PHYSICAL CPU we''re talking about. So you need to get the actual load of each physical CPU, then sort them "per socket" and do any changes based on the highest (or lowest) load of the actual socket. I''m not sure if all the necessary information is available in Dom0 unless it''s also running on all available cores - however, neither am I sure if there''s much point in pursuing the case where all CPU''s are NOT in Dom0... Allowing those cores that are not Dom0-cores to have a driver-domain to control the speed would be sensible solution - interesting things happen, however, if someone puts core0 of a socket to Dom0, and Core1 of the same socket as DomU - you can''t really control that one even with a driver domain... -- Mats> > -- > Who do you trust? > The people with all the right answers? > Or the people with the right questions? > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel > > >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 26/10/06 1:35 pm, "Petersson, Mats" <Mats.Petersson@amd.com> wrote:> So you need to get the actual load of each physical CPU, then sort them > "per socket" and do any changes based on the highest (or lowest) load of > the actual socket. I''m not sure if all the necessary information is > available in Dom0 unless it''s also running on all available cores - > however, neither am I sure if there''s much point in pursuing the case > where all CPU''s are NOT in Dom0... Allowing those cores that are not > Dom0-cores to have a driver-domain to control the speed would be > sensible solution - interesting things happen, however, if someone puts > core0 of a socket to Dom0, and Core1 of the same socket as DomU - you > can''t really control that one even with a driver domain...We should state that as an assumption of the cpufreq domain (that it has a one-to-one correspondence between VCPUs and PCPUs). We liekly also need an interface to allow a domain to find out the ''topology'' of its VCPUs, so that the cpufreq drivers know which VCPUs share a socket, etc. They could also work this out from cpuid in most cases I expect, but it''s probably a better fit with the Linux code to provide a generic ''topology info'' interface and people experimenting with NUMA will appreciate it too. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel