Hi, Currently the XEN credit scheduler has its pitfalls in supporting weights of domains together with cpu pinning (see the threads http://lists.xensource.com/archives/html/xen-devel/2007-02/msg00006.html http://lists.xensource.com/archives/html/xen-devel/2006-10/msg00365.html http://lists.xensource.com/archives/html/xen-devel/2007-07/msg00303.html which include a rejected patch). We are facing this problem, too. We tried the above patch, but it didn''t solve our problem completely, so we decided to start a new solution. Our basic requirement is to limit a set of domains to a set of physical cpus while specifying the scheduling weight for each domain. The general (and in my opinion best) solution would be the introduction of a "pool" concept in XEN. Each physical cpu is dedicated to exactly one pool. At start of XEN this is pool0. A domain is member of a single pool (dom0 will always be member of pool0), there may be several domains in one pool. Scheduling does not cross pool boundaries, so the weight of a domain is only related to the weight of the other domains in the same pool. So it is possible to have an own scheduler for each pool. What changes would be needed? - The hypervisor must be pool-aware. It needs information about the pool configuration (cpu mask, scheduler) and the pool membership of a domain. The scheduler must restrict itself to its own pool only. - There must be an interface to set and query the pool configuration. - At domain creation the domain must be added to a pool. - libxc must be expanded to support the new interfaces. - xend and the xm command must support pools, defaulting to pool0 if no pool is specified The xm commands could look like this: xm pool-create pool1 ncpu=4 # create a pool with 4 cpus xm pool-create pool2 cpu=1,3,5 # create a pool with 3 dedicated cpus xm pool-list # show pools: pool cpus sched domains pool0 0,2,4 credit 0 pool1 6-9 credit 1,7 pool2 1,3,5 credit 2,3 xm pool-modify pool1 ncpu=3 # set new number of cpus xm pool-modify pool1 cpu=6,7,9 # modify cpu-pinning xm pool-destroy pool1 # destroy pool xm create vm5 pool=pool1 # start domain in pool1 There is much more potential in this approach: - add memory to a pool? Could be interesting for NUMA - recent discussions on xen-devel related to scheduling (credit scheduler for client virtualization) show some demand for further work regarding priority and/or grouping of domains - this might be an interesting approach for migration of multiple related domains (pool migration) - move (or migrate?) a domain to another pool - ... Any comments, suggestions, work already done, ...? Otherwise we will be starting our effort soon. Juergen -- Juergen Gross Principal Developer IP SW OS6 Telephone: +49 (0) 89 636 47950 Fujitsu Siemens Computers e-mail: juergen.gross@fujitsu-siemens.com Otto-Hahn-Ring 6 Internet: www.fujitsu-siemens.com D-81739 Muenchen Company details: www.fujitsu-siemens.com/imprint.html _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
I remember seeing some post before about domain group scheduler. Not sure about its progress now, and maybe you can check that thread to see anything useful? Thanks, Kevin>From:Juergen Gross >Sent: Wednesday, December 10, 2008 9:10 PM > >Hi, > >Currently the XEN credit scheduler has its pitfalls in >supporting weights of >domains together with cpu pinning (see the threads >http://lists.xensource.com/archives/html/xen-devel/2007-02/msg0 >0006.html >http://lists.xensource.com/archives/html/xen-devel/2006-10/msg0 >0365.html >http://lists.xensource.com/archives/html/xen-devel/2007-07/msg0 >0303.html >which include a rejected patch). > >We are facing this problem, too. We tried the above patch, but >it didn''t solve >our problem completely, so we decided to start a new solution. > >Our basic requirement is to limit a set of domains to a set of >physical cpus >while specifying the scheduling weight for each domain. The >general (and in my >opinion best) solution would be the introduction of a "pool" >concept in XEN. > >Each physical cpu is dedicated to exactly one pool. At start >of XEN this is >pool0. A domain is member of a single pool (dom0 will always >be member of >pool0), there may be several domains in one pool. Scheduling >does not cross >pool boundaries, so the weight of a domain is only related to >the weight of >the other domains in the same pool. So it is possible to have >an own scheduler >for each pool. > >What changes would be needed? >- The hypervisor must be pool-aware. It needs information >about the pool > configuration (cpu mask, scheduler) and the pool membership >of a domain. > The scheduler must restrict itself to its own pool only. >- There must be an interface to set and query the pool configuration. >- At domain creation the domain must be added to a pool. >- libxc must be expanded to support the new interfaces. >- xend and the xm command must support pools, defaulting to >pool0 if no pool > is specified > >The xm commands could look like this: >xm pool-create pool1 ncpu=4 # create a pool with 4 cpus >xm pool-create pool2 cpu=1,3,5 # create a pool with >3 dedicated cpus >xm pool-list # show pools: > pool cpus sched domains > pool0 0,2,4 credit 0 > pool1 6-9 credit 1,7 > pool2 1,3,5 credit 2,3 >xm pool-modify pool1 ncpu=3 # set new number of cpus >xm pool-modify pool1 cpu=6,7,9 # modify cpu-pinning >xm pool-destroy pool1 # destroy pool >xm create vm5 pool=pool1 # start domain in pool1 > >There is much more potential in this approach: >- add memory to a pool? Could be interesting for NUMA >- recent discussions on xen-devel related to scheduling >(credit scheduler for > client virtualization) show some demand for further work >regarding priority > and/or grouping of domains >- this might be an interesting approach for migration of >multiple related > domains (pool migration) >- move (or migrate?) a domain to another pool >- ... > >Any comments, suggestions, work already done, ...? >Otherwise we will be starting our effort soon. > >Juergen > >-- >Juergen Gross Principal Developer >IP SW OS6 Telephone: +49 (0) 89 636 47950 >Fujitsu Siemens Computers e-mail: >juergen.gross@fujitsu-siemens.com >Otto-Hahn-Ring 6 Internet: www.fujitsu-siemens.com >D-81739 Muenchen Company details: >www.fujitsu-siemens.com/imprint.html > > > > > >_______________________________________________ >Xen-devel mailing list >Xen-devel@lists.xensource.com >http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
That was grouping domains to directly share scheduling credits, rather than grouping to share physical resources. -- Keir On 10/12/2008 13:21, "Tian, Kevin" <kevin.tian@intel.com> wrote:> I remember seeing some post before about domain group scheduler. > Not sure about its progress now, and maybe you can check that > thread to see anything useful? > > Thanks, > Kevin > >> From:Juergen Gross >> Sent: Wednesday, December 10, 2008 9:10 PM >> >> Hi, >> >> Currently the XEN credit scheduler has its pitfalls in >> supporting weights of >> domains together with cpu pinning (see the threads >> http://lists.xensource.com/archives/html/xen-devel/2007-02/msg0 >> 0006.html >> http://lists.xensource.com/archives/html/xen-devel/2006-10/msg0 >> 0365.html >> http://lists.xensource.com/archives/html/xen-devel/2007-07/msg0 >> 0303.html >> which include a rejected patch). >> >> We are facing this problem, too. We tried the above patch, but >> it didn''t solve >> our problem completely, so we decided to start a new solution. >> >> Our basic requirement is to limit a set of domains to a set of >> physical cpus >> while specifying the scheduling weight for each domain. The >> general (and in my >> opinion best) solution would be the introduction of a "pool" >> concept in XEN. >> >> Each physical cpu is dedicated to exactly one pool. At start >> of XEN this is >> pool0. A domain is member of a single pool (dom0 will always >> be member of >> pool0), there may be several domains in one pool. Scheduling >> does not cross >> pool boundaries, so the weight of a domain is only related to >> the weight of >> the other domains in the same pool. So it is possible to have >> an own scheduler >> for each pool. >> >> What changes would be needed? >> - The hypervisor must be pool-aware. It needs information >> about the pool >> configuration (cpu mask, scheduler) and the pool membership >> of a domain. >> The scheduler must restrict itself to its own pool only. >> - There must be an interface to set and query the pool configuration. >> - At domain creation the domain must be added to a pool. >> - libxc must be expanded to support the new interfaces. >> - xend and the xm command must support pools, defaulting to >> pool0 if no pool >> is specified >> >> The xm commands could look like this: >> xm pool-create pool1 ncpu=4 # create a pool with 4 cpus >> xm pool-create pool2 cpu=1,3,5 # create a pool with >> 3 dedicated cpus >> xm pool-list # show pools: >> pool cpus sched domains >> pool0 0,2,4 credit 0 >> pool1 6-9 credit 1,7 >> pool2 1,3,5 credit 2,3 >> xm pool-modify pool1 ncpu=3 # set new number of cpus >> xm pool-modify pool1 cpu=6,7,9 # modify cpu-pinning >> xm pool-destroy pool1 # destroy pool >> xm create vm5 pool=pool1 # start domain in pool1 >> >> There is much more potential in this approach: >> - add memory to a pool? Could be interesting for NUMA >> - recent discussions on xen-devel related to scheduling >> (credit scheduler for >> client virtualization) show some demand for further work >> regarding priority >> and/or grouping of domains >> - this might be an interesting approach for migration of >> multiple related >> domains (pool migration) >> - move (or migrate?) a domain to another pool >> - ... >> >> Any comments, suggestions, work already done, ...? >> Otherwise we will be starting our effort soon. >> >> Juergen >> >> -- >> Juergen Gross Principal Developer >> IP SW OS6 Telephone: +49 (0) 89 636 47950 >> Fujitsu Siemens Computers e-mail: >> juergen.gross@fujitsu-siemens.com >> Otto-Hahn-Ring 6 Internet: www.fujitsu-siemens.com >> D-81739 Muenchen Company details: >> www.fujitsu-siemens.com/imprint.html >> >> >> >> >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel >> > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Tian, Kevin wrote:> I remember seeing some post before about domain group scheduler. > Not sure about its progress now, and maybe you can check that > thread to see anything useful?Thanks for that hint. As you are mentioning it, I remember, too. Group scheduling is a little bit different, but the patches contain very useful stuff for hypercall, tools etc. which will make our work easier. Juergen -- Juergen Gross Principal Developer IP SW OS6 Telephone: +49 (0) 89 636 47950 Fujitsu Siemens Computers e-mail: juergen.gross@fujitsu-siemens.com Otto-Hahn-Ring 6 Internet: www.fujitsu-siemens.com D-81739 Muenchen Company details: www.fujitsu-siemens.com/imprint.html _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Grouping domains for the purpose of sharing scheduler credits was the intent of the sched-groups project done by Mike Day at IBM: http://xen.markmail.org/search/?q=sched-groups&x=0&y=0#query:sched- groups%20from%3A%22Mike%20D.%20Day%22+page:1+mid:k36x7tqjy3zquqv7+state:results) . A related project called Domain Groups (domgrps for short) also exists: http://xen.markmail.org/search/?q=%22domain+group%22#query:%22domain %20group%22%20from%3AChris%20date%3A200702%20+page:1+mid:b525tcfzvfgv3gzo+state:results The primary difference is that domgrps is a general-purpose domain grouping framework with integration into both the VMM and userspace tools. It does group migration, moving domains b/w groups, etc.. To demonstrate the flexibility of the domgrps architecture, I merged domgrps and schedgrps: http://xen.markmail.org/search/?q=domgrps+schedgrps#query:domgrps schedgrps from%3A"Chris"+page:1+mid:rvetcmnmm7bkgexp+state:results It should be equally straightforward to adapt domgrps to accommodate this newly proposed pooling concept. I''d be happy to get involved if Keir suggests which grouping features are of interest. If we first need a discussion to define the potential grouping features that''s OK too, but I''ll delay that effort until I''m asked. Cheers -Chris On Dec 10, 2008, at 8:34 AM, Keir Fraser wrote:> That was grouping domains to directly share scheduling credits, > rather than > grouping to share physical resources. > > -- Keir > > On 10/12/2008 13:21, "Tian, Kevin" <kevin.tian@intel.com> wrote: > >> I remember seeing some post before about domain group scheduler. >> Not sure about its progress now, and maybe you can check that >> thread to see anything useful? >> >> Thanks, >> Kevin >> >>> From:Juergen Gross >>> Sent: Wednesday, December 10, 2008 9:10 PM >>> >>> Hi, >>> >>> Currently the XEN credit scheduler has its pitfalls in >>> supporting weights of >>> domains together with cpu pinning (see the threads >>> http://lists.xensource.com/archives/html/xen-devel/2007-02/msg0 >>> 0006.html >>> http://lists.xensource.com/archives/html/xen-devel/2006-10/msg0 >>> 0365.html >>> http://lists.xensource.com/archives/html/xen-devel/2007-07/msg0 >>> 0303.html >>> which include a rejected patch). >>> >>> We are facing this problem, too. We tried the above patch, but >>> it didn''t solve >>> our problem completely, so we decided to start a new solution. >>> >>> Our basic requirement is to limit a set of domains to a set of >>> physical cpus >>> while specifying the scheduling weight for each domain. The >>> general (and in my >>> opinion best) solution would be the introduction of a "pool" >>> concept in XEN. >>> >>> Each physical cpu is dedicated to exactly one pool. At start >>> of XEN this is >>> pool0. A domain is member of a single pool (dom0 will always >>> be member of >>> pool0), there may be several domains in one pool. Scheduling >>> does not cross >>> pool boundaries, so the weight of a domain is only related to >>> the weight of >>> the other domains in the same pool. So it is possible to have >>> an own scheduler >>> for each pool. >>> >>> What changes would be needed? >>> - The hypervisor must be pool-aware. It needs information >>> about the pool >>> configuration (cpu mask, scheduler) and the pool membership >>> of a domain. >>> The scheduler must restrict itself to its own pool only. >>> - There must be an interface to set and query the pool >>> configuration. >>> - At domain creation the domain must be added to a pool. >>> - libxc must be expanded to support the new interfaces. >>> - xend and the xm command must support pools, defaulting to >>> pool0 if no pool >>> is specified >>> >>> The xm commands could look like this: >>> xm pool-create pool1 ncpu=4 # create a pool with 4 cpus >>> xm pool-create pool2 cpu=1,3,5 # create a pool with >>> 3 dedicated cpus >>> xm pool-list # show pools: >>> pool cpus sched domains >>> pool0 0,2,4 credit 0 >>> pool1 6-9 credit 1,7 >>> pool2 1,3,5 credit 2,3 >>> xm pool-modify pool1 ncpu=3 # set new number of cpus >>> xm pool-modify pool1 cpu=6,7,9 # modify cpu-pinning >>> xm pool-destroy pool1 # destroy pool >>> xm create vm5 pool=pool1 # start domain in pool1 >>> >>> There is much more potential in this approach: >>> - add memory to a pool? Could be interesting for NUMA >>> - recent discussions on xen-devel related to scheduling >>> (credit scheduler for >>> client virtualization) show some demand for further work >>> regarding priority >>> and/or grouping of domains >>> - this might be an interesting approach for migration of >>> multiple related >>> domains (pool migration) >>> - move (or migrate?) a domain to another pool >>> - ... >>> >>> Any comments, suggestions, work already done, ...? >>> Otherwise we will be starting our effort soon. >>> >>> Juergen >>> >>> -- >>> Juergen Gross Principal Developer >>> IP SW OS6 Telephone: +49 (0) 89 636 47950 >>> Fujitsu Siemens Computers e-mail: >>> juergen.gross@fujitsu-siemens.com >>> Otto-Hahn-Ring 6 Internet: www.fujitsu-siemens.com >>> D-81739 Muenchen Company details: >>> www.fujitsu-siemens.com/imprint.html >>> >>> >>> >>> >>> >>> _______________________________________________ >>> Xen-devel mailing list >>> Xen-devel@lists.xensource.com >>> http://lists.xensource.com/xen-devel >>> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Chris wrote:> Grouping domains for the purpose of sharing scheduler credits was the > intent of the sched-groups project done by Mike Day at IBM: > > http://xen.markmail.org/search/?q=sched-groups&x=0&y=0#query > <http://xen.markmail.org/search/?q=sched-groups&x=0&y=0#query>:sched-groups%20from%3A%22Mike%20D.%20Day%22+page:1+mid:k36x7tqjy3zquqv7+state:results). > > > A related project called Domain Groups (domgrps for short) also exists: > > http://xen.markmail.org/search/?q=%22domain+group%22#query:%22domain%20group%22%20from%3AChris%20date%3A200702%20+page:1+mid:b525tcfzvfgv3gzo+state:results > > The primary difference is that domgrps is a general-purpose domain > grouping framework with integration into both the VMM and userspace > tools. It does group migration, moving domains b/w groups, etc.. > > To demonstrate the flexibility of the domgrps architecture, I merged > domgrps and schedgrps: > > http://xen.markmail.org/search/?q=domgrps+schedgrps#query:domgrps > schedgrps from%3A"Chris"+page:1+mid:rvetcmnmm7bkgexp+state:results > > It should be equally straightforward to adapt domgrps to accommodate > this newly proposed pooling concept. I''d be happy to get involved if > Keir suggests which grouping features are of interest. If we first need > a discussion to define the potential grouping features that''s OK too, > but I''ll delay that effort until I''m asked.Chris, thanks for your support. I think Keir''s main interest is adding only features that are really needed. So let me explain WHY we want to implement a pool concept: We have a mainframe operating system called BS2000 originally designed for /390 machines. As our customers often require small machines (software prices are based on machine power) we decided some years ago to port BS2000 to other architectures as /390 machines are rather expensive. We did ports to MIPS (we had MIPS UNIX machines in these times), to SPARC (after our UNIX strategy switched to SPARC/Solaris), and now to x86_64. We always had a virtual machine concept in BS2000 called VM2000 which is quite important for our customers. On x86_64 we decided to use XEN as base for our virtual machines to be able to share the hardware with other systems as well. The software prices for BS2000 will be still related to the machine power. But often customers need only a small portion of the complete x86 machine power for BS2000, so we added a license scheme to limit the power available to BS2000 by pinning the domains with BS2000 to a subset of cpus. BTW: on /390 and SPARC we already support VM-pools, so adding pools to XEN would make this feature available to our customers who are still using it today on other platforms. Juergen -- Juergen Gross Principal Developer IP SW OS6 Telephone: +49 (0) 89 636 47950 Fujitsu Siemens Computers e-mail: juergen.gross@fujitsu-siemens.com Otto-Hahn-Ring 6 Internet: www.fujitsu-siemens.com D-81739 Muenchen Company details: www.fujitsu-siemens.com/imprint.html _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Thu, Dec 11, 2008 at 6:23 AM, Juergen Gross <juergen.gross@fujitsu-siemens.com> wrote:>> The software prices for BS2000 will be still related to the machine power. > But often customers need only a small portion of the complete x86 machine > power for BS2000, so we added a license scheme to limit the power available > to BS2000 by pinning the domains with BS2000 to a subset of cpus.OK, so... you sell software, and the cost of it depends on how powerful a machine you run it on. But most people don''t need even one full core''s worth of power. You want to be able to charge people who need a full core of processing power one price, and charge people who need only say, 20%, less? So to artificially limit the power available to a given instance, you pin all instances to some subset of cpus? I presume then, that there are multiple instances, and people pay for how many cores they can use total...? And that your customers generally have other things running on the server as well. It seems like what you really want is to cap the number of credits the VMs can get, and then take them offline when their credits go negative (instead of competing for resources in a "best-effort" fashion). :-) However, it does seem like being able to partition up a Xen server into "pools" of cpu resources, each with its own scheduler, that don''t compete with each other, might be generally useful. It should be relatively straightforward to slide in under the current scheduler architecture, without having to change much in the schedulers themselves. That''s how I''d prefer it done, if possible: a clean layer underneath a scheduler. -George> > BTW: on /390 and SPARC we already support VM-pools, so adding pools to XEN > would make this feature available to our customers who are still using it > today on other platforms. > > > Juergen > > -- > Juergen Gross Principal Developer > IP SW OS6 Telephone: +49 (0) 89 636 47950 > Fujitsu Siemens Computers e-mail: juergen.gross@fujitsu-siemens.com > Otto-Hahn-Ring 6 Internet: www.fujitsu-siemens.com > D-81739 Muenchen Company details: www.fujitsu-siemens.com/imprint.html > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Thu, Dec 11, 2008 at 3:13 PM, George Dunlap <dunlapg@umich.edu> wrote:> It seems like what you really want is to cap the number of credits the > VMs can get, and then take them offline when their credits go negative > (instead of competing for resources in a "best-effort" fashion). :-)Actually, it turns out this functionality is already present. xm sched-credit -d [domid] -c 30 will "cap" the domain to 30% of one cpu, even if there are no other cpus running. You should be able to utilize that directly, no development needed. -George _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
George Dunlap wrote:> On Thu, Dec 11, 2008 at 6:23 AM, Juergen Gross > <juergen.gross@fujitsu-siemens.com> wrote: >>> The software prices for BS2000 will be still related to the machine power. >> But often customers need only a small portion of the complete x86 machine >> power for BS2000, so we added a license scheme to limit the power available >> to BS2000 by pinning the domains with BS2000 to a subset of cpus. > > OK, so... you sell software, and the cost of it depends on how > powerful a machine you run it on. But most people don''t need even one > full core''s worth of power. You want to be able to charge people who > need a full core of processing power one price, and charge people who > need only say, 20%, less? So to artificially limit the power > available to a given instance, you pin all instances to some subset of > cpus?More or less. But the processors of our servers are 6-core. A customer might want to pay for only some cores of power.> > I presume then, that there are multiple instances, and people pay for > how many cores they can use total...? And that your customers > generally have other things running on the server as well.Correct.> > It seems like what you really want is to cap the number of credits the > VMs can get, and then take them offline when their credits go negative > (instead of competing for resources in a "best-effort" fashion). :-)In theory, yes. But if a customer has a license for the power of 3 cores for BS2000, he should be able to run for example 4 BS2000 domains which must not consume more power in sum, but if 3 domains are more or less idle, the remaining domain could take all 3 cores. I don''t think this is an easy job with caps.> > However, it does seem like being able to partition up a Xen server > into "pools" of cpu resources, each with its own scheduler, that don''t > compete with each other, might be generally useful. It should be > relatively straightforward to slide in under the current scheduler > architecture, without having to change much in the schedulers > themselves. That''s how I''d prefer it done, if possible: a clean layer > underneath a scheduler.That is the direction I would go. Juergen -- Juergen Gross Principal Developer IP SW OS6 Telephone: +49 (0) 89 636 47950 Fujitsu Siemens Computers e-mail: juergen.gross@fujitsu-siemens.com Otto-Hahn-Ring 6 Internet: www.fujitsu-siemens.com D-81739 Muenchen Company details: www.fujitsu-siemens.com/imprint.html _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
>> However, it does seem like being able to partition up a Xen server >> into "pools" of cpu resources, each with its own scheduler, that >> don''t >> compete with each other, might be generally useful. It should be >> relatively straightforward to slide in under the current scheduler >> architecture, without having to change much in the schedulers >> themselves. That''s how I''d prefer it done, if possible: a clean >> layer >> underneath a scheduler. > > That is the direction I would goAgreed; I do not mean to suggest otherwise. My point is that if this project and others all implement their own management tools and inevitably conflicting means of storing information about of sets of domains, then that''s not a path to a happy place. I also recognize that I''m trying to head off a problem before it exists, which may not be the way to go. If you decide that your needs are best served by developing a new set of tools, please keep the discussion alive on the list. It''s a potentially useful capability and I''d like to see how it progresses. Regards -Chris _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Chris wrote:> Agreed; I do not mean to suggest otherwise. > > My point is that if this project and others all implement their own > management tools and inevitably conflicting means of storing information > about of sets of domains, then that''s not a path to a happy place.You are absolutely right!> If you decide that your needs are best served by developing a new set of > tools, please keep the discussion alive on the list. It''s a potentially > useful capability and I''d like to see how it progresses.Okay, here is our proposal for the user interface. We plan to do the implementation work in the next month(s). Support of cpu-pools The physical cpus of a server are grouped in cpu-pools. Each physical CPU is assigned at most to one cpu-pool. Domains are each restricted to a single cpu-pool. Scheduling does not cross cpu-pool boundaries, so each cpu-pool has an own scheduler. The cpu-pool of the domain Dom0 will be pool0, at boot-time we have to create this cpu-pool implicitly. The creation of a cpu-pool is similar to the creation of a domain. In a first step the attributes of a cpu-pool have to be defined, in the second step the cpu-pool will be activated (started) by assigning the resources (CPUs) and initializing the scheduler. An activated cpu-pool owns at least one physical CPU. A physical CPU may be removed from an activated cpu-pool. Further CPUs may be assigned to the cpu-pool. The CPU may be assigned to a cpu-pool only if that CPU is free (not assigned to any other activated cpu-pool). The free CPUs are internally managed. The poolname will be added as an additional parameter to the configfile of a domain. If no poolname is specified the poolname is defaults to pool0. At domain start the domain is added to its cpu-pool, which must be activated. A started domain can be moved to another activated cpu-pool. A cpu-pool may be deactivated if there is no active domain assigned to it. A cpu-pool is deactivated by removing the resources (CPUs). A cpu-pool may be deleted if it is deactivated. The administration of the cpu-pools is done by xm subcommands. new xm subcommands: xm pool-new poolconfigfile [name=value].. define a cpu-pool. The pool-new sub command requires a config file and can optionally take a series of name value pairs that add to or override variables defined in the config file. See poolconfigfile description. poolconfigfile can either be an absolute path to a file, or a relative path to a file located in /etc/xen/pool. xm pool-start <poolname> Activate a Xend managed cpu-pool. (The cpu-pool must be defined by xm pool-new (or by API). The cpu-pool will be activated only if the requested CPUs are available.) xm pool-create poolconfigfile [name=value].. define a new cpu-pool and activate it. xm pool-list [options] [poolname,...] List information about all/some cpu-pools. -l|--long Output all cpu-pool details in SXP output format for deactivated cpu-pools without option --long cpu-pool cpus sched domains <poolname> <# CPUs> <sched> output format for activated cpu-pools without option --long cpu-pool cpus sched domains <poolname> <list of CPUs> <sched> <list of active domains> output format with option --long (pool (name <poolname>) (uuid <uuid>) (state <activated|deactivated>) (cpun <number of CPUs>) (cpus <list of CPUs>) (sched <sched>) (domains <list of started domains running in the cpu-pool>) ) xm pool-destroy <poolname> deactivate a cpu-pool. (Unassign the CPUs of the cpu-pool. Delete the cpu-pool if it was created by xm pool-create. A cpu-pool may be deactivated only if there are no active domains added to it.) xm pool-delete <poolname> Delete the cpu-pool. (deletion is only possible if the cpu-pool is deactivated.) xm pool-cpu-add <poolname> <cpu> Add a free cpu to the activated cpu-pool. xm pool-cpu-remove <poolname> <cpu> Remove a cpu from the activated cpu-pool. (subsequent the removed CPU may be added to another cpu-pool. The last CPU of the cpu-pool cannot be removed explicitly.) xm pool-migrate <domain> <poolname> move the domain to the specified cpu-pool. extended subcommands xm list ... [--pool=<poolname>] select only domains added to the cpu-pool. xm list --long|-l additional info: (cpu_pool <poolname>) xm info the line xen_scheduler : <sched> shows the scheduler of pool0. additional info: free_cpus : <n> (number of free CPUs) xen pool configfile pool configuration files live in /etc/xen/pool by default. If you store config files anywhere else the full path to the config file must be specified in the xm pool-create command. supported options: name A unique name for the cpu-pool. Attempting to create two cpu-pools with the same name will cause an error. sched Name of scheduler (Default: credit) cpus List of CPUs to be activated or Number of CPUs to be activated (Default: 1) other_config ["<param> = <value>" ...] xen domain configfile additional option: pool Name of cpu-pool the domain is added to. xm create or xm start is rejected if the cpu-pool is not activated. For pinning only CPUs of the cpu-pool are taken into account. Autostart of cpu-pools and domains Before starting a domain automatically its cpu-pool has to be started. By default all CPUs are assigned to the cpu-pool pool0. As for automatically starting a cpu-pool free CPUs are needed, a new boot parameter in /boot/grub/menu.lst is required to create pool0 with less CPUs. To specify a cpu-pool is to be started at boot-time, its configfile (or a link to it) has to be placed in /etc/xen/pool/auto/. The xendomains script for automatically starting domains has to be extended. Before starting the domains the autostart of the cpu-pools is done. Juergen -- Juergen Gross Principal Developer IP SW OS6 Telephone: +49 (0) 89 636 47950 Fujitsu Siemens Computers e-mail: juergen.gross@fujitsu-siemens.com Otto-Hahn-Ring 6 Internet: www.fujitsu-siemens.com D-81739 Muenchen Company details: www.fujitsu-siemens.com/imprint.html _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel