I''ve put together two patches. The first introduces a new dom0_op, set_max_vcpus, which with an associated variable and a check in the VCPUOP handler fixes [1]bug 288. Also included is a new VCPUOP, VCPUOP_create, which handles all of the vcpu creation tasks and leaves initialization and unpausing to VCPUOP_initialize. The separation allows for build-time allocation of vcpus which becomes more important when trying to allocate memory in a NUMA-aware manner (i.e. knowing which physical cpus are being used in a domain before we allocate the memory). The second patch adds a new domain allocmap cpumap_t which is a bitmap of physical cpus from which vcpus are to be allocated. As vcpus are created, the selection of which physical cpu is balanced across the set of physical cpus within the map. The patch lets us control vcpu allocation at a high level (start all VCPUS on hyperthreads, NODE1, CPU2). 1. http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=288 -- Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx (512) 838-9253 T/L: 678-9253 ryanh@us.ibm.com _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ryan Harper
2005-Oct-10 15:28 UTC
Re: [Xen-devel] [PATCH] 0/2 VCPU creation and allocation
* Keir Fraser <Keir.Fraser@cl.cam.ac.uk> [2005-10-10 10:26]:> > On 10 Oct 2005, at 16:16, Ryan Harper wrote: > > >Also included is a new VCPUOP, > >VCPUOP_create, which handles all of the vcpu creation tasks and leaves > >initialization and unpausing to VCPUOP_initialize. > > I think two new Xen operations is one too many. > > Either we should take set_max_vcpus, and have that implicitly do the > work of VCPUOP_create, or we should take VCPUOP_create (callable only > by domain0) and have that implicitly increase max_vcpus for the subject > domain.OK. I''ll re-work VCPUOP_create to be callable only by dom0 and auto-increment max_vcpus. -- Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx (512) 838-9253 T/L: 678-9253 ryanh@us.ibm.com _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2005-Oct-10 15:29 UTC
Re: [Xen-devel] [PATCH] 0/2 VCPU creation and allocation
On 10 Oct 2005, at 16:16, Ryan Harper wrote:> Also included is a new VCPUOP, > VCPUOP_create, which handles all of the vcpu creation tasks and leaves > initialization and unpausing to VCPUOP_initialize.I think two new Xen operations is one too many. Either we should take set_max_vcpus, and have that implicitly do the work of VCPUOP_create, or we should take VCPUOP_create (callable only by domain0) and have that implicitly increase max_vcpus for the subject domain. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> > Also included is a new VCPUOP, > > VCPUOP_create, which handles all of the vcpu creation tasks > and leaves > > initialization and unpausing to VCPUOP_initialize. > > I think two new Xen operations is one too many. > > Either we should take set_max_vcpus, and have that implicitly > do the work of VCPUOP_create, or we should take VCPUOP_create > (callable only by domain0) and have that implicitly increase > max_vcpus for the subject domain.While we''re at it, let''s clean up the xm commands too. I tihnk its pointless letting the user specify the vcpu they want to enable/disable. Let''s change the command names to vcpu-add / vcpu-remove and have them work on the lowest numbered free vcpu in the case of vcpu-add and the highest currently active in the case of vcpu-remove. Ian _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2005-Oct-10 16:01 UTC
Re: [Xen-devel] [PATCH] 0/2 VCPU creation and allocation
On 10 Oct 2005, at 16:28, Ryan Harper wrote:>> Either we should take set_max_vcpus, and have that implicitly do the >> work of VCPUOP_create, or we should take VCPUOP_create (callable only >> by domain0) and have that implicitly increase max_vcpus for the >> subject >> domain. > > OK. I''ll re-work VCPUOP_create to be callable only by dom0 and > auto-increment max_vcpus.Do you even need a max_vcpus variable? Surely the appropriate check is implicit in VCPUOP_initialise detecting whether or not the relevant VCPU has been created? I like small code and simple interfaces. :-) -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ryan Harper
2005-Oct-10 16:05 UTC
Re: [Xen-devel] [PATCH] 0/2 VCPU creation and allocation
* Keir Fraser <Keir.Fraser@cl.cam.ac.uk> [2005-10-10 10:56]:> > On 10 Oct 2005, at 16:28, Ryan Harper wrote: > > >>Either we should take set_max_vcpus, and have that implicitly do the > >>work of VCPUOP_create, or we should take VCPUOP_create (callable only > >>by domain0) and have that implicitly increase max_vcpus for the > >>subject > >>domain. > > > >OK. I''ll re-work VCPUOP_create to be callable only by dom0 and > >auto-increment max_vcpus. > > Do you even need a max_vcpus variable? Surely the appropriate check is > implicit in VCPUOP_initialise detecting whether or not the relevant > VCPU has been created?I was going to ensure ordered VCPU creation. Without something like vcpuid < max_vcpus+1, and increment on successful creation, one can create vcpus in any order, 1,5,7, 10. I don''t think it *should* matter but I''ve not looked elsewhere through the code to see if there are any other areas assuming all struct vcpu* being valid between 0 and n in the d->vcpus[] array. -- Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx (512) 838-9253 T/L: 678-9253 ryanh@us.ibm.com _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2005-Oct-10 16:17 UTC
Re: [Xen-devel] [PATCH] 0/2 VCPU creation and allocation
On 10 Oct 2005, at 17:05, Ryan Harper wrote:>> Do you even need a max_vcpus variable? Surely the appropriate check is >> implicit in VCPUOP_initialise detecting whether or not the relevant >> VCPU has been created? > > I was going to ensure ordered VCPU creation. Without something like > vcpuid < max_vcpus+1, and increment on successful creation, one can > create vcpus in any order, 1,5,7, 10. I don''t think it *should* matter > but I''ve not looked elsewhere through the code to see if there are any > other areas assuming all struct vcpu* being valid between 0 and n in > the d->vcpus[] array.Then the vcpu parameter to VCPUOP_create is redundant -- there''s only one value you will be prepared to accept! If we don''t want the flexibility of a sparse vcpu map (and I think we don''t) then perhaps we are better off without VCPUOP_create (which is maybe even a bit neater, leaving vcpu_op as a completely unpriv local hypercall) and stick with the set_max_vcpus dom0_op? And have that implicitly create the vcpu struct for vcpus 0...n-1? -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ryan Harper
2005-Oct-10 16:23 UTC
Re: [Xen-devel] [PATCH] 0/2 VCPU creation and allocation
* Keir Fraser <Keir.Fraser@cl.cam.ac.uk> [2005-10-10 11:13]:> > On 10 Oct 2005, at 17:05, Ryan Harper wrote: > > >>Do you even need a max_vcpus variable? Surely the appropriate check is > >>implicit in VCPUOP_initialise detecting whether or not the relevant > >>VCPU has been created? > > > >I was going to ensure ordered VCPU creation. Without something like > >vcpuid < max_vcpus+1, and increment on successful creation, one can > >create vcpus in any order, 1,5,7, 10. I don''t think it *should* matter > >but I''ve not looked elsewhere through the code to see if there are any > >other areas assuming all struct vcpu* being valid between 0 and n in > >the d->vcpus[] array. > > Then the vcpu parameter to VCPUOP_create is redundant -- there''s only > one value you will be prepared to accept! If we don''t want the > flexibility of a sparse vcpu map (and I think we don''t) then perhaps we > are better off without VCPUOP_create (which is maybe even a bit neater, > leaving vcpu_op as a completely unpriv local hypercall) and stick with > the set_max_vcpus dom0_op? And have that implicitly create the vcpu > struct for vcpus 0...n-1?OK, that makes sense. I''ll turn VCPUOP_create into set_max_vcpus(max) which will create vcpus 1-(max-1). Any preference on the hypercall name? Does set_max_vcpus() still make sense if it is also creating vcpus? How about DOM0_CREATEVCPUS and do_createvcpus(struct domain* d, unsigned int max_vcpus). -- Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx (512) 838-9253 T/L: 678-9253 ryanh@us.ibm.com _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser
2005-Oct-10 16:39 UTC
Re: [Xen-devel] [PATCH] 0/2 VCPU creation and allocation
On 10 Oct 2005, at 17:23, Ryan Harper wrote:> OK, that makes sense. I''ll turn VCPUOP_create into set_max_vcpus(max) > which will create vcpus 1-(max-1). Any preference on the hypercall > name? Does set_max_vcpus() still make sense if it is also creating > vcpus? > > How about DOM0_CREATEVCPUS and > do_createvcpus(struct domain* d, unsigned int max_vcpus).I''m not fussed. I guess I have a slight preference for the latter (createvcpus). -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ryan Harper
2005-Oct-11 15:13 UTC
[Xen-devel] [PATCH] 2nd try: 0/2 VCPU creation and allocation
* Keir Fraser <Keir.Fraser@cl.cam.ac.uk> [2005-10-10 11:34]:> > On 10 Oct 2005, at 17:23, Ryan Harper wrote: > > >OK, that makes sense. I''ll turn VCPUOP_create into set_max_vcpus(max) > >which will create vcpus 1-(max-1). Any preference on the hypercall > >name? Does set_max_vcpus() still make sense if it is also creating > >vcpus? > > > >How about DOM0_CREATEVCPUS and > > do_createvcpus(struct domain* d, unsigned int max_vcpus). > > I''m not fussed. I guess I have a slight preference for the latter > (createvcpus).The first patch creates a new dom0_op, DOM0_VCPUSINCREASE, which will create VCPU 1 to (limit-1) in the target domain. The max_vcpus value is gone as is the VCPUOP_create. The second patch is the same as before. -- Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx (512) 838-9253 T/L: 678-9253 ryanh@us.ibm.com _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ryan Harper
2005-Oct-11 15:15 UTC
[Xen-devel] [PATCH] 2nd try: 1/2 VCPU creation and allocation
New dom0_op DOM0_VCPUSINCREASE.
--
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253 T/L: 678-9253
ryanh@us.ibm.com
diffstat output:
tools/libxc/xc_domain.c | 9 +++++++++
tools/libxc/xenctrl.h | 11 +++++++++++
tools/python/xen/lowlevel/xc/xc.c | 28 ++++++++++++++++++++++++++++
tools/python/xen/xend/XendDomainInfo.py | 4 ++++
xen/arch/x86/domain_build.c | 6 ++++++
xen/common/dom0_ops.c | 27 +++++++++++++++++++++++++++
xen/common/domain.c | 25 ++++++++++---------------
xen/include/public/dom0_ops.h | 8 ++++++++
xen/include/xen/domain.h | 2 ++
9 files changed, 105 insertions(+), 15 deletions(-)
Signed-off-by: Ryan Harper <ryanh@us.ibm.com>
---
diff -r 015f8ae81276 tools/libxc/xc_domain.c
--- a/tools/libxc/xc_domain.c Mon Oct 10 14:38:01 2005
+++ b/tools/libxc/xc_domain.c Mon Oct 10 17:26:54 2005
@@ -329,6 +329,15 @@
return err;
}
+int xc_domain_vcpus_increase(int xc_handle, u32 domid, unsigned int limit)
+{
+ dom0_op_t op;
+ op.cmd = DOM0_VCPUSINCREASE;
+ op.u.vcpusincrease.domain = (domid_t)domid;
+ op.u.vcpusincrease.limit = (u8)limit;
+ return do_dom0_op(xc_handle, &op);
+}
+
/*
* Local variables:
* mode: C
diff -r 015f8ae81276 tools/libxc/xenctrl.h
--- a/tools/libxc/xenctrl.h Mon Oct 10 14:38:01 2005
+++ b/tools/libxc/xenctrl.h Mon Oct 10 17:26:54 2005
@@ -152,6 +152,17 @@
u32 ssidref,
u32 *pdomid);
+/*
+ * This function increase the number vcpus in a domain to limit.
+ *
+ * @parm xc_handle a handle to an open hypervisor interface.
+ * @parm domid the domain id in which vcpus are to be created.
+ * @parm limit the total number of vcpus to run in domain.
+ * @return 0 on success, -1 on failure.
+ */
+int xc_domain_vcpus_increase(int xc_handle,
+ u32 domid,
+ unsigned int limit);
int xc_domain_dumpcore(int xc_handle,
u32 domid,
diff -r 015f8ae81276 tools/python/xen/lowlevel/xc/xc.c
--- a/tools/python/xen/lowlevel/xc/xc.c Mon Oct 10 14:38:01 2005
+++ b/tools/python/xen/lowlevel/xc/xc.c Mon Oct 10 17:26:54 2005
@@ -91,6 +91,26 @@
return PyErr_SetFromErrno(xc_error);
return PyInt_FromLong(dom);
+}
+
+static PyObject *pyxc_domain_vcpus_increase(PyObject *self,
+ PyObject *args,
+ PyObject *kwds)
+{
+ XcObject *xc = (XcObject *)self;
+
+ u32 dom, limit;
+
+ static char *kwd_list[] = { "dom", "limit", NULL };
+
+ if ( !PyArg_ParseTupleAndKeywords(args, kwds, "ii", kwd_list,
&dom, &limit) )
+ return NULL;
+
+ if ( xc_domain_vcpus_increase(xc->xc_handle, dom, limit) != 0 )
+ return PyErr_SetFromErrno(xc_error);
+
+ Py_INCREF(zero);
+ return zero;
}
static PyObject *pyxc_domain_pause(PyObject *self,
@@ -783,6 +803,14 @@
" dom [int, 0]: Domain identifier to use (allocated if
zero).\n"
"Returns: [int] new domain identifier; -1 on error.\n" },
+ { "domain_vcpus_increase",
+ (PyCFunction)pyxc_domain_vcpus_increase,
+ METH_VARARGS | METH_KEYWORDS, "\n"
+ "Increase the number of VCPUs in domain.\n"
+ " dom [int, 0]: Domain identifier to use.\n"
+ " limit [int, 0]: New total number of VCPUs in
domain.\n"
+ "Returns: [int] 0 on success; -1 on error.\n" },
+
{ "domain_dumpcore",
(PyCFunction)pyxc_domain_dumpcore,
METH_VARARGS | METH_KEYWORDS, "\n"
diff -r 015f8ae81276 tools/python/xen/xend/XendDomainInfo.py
--- a/tools/python/xen/xend/XendDomainInfo.py Mon Oct 10 14:38:01 2005
+++ b/tools/python/xen/xend/XendDomainInfo.py Mon Oct 10 17:26:54 2005
@@ -1018,6 +1018,10 @@
self.image.handleBootloading()
xc.domain_setcpuweight(self.domid,
self.info[''cpu_weight''])
+
+ # increase the total number of vcpus in domain
+ xc.domain_vcpus_increase(self.domid,
int(self.info[''vcpus'']));
+
# XXX Merge with configure_maxmem?
m =
self.image.getDomainMemory(self.info[''memory_KiB''])
xc.domain_setmaxmem(self.domid, m)
diff -r 015f8ae81276 xen/arch/x86/domain_build.c
--- a/xen/arch/x86/domain_build.c Mon Oct 10 14:38:01 2005
+++ b/xen/arch/x86/domain_build.c Mon Oct 10 17:26:54 2005
@@ -14,6 +14,7 @@
#include <xen/event.h>
#include <xen/elf.h>
#include <xen/kernel.h>
+#include <xen/domain.h>
#include <asm/regs.h>
#include <asm/system.h>
#include <asm/io.h>
@@ -559,6 +560,11 @@
d->shared_info->vcpu_data[i].evtchn_upcall_mask = 1;
d->shared_info->n_vcpu = num_online_cpus();
+ /* create extra vcpus */
+ if ( (d->shared_info->n_vcpu > 1) )
+ for ( i = 1; i < d->shared_info->n_vcpu; i++ )
+ do_createvcpu(d, i);
+
/* Set up monitor table */
update_pagetables(v);
diff -r 015f8ae81276 xen/common/dom0_ops.c
--- a/xen/common/dom0_ops.c Mon Oct 10 14:38:01 2005
+++ b/xen/common/dom0_ops.c Mon Oct 10 17:26:54 2005
@@ -227,6 +227,33 @@
}
break;
+ case DOM0_VCPUSINCREASE:
+ {
+ struct domain *d;
+ unsigned int i, limit = op->u.vcpusincrease.limit;
+
+ if ( limit >= MAX_VIRT_CPUS )
+ ret = -EINVAL;
+
+ d = find_domain_by_id(op->u.vcpusincrease.domain);
+
+ ret = -ESRCH;
+ if ( d != NULL )
+ {
+ ret = -EINVAL;
+ LOCK_BIGLOCK(d);
+ /* NB: VCPU0 is allocated on domain creation */
+ for ( i=1; i<limit; i++ ) {
+ if ( d->vcpu[i] == NULL )
+ if ( (ret = do_createvcpu(d, i)) != 0 )
+ break;
+ }
+ UNLOCK_BIGLOCK(d);
+ put_domain(d);
+ }
+ }
+ break;
+
case DOM0_DESTROYDOMAIN:
{
struct domain *d = find_domain_by_id(op->u.destroydomain.domain);
diff -r 015f8ae81276 xen/common/domain.c
--- a/xen/common/domain.c Mon Oct 10 14:38:01 2005
+++ b/xen/common/domain.c Mon Oct 10 17:26:54 2005
@@ -368,10 +368,9 @@
return rc;
}
-int boot_vcpu(struct domain *d, int vcpuid, struct vcpu_guest_context *ctxt)
-{
- struct vcpu *v;
- int rc;
+int do_createvcpu(struct domain *d, int vcpuid)
+{
+ struct vcpu *v;
ASSERT(d->vcpu[vcpuid] == NULL);
@@ -387,20 +386,11 @@
arch_do_boot_vcpu(v);
- if ( (rc = arch_set_info_guest(v, ctxt)) != 0 )
- goto out;
-
sched_add_domain(v);
set_bit(_VCPUF_down, &v->vcpu_flags);
- clear_bit(_VCPUF_ctrl_pause, &v->vcpu_flags);
return 0;
-
- out:
- arch_free_vcpu_struct(d->vcpu[vcpuid]);
- d->vcpu[vcpuid] = NULL;
- return rc;
}
long do_vcpu_op(int cmd, int vcpuid, void *arg)
@@ -413,7 +403,7 @@
if ( (vcpuid < 0) || (vcpuid >= MAX_VIRT_CPUS) )
return -EINVAL;
- if ( ((v = d->vcpu[vcpuid]) == NULL) && (cmd !=
VCPUOP_initialise) )
+ if ( ((v = d->vcpu[vcpuid]) == NULL) )
return -ENOENT;
switch ( cmd )
@@ -433,7 +423,12 @@
}
LOCK_BIGLOCK(d);
- rc = (d->vcpu[vcpuid] == NULL) ? boot_vcpu(d, vcpuid, ctxt) :
-EEXIST;
+ clear_bit(_VCPUF_ctrl_pause, &v->vcpu_flags);
+ if ( (rc = arch_set_info_guest(v, ctxt)) != 0 ) {
+ sched_rem_domain(v);
+ arch_free_vcpu_struct(d->vcpu[vcpuid]);
+ d->vcpu[vcpuid] = NULL;
+ }
UNLOCK_BIGLOCK(d);
xfree(ctxt);
diff -r 015f8ae81276 xen/include/public/dom0_ops.h
--- a/xen/include/public/dom0_ops.h Mon Oct 10 14:38:01 2005
+++ b/xen/include/public/dom0_ops.h Mon Oct 10 17:26:54 2005
@@ -386,6 +386,13 @@
int is_ram;
} *memory_map;
} dom0_physical_memory_map_t;
+
+#define DOM0_VCPUSINCREASE 41
+typedef struct {
+ domid_t domain; /* domain to be affected */
+ u8 limit; /* total number of vcpus to run */
+} dom0_vcpusincrease_t;
+
typedef struct {
u32 cmd;
@@ -422,6 +429,7 @@
dom0_getdomaininfolist_t getdomaininfolist;
dom0_platform_quirk_t platform_quirk;
dom0_physical_memory_map_t physical_memory_map;
+ dom0_vcpusincrease_t vcpusincrease;
} u;
} dom0_op_t;
diff -r 015f8ae81276 xen/include/xen/domain.h
--- a/xen/include/xen/domain.h Mon Oct 10 14:38:01 2005
+++ b/xen/include/xen/domain.h Mon Oct 10 17:26:54 2005
@@ -1,6 +1,8 @@
#ifndef __XEN_DOMAIN_H__
#define __XEN_DOMAIN_H__
+
+extern int do_createvcpu(struct domain *d, int vcpuid);
/*
* Arch-specifics.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Ryan Harper
2005-Oct-11 15:16 UTC
[Xen-devel] [PATCH] 2nd try: 2/2 VCPU creation and allocation
Allocation map.
--
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253 T/L: 678-9253
ryanh@us.ibm.com
diffstat output:
tools/examples/xmexample1 | 4 +
tools/examples/xmexample2 | 3 +
tools/python/xen/lowlevel/xc/xc.c | 2
tools/python/xen/xend/XendDomainInfo.py | 16 ++++-
tools/python/xen/xm/create.py | 6 ++
xen/common/dom0_ops.c | 90 +++++++++++++++++---------------
xen/common/domain.c | 28 +++++++++
xen/common/schedule.c | 11 ++-
xen/include/public/dom0_ops.h | 2
xen/include/xen/domain.h | 2
xen/include/xen/sched.h | 2
11 files changed, 114 insertions(+), 52 deletions(-)
Signed-off-by: Ryan Harper <ryanh@us.ibm.com>
---
diff -r 12d1e93653c2 tools/examples/xmexample1
--- a/tools/examples/xmexample1 Tue Oct 11 00:13:12 2005
+++ b/tools/examples/xmexample1 Mon Oct 10 19:35:55 2005
@@ -27,6 +27,10 @@
# Number of Virtual CPUS to use, default is 1
#vcpus = 1
+
+# A bitmap of which physical cpus are vcpus allowed to use.
+# ex1: 0x2 <-- bit 1 set means all vcpus will be created on CPU1
+#allocmap = 0xffffffff # default value, vcpus can run on any cpu.
#----------------------------------------------------------------------------
# Define network interfaces.
diff -r 12d1e93653c2 tools/examples/xmexample2
--- a/tools/examples/xmexample2 Tue Oct 11 00:13:12 2005
+++ b/tools/examples/xmexample2 Mon Oct 10 19:35:55 2005
@@ -58,6 +58,9 @@
# Number of Virtual CPUS to use, default is 1
#vcpus = 1
vcpus = 4 # make your domain a 4-way
+
+# A bitmap of which physical cpus are vcpus allowed to use.
+allocmap = 0x2 # start all of your VCPUs on CPU1
#----------------------------------------------------------------------------
# Define network interfaces.
diff -r 12d1e93653c2 tools/python/xen/lowlevel/xc/xc.c
--- a/tools/python/xen/lowlevel/xc/xc.c Tue Oct 11 00:13:12 2005
+++ b/tools/python/xen/lowlevel/xc/xc.c Mon Oct 10 19:35:55 2005
@@ -185,7 +185,7 @@
static char *kwd_list[] = { "dom", "vcpu",
"cpumap", NULL };
- if ( !PyArg_ParseTupleAndKeywords(args, kwds, "i|ii", kwd_list,
+ if ( !PyArg_ParseTupleAndKeywords(args, kwds, "i|iL", kwd_list,
&dom, &vcpu, &cpumap) )
return NULL;
diff -r 12d1e93653c2 tools/python/xen/xend/XendDomainInfo.py
--- a/tools/python/xen/xend/XendDomainInfo.py Tue Oct 11 00:13:12 2005
+++ b/tools/python/xen/xend/XendDomainInfo.py Mon Oct 10 19:35:55 2005
@@ -266,6 +266,7 @@
result[''maxmem''] = get_cfg(''maxmem'',
int)
result[''maxmem_kb''] =
get_cfg(''maxmem_kb'', int)
result[''cpu''] = get_cfg(''cpu'',
int)
+ result[''allocmap''] =
get_cfg(''allocmap'', int)
result[''image''] = get_cfg(''image'')
try:
@@ -438,6 +439,7 @@
defaultInfo(''cpu_weight'', lambda: 1.0)
defaultInfo(''vcpus'', lambda: 1)
defaultInfo(''vcpu_avail'', lambda: (1 <<
self.info[''vcpus'']) - 1)
+ defaultInfo(''allocmap'' , lambda: None)
defaultInfo(''bootloader'', lambda: None)
defaultInfo(''backend'', lambda: [])
defaultInfo(''device'', lambda: [])
@@ -1018,6 +1020,16 @@
self.image.handleBootloading()
xc.domain_setcpuweight(self.domid,
self.info[''cpu_weight''])
+ cpu = self.info[''cpu'']
+ if cpu is not None and cpu != -1:
+ xc.domain_pincpu(self.domid, 0, 1 << cpu)
+
+ # set the domain alloc map for future vcpus,
+ # repin VCPU0 according to the alloc map
+ allocmap = self.info[''allocmap'']
+ if self.domid and allocmap:
+ xc.domain_pincpu(self.domid, -1, allocmap) # domain allocmap
+ xc.domain_pincpu(self.domid, 0, allocmap) # repin VCPU0
# increase the total number of vcpus in domain
xc.domain_vcpus_increase(self.domid,
int(self.info[''vcpus'']));
@@ -1026,10 +1038,6 @@
m =
self.image.getDomainMemory(self.info[''memory_KiB''])
xc.domain_setmaxmem(self.domid, m)
xc.domain_memory_increase_reservation(self.domid, m, 0, 0)
-
- cpu = self.info[''cpu'']
- if cpu is not None and cpu != -1:
- xc.domain_pincpu(self.domid, 0, 1 << cpu)
self.info[''start_time''] = time.time()
diff -r 12d1e93653c2 tools/python/xen/xm/create.py
--- a/tools/python/xen/xm/create.py Tue Oct 11 00:13:12 2005
+++ b/tools/python/xen/xm/create.py Mon Oct 10 19:35:55 2005
@@ -151,6 +151,10 @@
gopts.var(''cpu'', val=''CPU'',
fn=set_int, default=None,
use="CPU to run the domain on.")
+
+gopts.var(''allocmap'', val=''ALLOCMAP'',
+ fn=set_int, default=None,
+ use="Set default cpumap used for allocating vcpus.")
gopts.var(''vcpus'', val=''VCPUS'',
fn=set_int, default=1,
@@ -562,6 +566,8 @@
config.append([''maxmem'', vals.maxmem])
if vals.cpu is not None:
config.append([''cpu'', vals.cpu])
+ if vals.allocmap is not None:
+ config.append([''allocmap'', vals.allocmap])
if vals.cpu_weight is not None:
config.append([''cpu_weight'', vals.cpu_weight])
if vals.blkif:
diff -r 12d1e93653c2 xen/common/dom0_ops.c
--- a/xen/common/dom0_ops.c Tue Oct 11 00:13:12 2005
+++ b/xen/common/dom0_ops.c Mon Oct 10 19:35:55 2005
@@ -275,7 +275,7 @@
{
domid_t dom = op->u.pincpudomain.domain;
struct domain *d = find_domain_by_id(dom);
- struct vcpu *v;
+ struct vcpu *v = NULL;
cpumap_t cpumap;
@@ -285,27 +285,33 @@
break;
}
- if ( (op->u.pincpudomain.vcpu >= MAX_VIRT_CPUS) ||
- !d->vcpu[op->u.pincpudomain.vcpu] )
- {
- ret = -EINVAL;
- put_domain(d);
- break;
- }
-
- v = d->vcpu[op->u.pincpudomain.vcpu];
- if ( v == NULL )
- {
- ret = -ESRCH;
- put_domain(d);
- break;
- }
-
- if ( v == current )
- {
- ret = -EINVAL;
- put_domain(d);
- break;
+ /* don''t bail on vcpu = -1 as that sets domain cpumap */
+ if ( (op->u.pincpudomain.vcpu != -1) &&
+ ((op->u.pincpudomain.vcpu >= MAX_VIRT_CPUS) ||
+ !d->vcpu[op->u.pincpudomain.vcpu])
+ )
+ {
+ ret = -EINVAL;
+ put_domain(d);
+ break;
+ }
+
+ /* don''t get a struct vcpu pointer for -1 op */
+ if (op->u.pincpudomain.vcpu != -1) {
+ v = d->vcpu[op->u.pincpudomain.vcpu];
+ if ( v == NULL )
+ {
+ ret = -ESRCH;
+ put_domain(d);
+ break;
+ }
+
+ if ( v == current )
+ {
+ ret = -EINVAL;
+ put_domain(d);
+ break;
+ }
}
if ( copy_from_user(&cpumap, op->u.pincpudomain.cpumap,
@@ -316,24 +322,28 @@
break;
}
- /* update cpumap for this vcpu */
- v->cpumap = cpumap;
-
- if ( cpumap == CPUMAP_RUNANYWHERE )
- {
- clear_bit(_VCPUF_cpu_pinned, &v->vcpu_flags);
- }
- else
- {
- /* pick a new cpu from the usable map */
- int new_cpu = (int)find_first_set_bit(cpumap) % num_online_cpus();
-
- vcpu_pause(v);
- vcpu_migrate_cpu(v, new_cpu);
- set_bit(_VCPUF_cpu_pinned, &v->vcpu_flags);
- vcpu_unpause(v);
- }
-
+ /* update domain vcpu alloc map */
+ if ( v == NULL ) {
+ d->allocmap = cpumap;
+ } else {
+ /* update cpumap for this vcpu */
+ v->cpumap = cpumap;
+
+ if ( cpumap == CPUMAP_RUNANYWHERE )
+ {
+ clear_bit(_VCPUF_cpu_pinned, &v->vcpu_flags);
+ }
+ else
+ {
+ /* pick a new cpu from the usable map */
+ int new_cpu = get_next_processor(d, v, &cpumap);
+
+ vcpu_pause(v);
+ vcpu_migrate_cpu(v, new_cpu);
+ set_bit(_VCPUF_cpu_pinned, &v->vcpu_flags);
+ vcpu_unpause(v);
+ }
+ }
put_domain(d);
}
break;
diff -r 12d1e93653c2 xen/common/domain.c
--- a/xen/common/domain.c Tue Oct 11 00:13:12 2005
+++ b/xen/common/domain.c Mon Oct 10 19:35:55 2005
@@ -41,6 +41,7 @@
atomic_set(&d->refcnt, 1);
atomic_set(&v->pausecnt, 0);
+ d->allocmap = CPUMAP_RUNANYWHERE;
d->domain_id = dom_id;
v->processor = cpu;
@@ -380,7 +381,7 @@
v = d->vcpu[vcpuid];
atomic_set(&v->pausecnt, 0);
- v->cpumap = CPUMAP_RUNANYWHERE;
+ v->cpumap = d->allocmap;
memcpy(&v->arch, &idle0_vcpu.arch, sizeof(v->arch));
@@ -469,6 +470,31 @@
return -ENOSYS;
}
+
+/* find the least loaded processor , ignorning vcpu v, in cpumap_t *map */
+int get_next_processor(struct domain* d, struct vcpu *v, cpumap_t *map) {
+ struct vcpu *vc = NULL;
+ int pro, i, cnt[NR_CPUS] = { 0 };
+
+ /* count the processor layout for this dom, except for vcpu v
+ * whose processor field may not have been set yet. */
+ for_each_vcpu( d, vc ) {
+ if (vc->vcpu_id != v->vcpu_id)
+ cnt[vc->processor]++;
+ }
+
+ /* start from the first allowable cpu, guard against bogus cpus */
+ pro = (int)find_first_set_bit(*map) % num_online_cpus();
+
+ /* pick least loaded processor in the map */
+ for ( i = pro; i < num_online_cpus(); i++ ) {
+ if ( test_bit(i, &*map) && (cnt[i] <= cnt[pro]) )
+ pro = i;
+ }
+
+ return pro;
+}
+
/*
* Local variables:
diff -r 12d1e93653c2 xen/common/schedule.c
--- a/xen/common/schedule.c Tue Oct 11 00:13:12 2005
+++ b/xen/common/schedule.c Mon Oct 10 19:35:55 2005
@@ -125,12 +125,13 @@
v->next_in_list = vc->next_in_list;
vc->next_in_list = v;
- if (test_bit(_VCPUF_cpu_pinned, &vc->vcpu_flags)) {
- v->processor = (vc->processor + 1) % num_online_cpus();
+
+ /* XXX: if previous vcpu was pinned, mark new vcpu as pinned why? */
+ if (test_bit(_VCPUF_cpu_pinned, &vc->vcpu_flags))
set_bit(_VCPUF_cpu_pinned, &v->vcpu_flags);
- } else {
- v->processor = (vc->processor + 1) % num_online_cpus();
- }
+
+ v->processor = get_next_processor(d, v, &d->allocmap);
+
}
return v;
diff -r 12d1e93653c2 xen/include/public/dom0_ops.h
--- a/xen/include/public/dom0_ops.h Tue Oct 11 00:13:12 2005
+++ b/xen/include/public/dom0_ops.h Mon Oct 10 19:35:55 2005
@@ -181,7 +181,7 @@
typedef struct {
/* IN variables. */
domid_t domain;
- u16 vcpu;
+ s16 vcpu;
cpumap_t *cpumap;
} dom0_pincpudomain_t;
diff -r 12d1e93653c2 xen/include/xen/domain.h
--- a/xen/include/xen/domain.h Tue Oct 11 00:13:12 2005
+++ b/xen/include/xen/domain.h Mon Oct 10 19:35:55 2005
@@ -27,4 +27,6 @@
extern void dump_pageframe_info(struct domain *d);
+int get_next_processor(struct domain *d, struct vcpu *v, cpumap_t *map);
+
#endif /* __XEN_DOMAIN_H__ */
diff -r 12d1e93653c2 xen/include/xen/sched.h
--- a/xen/include/xen/sched.h Tue Oct 11 00:13:12 2005
+++ b/xen/include/xen/sched.h Mon Oct 10 19:35:55 2005
@@ -134,6 +134,8 @@
/* Bitmask of CPUs which are holding onto this domain''s state. */
cpumask_t cpumask;
+
+ cpumap_t allocmap; /* vcpu allocation bitmap */
struct arch_domain arch;
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
Keir Fraser
2005-Oct-12 16:10 UTC
[Xen-devel] Re: [PATCH] 2nd try: 2/2 VCPU creation and allocation
On 11 Oct 2005, at 16:16, Ryan Harper wrote:> Allocation map.What does this patch buy us, given that we already support per-vcpu maps? If I understand correctly, this adds a per-domain map, which seems redundant to me. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Ryan Harper
2005-Oct-12 16:15 UTC
[Xen-devel] Re: [PATCH] 2nd try: 2/2 VCPU creation and allocation
* Keir Fraser <Keir.Fraser@cl.cam.ac.uk> [2005-10-12 11:05]:> > On 11 Oct 2005, at 16:16, Ryan Harper wrote: > > >Allocation map. > > What does this patch buy us, given that we already support per-vcpu > maps? If I understand correctly, this adds a per-domain map, which > seems redundant to me.I guess not as much as it did before we had build-time vcpu creation. I can easily translate the allocation map in the configuration file into a set of pincpu ops before allocating domain memory and achieve the same effect as the patch does without the changes to the domain structure and pincpu dom0_op. I''ll resend without the changes to xen. -- Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx (512) 838-9253 T/L: 678-9253 ryanh@us.ibm.com _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Reasonably Related Threads
- [PATCH] make x86_64 vcpu hotplug work like i386
- [PATCH 00 of 10 v3] Automatic NUMA placement for xl
- [PATCH] libxl: fix rtc_timeoffset setting
- [PATCH 5/6] xen, tools: calculate nr_cpus via num_online_cpus
- xl doesn't honour the parameter cpu_weight from my config file while xm does honour it