Xen devel - Jun 2005 - RE: [PATCH][RESUBMIT] don''t schedule unplugged vcpus

> I''ve added in changes to DOM0_GETDOMINFO and 
> DOM0_GETVCPUCONTEXT hypercalls.  dominfo now creates a 
> vcpu_online_map bitmap which marks whether vcpus are up or 
> down.  I didn''t want to clobber either the total number of 
> vcpus allocated to a domain (n_vcpu) nor the vcpu_to_cpu 
> mapping since both are still valid whether the vcpu is being 
> scheduled or not.
I don''t see why we care about vcpus that are down. From the
user''s point
of view they''ve gone for good -- it just happens that Xen
hasn''t freed
the memory in anticipation of it being used again. What do you think?

I''d be inclined just to enter ''-1'' in the vcpu_to_cpu
map. BTW: we could
make it an s16 rather than s32 at the same time. I think 32,768 CPUs
should keep be enough for anyone :-) 

Ian
 
> I modified vcpucontext to give the context for the first vcpu 
> not down.
> If the requested vcpu is down, it will return the context of 
> the next vcpu that is up, or -ESRCH if no vcpu past the 
> requested vcpu is valid.
> 
> I modified xm list -v to display an ONLINE column which 
> indicates the online status of each vcpu in a domain.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 8 Jun 2005, at 22:42, Ian Pratt wrote:
> I don''t see why we care about vcpus that are down. From the
user''s
> point
> of view they''ve gone for good -- it just happens that Xen
hasn''t freed
> the memory in anticipation of it being used again. What do you think?
>
> I''d be inclined just to enter ''-1'' in the
vcpu_to_cpu map. BTW: we
> could
> make it an s16 rather than s32 at the same time. I think 32,768 CPUs
> should keep be enough for anyone :-)
This is how I view it. We don''t free the vcpu structure only because it
isn''t reference counted. We can only be sure that noone has a reference
to the structure when the entire domain''s refcnt falls to zero. Given 
the small amount of memory involved, it''s not worth the pain or 
run-time cost of adding per-vcpu reference counts.

So VCPU_down == invisible outside Xen.

  -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

* Ian Pratt <m+Ian.Pratt@cl.cam.ac.uk> [2005-06-08
16:44]:
> > I''ve added in changes to DOM0_GETDOMINFO and 
> > DOM0_GETVCPUCONTEXT hypercalls.  dominfo now creates a 
> > vcpu_online_map bitmap which marks whether vcpus are up or 
> > down.  I didn''t want to clobber either the total number of 
> > vcpus allocated to a domain (n_vcpu) nor the vcpu_to_cpu 
> > mapping since both are still valid whether the vcpu is being 
> > scheduled or not.
> 
> I don''t see why we care about vcpus that are down. From the
user''s point
> of view they''ve gone for good -- it just happens that Xen
hasn''t freed
> the memory in anticipation of it being used again. What do you think?
I don''t think the users care so much as I want tools to be able to
extract the current state of affairs for load balancing or other
operations that want to account for domain state.
> 
> I''d be inclined just to enter ''-1'' in the
vcpu_to_cpu map. BTW: we could
> make it an s16 rather than s32 at the same time. I think 32,768 CPUs
> should keep be enough for anyone :-) 
Sure.  

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

* Keir Fraser <Keir.Fraser@cl.cam.ac.uk> [2005-06-08
16:57]:
> 
> On 8 Jun 2005, at 22:42, Ian Pratt wrote:
> 
> >I don''t see why we care about vcpus that are down. From the
user''s
> >point
> >of view they''ve gone for good -- it just happens that Xen
hasn''t freed
> >the memory in anticipation of it being used again. What do you think?
> >
> >I''d be inclined just to enter ''-1'' in the
vcpu_to_cpu map. BTW: we
> >could
> >make it an s16 rather than s32 at the same time. I think 32,768 CPUs
> >should keep be enough for anyone :-)
> 
> This is how I view it. We don''t free the vcpu structure only
because it
> isn''t reference counted. We can only be sure that noone has a
reference
> to the structure when the entire domain''s refcnt falls to zero.
Given
> the small amount of memory involved, it''s not worth the pain or 
> run-time cost of adding per-vcpu reference counts.
> 
> So VCPU_down == invisible outside Xen.
So, when I trigger a vcpu to go down via dom0 xm operation, I have to
trust that it worked?  I have no way of knowing at some point later
which vcpus are up or down?  I don''t see any cost to this other than
during the getdominfo hcall.  

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 8 Jun 2005, at 23:00, Ryan Harper wrote:
> So, when I trigger a vcpu to go down via dom0 xm operation, I have to
> trust that it worked?  I have no way of knowing at some point later
> which vcpus are up or down?  I don''t see any cost to this other
than
> during the getdominfo hcall.
The ones you can retrieve info about are up. The ones you can''t, are 
down. :-)

Let me put it this way: I reserve the right in future to change my mind 
and free the vcpu info structures for vcpus that are down (and 
re-allocate when they come back up). If I do that then you will not get 
info about them via hypercalls. There''s no good reason to return info 
about them -- you can infer they are ''down''/non-existent from
their
absence -- so why bother?

  -- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

* Keir Fraser <Keir.Fraser@cl.cam.ac.uk> [2005-06-08
17:07]:
> 
> On 8 Jun 2005, at 23:00, Ryan Harper wrote:
> 
> >So, when I trigger a vcpu to go down via dom0 xm operation, I have to
> >trust that it worked?  I have no way of knowing at some point later
> >which vcpus are up or down?  I don''t see any cost to this
other than
> >during the getdominfo hcall.
> 
> The ones you can retrieve info about are up. The ones you can''t,
are
> down. :-)
Indeed. =)
> Let me put it this way: I reserve the right in future to change my mind 
> and free the vcpu info structures for vcpus that are down (and 
> re-allocate when they come back up). If I do that then you will not get 
> info about them via hypercalls. There''s no good reason to return
info
> about them -- you can infer they are ''down''/non-existent
from their
> absence -- so why bother?
It seemed nicer/easier/cheaper (to me) to mark up/down in the dominfo
hypercall rather than issue MAX_VIRT_CPUS vcpucontext hypercalls.  

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 8 Jun 2005, at 23:18, Ryan Harper wrote:
>> Let me put it this way: I reserve the right in future to change my 
>> mind
>> and free the vcpu info structures for vcpus that are down (and
>> re-allocate when they come back up). If I do that then you will not 
>> get
>> info about them via hypercalls. There''s no good reason to
return info
>> about them -- you can infer they are
''down''/non-existent from their
>> absence -- so why bother?
>
> It seemed nicer/easier/cheaper (to me) to mark up/down in the dominfo
> hypercall rather than issue MAX_VIRT_CPUS vcpucontext hypercalls.
Setting vcpu_to_cpu[vcpu] to -1 for down vcpus sounds sensible to me. 
Then you only need to issue vcpucontext calls for vcpus for which 
vcpu_to_cpu[] >= 0.

If it might also be possible to have vcpus that aren''t down, but also 
aren;t allocated to a cpu right now for some reason, we probably want 
two invalid values to distinguish the two cases. Perhaps the latter 
situation doesn''t make sense though, in which case we can ignore it.

  -- Kier


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel