diffstat domgrps-vmm.patch arch/ia64/xen/xensetup.c | 7 arch/powerpc/powerpc64/hypercall_table.S | 1 arch/powerpc/setup.c | 7 arch/x86/setup.c | 8 arch/x86/x86_32/entry.S | 2 arch/x86/x86_64/entry.S | 3 common/Makefile | 2 common/domain.c | 8 common/domctl.c | 10 common/domgrp.c | 315 +++++++++++++++++++++++++++++++ common/domgrpctl.c | 134 +++++++++++++ include/public/domctl.h | 2 include/public/domgrpctl.h | 86 ++++++++ include/public/xen.h | 7 include/xen/domgrp.h | 36 +++ include/xen/hypercall.h | 5 include/xen/sched.h | 26 ++ 17 files changed, 657 insertions(+), 2 deletions(-) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Samuel Thibault
2007-Dec-18 16:13 UTC
Re: [Xen-devel] [RFC 1/2] Domain Groups - VMM Support
Hello, Chris, le Tue 18 Dec 2007 11:02:55 -0500, a écrit :> @@ -520,7 +523,9 @@ > unsigned long nr_pt_frames; /* Number of bootstrap p.t. frames. */ > unsigned long mfn_list; /* VIRTUAL address of page-frame list. */ > unsigned long mod_start; /* VIRTUAL address of pre-loaded module. */ > + /* or VIRTUAL address of module_t array */ > unsigned long mod_len; /* Size (bytes) of pre-loaded module. */ > + /* or length of module_t array. */ > int8_t cmd_line[MAX_GUEST_CMDLINE]; > }; > typedef struct start_info start_info_t;It looks like this doesn''t belong to the patch. However, I''m interested in this for the GNU/Mach Xen support, where we would need to have several modules, with several command lines. Could you give more details? Samuel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Dec 18, 2007, at 11:13 AM, Samuel Thibault wrote:> Chris, le Tue 18 Dec 2007 11:02:55 -0500, a écrit : >> @@ -520,7 +523,9 @@ >> unsigned long nr_pt_frames; /* Number of bootstrap p.t. >> frames. */ >> unsigned long mfn_list; /* VIRTUAL address of page-frame >> list. */ >> unsigned long mod_start; /* VIRTUAL address of pre-loaded >> module. */ >> + /* or VIRTUAL address of module_t >> array */ >> unsigned long mod_len; /* Size (bytes) of pre-loaded >> module. */ >> + /* or length of module_t >> array. */ >> int8_t cmd_line[MAX_GUEST_CMDLINE]; >> }; >> typedef struct start_info start_info_t; > > It looks like this doesn''t belong to the patch. However, I''m > interested > in this for the GNU/Mach Xen support, where we would need to have > several modules, with several command lines. Could you give more > details?Samuel, You''re right, those comments snuck in from another project. An interest of our team is domain decomposition. Members of the team created a domain builder separate from dom0 (conceptually similar to Derek Murray''s recent project based on Rolf Neugebauer''s "initial domains" patch) that accepts grub modules as input. So, your suspicion is correct that the comments don''t belong in this patch and are related to modifications used to accommodate several modules and command line arguments. Thanks for catching my mistake. I also welcome such an attentive critique of the proposed grouping functionality. :) Cheers, Chris _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Chris
2008-Jan-22 14:20 UTC
Re: [Xen-devel] [RESEND] [RFC 1/2] Domain Groups - VMM Support
On Dec 18, 2007, at 11:13 AM, Samuel Thibault wrote:> Chris, le Tue 18 Dec 2007 11:02:55 -0500, a écrit : >> @@ -520,7 +523,9 @@ >> unsigned long nr_pt_frames; /* Number of bootstrap p.t. >> frames. */ >> unsigned long mfn_list; /* VIRTUAL address of page-frame >> list. */ >> unsigned long mod_start; /* VIRTUAL address of pre-loaded >> module. */ >> + /* or VIRTUAL address of module_t >> array */ >> unsigned long mod_len; /* Size (bytes) of pre-loaded >> module. */ >> + /* or length of module_t >> array. */ >> int8_t cmd_line[MAX_GUEST_CMDLINE]; >> }; >> typedef struct start_info start_info_t; > > It looks like this doesn''t belong to the patch. However, I''m > interested > in this for the GNU/Mach Xen support, where we would need to have > several modules, with several command lines. Could you give more > details?Samuel, You''re right, those comments snuck in from another project. An interest of our team is domain decomposition. Members of the team created a domain builder separate from dom0 (conceptually similar to Derek Murray''s recent project based on Rolf Neugebauer''s "initial domains" patch) that accepts grub modules as input. So, your suspicion is correct that the comments don''t belong in this patch and are related to modifications used to accommodate several modules and command line arguments. Thanks for catching my mistake. I also welcome such an attentive critique of the proposed grouping functionality. :) Cheers, Chris _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Samuel Thibault
2008-Jan-22 14:40 UTC
Re: [Xen-devel] [RESEND] [RFC 1/2] Domain Groups - VMM Support
Hello, Actually I thought I had already answered this, but I can''t find a trace... The patch looked sane to me, but maybe it would be useful to see it merged with the other group patch proposed by Mike D. Day. I guess it is mostly a matter of plumbing both together. Samuel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Chris
2008-Jan-22 14:51 UTC
Re: [Xen-devel] [RESEND] [RFC 1/2] Domain Groups - VMM Support
On Jan 22, 2008, at 9:40 AM, Samuel Thibault wrote:> Actually I thought I had already answered this, but I can''t find a > trace...I think there was some mischief from the xen-devel list server late last week. Please disregard my duplicated response, assuming your question was answered.> The patch looked sane to me, but maybe it would be useful to see it > merged with the other group patch proposed by Mike D. Day. I guess it > is mostly a matter of plumbing both together.I''m in the process of updating my domgrps patches to facilitate scheduling information for compatibility with Mike''s schegrps. I''ll certainly post the revisions once completed. Cheers, Chris _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> How do you recognize that it is a module_t array? Magic values?A magic argument ("multimodules") is appended to the VMM boot parameters, which signals that the mod_start and mod_len variables are to be interpreted in the context of multiple grub modules.> Would it be possible to see that merged into xen-unstable?At the moment the changes are not suitable for submission primarily because they''re applicable to a stale release of xen. I''ve asked the folks responsible to work something up for general consumption. We''ll test it internally and then submit it for public discussion. Cheers, Chris _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 18/12/07 11:02 -0500, Chris wrote:> --- xen-unstable/xen/arch/x86/x86_32/entry.S 2007-11-19 10:38:08.000000000 -0500 > +++ xen-unstable-trp-domgrps-rebase-tip/xen/arch/x86/x86_32/entry.S 2007-11-19 18:42:00.000000000 -0500 > @@ -682,6 +682,7 @@ > .long do_sysctl /* 35 */ > .long do_domctl > .long do_kexec_op > + .long do_domgrpctl > .rept NR_hypercalls-((.-hypercall_table)/4) > .long do_ni_hypercall > .endr > @@ -725,6 +726,7 @@ > .byte 1 /* do_sysctl */ /* 35 */ > .byte 1 /* do_domctl */ > .byte 2 /* do_kexec_op */ > + .byte 1 /* do_domgrpctl */ > .rept NR_hypercalls-(.-hypercall_args_table)I looked at creating a new hypervisor for scheduling groups, but found it much simpler to add a new sub-function under the dom_ctl hypercall. It markedly reduced the size of the patch and I didn''t see any downside. It might make sense to do the same thing here. That way you wouldn''t need to add the new hypercall to every jump table.> --- xen-unstable/xen/common/domgrp.c 1969-12-31 19:00:00.000000000 -0500 > +++ xen-unstable-trp-domgrps-rebase-tip/xen/common/domgrp.c 2007-11-26 16:46:46.000000000 -0500 > @@ -0,0 +1,315 @@ > +/****************************************************************************** > + * domgrp.c > + * > + * Generic domain group-handling functions. > + *This is a really good grouping mechanism. To merge with scheduling groups, it may be nice to have an extensible structure like we do with schedulers in Xen. The scheduler pattern is a good one to follow, it separates general scheduling operations from specific implementations. We could do something like that here. I doubt that the set of group operations in this patch is the sum total of all useful group operations on domains so it needs to be simple to add group operations in the future without breaking any hypercalls and without disruptive patches. I think its worth an attempt to merge the scheduling groups with domain groups. I''ll start looking at some of the ideas I mentioned above, be happy to work with anyone else who wants to have a go. Mike -- Mike D. Day IBM LTC Cell: 919 412-3900 Sametime: ncmike@us.ibm.com AIM: ncmikeday Yahoo: ultra.runner PGP key: http://www.ncultra.org/ncmike/pubkey.asc _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Jan 31, 2008, at 3:04 PM, Mike D. Day wrote:> I looked at creating a new hypervisor for scheduling groups, but found > it much simpler to add a new sub-function under the dom_ctl > hypercall. It markedly reduced the size of the patch and I didn''t see > any downside. It might make sense to do the same thing here. That way > you wouldn''t need to add the new hypercall to every jump table.I have no objection to making domgrpctl become a set of sub-ops to the existing domain management hypercall. In fact, with the predecessor to domctl (dom0_op) that''s how it was implemented. During the domctl developmental churn I separated them to make my life easier, but now it''s just an artifact.> To merge with scheduling groups, it may be nice to have an > extensible structure like we do with schedulers in Xen.I agree. There''s more to groups than what the two of us have released thus far and adding new features should be made as painless as possible. My suggestion is to save this type of overhaul until we settle on an agreeable functionality merge.> I think its worth an attempt to merge the scheduling groups with > domain groups. I''ll start looking at some of the ideas I mentioned > above, be happy to work with anyone else who wants to have a go.I''m also having a go at the merge and will release a patch set RFC hopefully well before leaving on travel at the end of next week. With regard to the merge, I''ve got a question for you. In schedgrps, when a domain joins a group the assumption is that credits are thereafter shared among group members (for as long as it''s a member). I think there are cases where it makes sense to group domains without implicitly sharing credits (the null-group in domgrps is such a case). So, I''d like to make it optional for members to delegate their VCPUs to the group. The question is, do you have an opinion whether domains should have to opt-in or opt-out of sharing their credits when joining a group? Cheers, Chris _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Jan 31, 2008, at 3:04 PM, Mike D. Day wrote:> I think its worth an attempt to merge the scheduling groups with > domain groups. I''ll start looking at some of the ideas I mentioned > above, be happy to work with anyone else who wants to have a go.Hi Mike, Here''s a rough cut of the domgrps/schedgrps merge. It includes a slightly revised version of domgrps with some enhancements for performance and compatibility with schedgrps. I''d appreciate your feedback to make sure the merged result accurately represents your intent. DomGrps changes: - add per-domgrp scheduler-specific data SchedDom Changes: - when each domgrp is created, make a csched_dom struct to act as the group representative - when doms join the group, delegate its VCPUs to its representative (instead of a master dom) - removed group management functions, changes to tools, and command line interface As of right now, when a domain joins a group it must relinquish its VCPU(s) to the group. In the near future I''d like to add the following: - configuration option choose whether a dom will relinquish its VCPUs to the group - modify xm to show whether members have relinquished their VCPUs - documentation :) I haven''t done any benchmarks, but on the fast path the extra code should be nearly identical to your original schedgrps implementation, so performance should be the very similar. Other than the merge itself, the practical benefit is that schegrps are now persistent across reboots and host migration (although I haven''t tested the latter). The attached patches should apply cleanly to xen-unstable.hg cs 16975. The intended patch order is: domgrps-vmm.patch, schedgrps.patch, domgrps-tools.patch. Cheers, Chris _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 4/2/08 19:14, "Chris" <hap10@tycho.ncsc.mil> wrote:> The attached patches should apply cleanly to xen-unstable.hg cs > 16975. The intended patch order is: domgrps-vmm.patch, > schedgrps.patch, domgrps-tools.patch.Hmmm... again the generic domgrp stuff seems like a huge amount of code with few moving parts that would affect user experience of Xen. The original schedgrps code by comparison is tiny. I don''t mind taking big gobs of code if there are genuine use cases, but I won''t big pieces of ''architecture'' without good reason. I''m inclined to take Mike''s patches roughly as he previously posted them. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Feb 4, 2008, at 5:56 PM, Keir Fraser wrote:> On 4/2/08 19:14, "Chris" <hap10@tycho.ncsc.mil> wrote: > Hmmm... again the generic domgrp stuff seems like a huge amount of > code with > few moving parts that would affect user experience of Xen. The > original > schedgrps code by comparison is tiny. I don''t mind taking big gobs > of code > if there are genuine use cases, but I won''t big pieces of > ''architecture'' > without good reason. I''m inclined to take Mike''s patches roughly as he > previously posted them.Use cases for domain groups that exist today: - domain management (e.g. group migration) - scheduler integration (Mike''s schedgrps) Use cases coming down the pike: - XSM integration (e.g. express policy in terms of groups) - domain relationships (e.g. failure handling) - resource management (e.g. allocating a resource to a set of domains) - set_target/IS_PRIV_FOR integration (e.g. to establish privilege over a set of domains) These use cases stem from where we see domain decomposition headed and the challenges it creates. Can some notion of groups can be implemented in each case without an underlying VMM-backed, non-hierarchical group architecture? You bet. However, each implementation is likely to have a separate management interface and impose (possibly conflicting) hierarchies between domains. For such independent components to take advantage of one another, each would have to translate their respective notions of groups, assuming such a translation exists. On the issue of code size, take Mike''s schedgrps for example, which was very small as originally posted. After integration with domgrps, it shrank to less than 40% of its original size (259 insertions down from 681) and it no longer induced a domain hierarchy. Could domgrps be smaller? Yes, and thanks to Mike''s recent suggestions, I have some concrete ways to make that happen. If you have some specific areas where you think domgrps is unnecessarily large, I''m willing to make changes. Another way to look at the code size issue is that the integration with schedgrps shaved off well over over 400 LOC, mostly by removing redundant management code from schedgrps. At that rate it would take two, maybe three more group- aware projects for the domgrps architecture to break even in terms of code size. But it sounds like the main objection is lack of existing use cases. They''re coming... slowly. The best I can say is that I''m working to identify and mitigate future challenges before they cause problems. Is there critical mass for a generic group architecture yet? I think so, but the case should only get stronger with time. Cheers, Chris _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 5/2/08 22:20, "Chris" <hap10@tycho.ncsc.mil> wrote:> On the issue of code size, take Mike''s schedgrps for example, which > was very small as originally posted. After integration with domgrps, > it shrank to less than 40% of its original size (259 insertions down > from 681) and it no longer induced a domain hierarchy.If credit-sharing is made configurable (as you would surely want it to be if domgrps are to have other uses) then a reasonable number of those lines of code will reappear, and spread across tools and hypervisor.> But it sounds like the main objection is lack of existing use cases. > They''re coming... slowly. The best I can say is that I''m working to > identify and mitigate future challenges before they cause problems. > Is there critical mass for a generic group architecture yet? I think > so, but the case should only get stronger with time.I''m driven by concrete use cases. Several of the upcoming uses you mention need careful consideration of what they are useful for, to determine the best way to design them into the system. Take resource sharing. Stub domains sharing scheduler credits with the HVM guest is a rather special case, and one where a master/slave relationship is not unreasonable (and hence in this case I think it is arguable whether it is actually a good fit with domgrps after all). If you really want resource sharing amongst a peer group of domains, what does that actually mean? Does each domain have a private allocation of resource plus membership of one or more pools? If so, which resource set satisfies each resource request by a particular domain? Even if not, how is resource contention between domains belonging to the resource pool (e.g., for the static assignment of memory pages to that resource pool) managed and resolved? The obvious place would be the dom0 control plane -- in which case these resource pools can probably be implemented as a high-level tools abstraction with no direct hypervisor involvement at all. My fear is that a nice domgrps infrastructure doesn''t itself resolve the hard devils in the details, for all that it sounds architecturally neat. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser, le Wed 06 Feb 2008 09:20:51 +0000, a écrit :> On 5/2/08 22:20, "Chris" <hap10@tycho.ncsc.mil> wrote: > > > On the issue of code size, take Mike''s schedgrps for example, which > > was very small as originally posted. After integration with domgrps, > > it shrank to less than 40% of its original size (259 insertions down > > from 681) and it no longer induced a domain hierarchy. > > If credit-sharing is made configurable (as you would surely want it to be if > domgrps are to have other uses) then a reasonable number of those lines of > code will reappear, and spread across tools and hypervisor. > > > But it sounds like the main objection is lack of existing use cases. > > They''re coming... slowly. The best I can say is that I''m working to > > identify and mitigate future challenges before they cause problems. > > Is there critical mass for a generic group architecture yet? I think > > so, but the case should only get stronger with time. > > I''m driven by concrete use cases. Several of the upcoming uses you mention > need careful consideration of what they are useful for, to determine the > best way to design them into the system. Take resource sharing. Stub domains > sharing scheduler credits with the HVM guest is a rather special case, and > one where a master/slave relationship is not unreasonable (and hence in this > case I think it is arguable whether it is actually a good fit with domgrps > after all).Actually, in my former research team in Bordeaux, they would like to write a small domain that computes the scheduling of a bunch of others, for parallel scientific computing. Samuel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 6/2/08 10:27, "Samuel Thibault" <samuel.thibault@eu.citrix.com> wrote:>> I''m driven by concrete use cases. Several of the upcoming uses you mention >> need careful consideration of what they are useful for, to determine the >> best way to design them into the system. Take resource sharing. Stub domains >> sharing scheduler credits with the HVM guest is a rather special case, and >> one where a master/slave relationship is not unreasonable (and hence in this >> case I think it is arguable whether it is actually a good fit with domgrps >> after all). > > Actually, in my former research team in Bordeaux, they would like to > write a small domain that computes the scheduling of a bunch of others, > for parallel scientific computing.Well that sounds plausible, but I''m not sure how resource pools or domgrps would help with that. It sounds like a delegation mechanism (of privilege and/or resource) would be more appropriate. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Keir Fraser, le Wed 06 Feb 2008 10:37:37 +0000, a écrit :> On 6/2/08 10:27, "Samuel Thibault" <samuel.thibault@eu.citrix.com> wrote: > > >> I''m driven by concrete use cases. Several of the upcoming uses you mention > >> need careful consideration of what they are useful for, to determine the > >> best way to design them into the system. Take resource sharing. Stub domains > >> sharing scheduler credits with the HVM guest is a rather special case, and > >> one where a master/slave relationship is not unreasonable (and hence in this > >> case I think it is arguable whether it is actually a good fit with domgrps > >> after all). > > > > Actually, in my former research team in Bordeaux, they would like to > > write a small domain that computes the scheduling of a bunch of others, > > for parallel scientific computing. > > Well that sounds plausible, but I''m not sure how resource pools or domgrps > would help with that. It sounds like a delegation mechanism (of privilege > and/or resource) would be more appropriate.Sure, delegation will be needed, but there needs to be a way to specify which domains that scheduler has control on. Samuel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 6/2/08 10:42, "Samuel Thibault" <samuel.thibault@eu.citrix.com> wrote:>> Well that sounds plausible, but I''m not sure how resource pools or domgrps >> would help with that. It sounds like a delegation mechanism (of privilege >> and/or resource) would be more appropriate. > > Sure, delegation will be needed, but there needs to be a way to specify > which domains that scheduler has control on.It''s not clear to me that supporting any more than a delegation relation across pairs of domains is needed at the hypervisor level. Nor whether a richer grouping mechanism in this case would lead to greater run-time efficiency, fewer lines of code, or clearer code. -- Keir _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On Feb 6, 2008, at 4:20 AM, Keir Fraser wrote:> If credit-sharing is made configurable (as you would surely want it > to be if > domgrps are to have other uses) then a reasonable number of those > lines of > code will reappear, and spread across tools and hypervisor.Without testing, I''d say that making the domgrps/schedgrps merge provide logically equivalent functionality to the original schedgrps requires the addition of 1 conditional statement in the VMM: diff -r 82d60475e726 xen/common/domgrp.c --- a/xen/common/domgrp.c Wed Feb 06 09:01:54 2008 -0500 +++ b/xen/common/domgrp.c Wed Feb 06 09:47:38 2008 -0500 @@ -215,7 +215,8 @@ int add_dom_to_grp(struct domain *dom, d grp->size++; /* add dom to schedgrp */ - sched_join_group(dom); // FIXME: should be conditional/optional + if (dgid == NULL_GROUP_ID) + sched_join_group(dom); out: return ret; } In the domgrps model, all ungrouped domains are in the "null group." And in schedgrps model, all grouped domains automatically relinquish their VCPUs. So by preventing domains in the null group from relinquishing their VCPUs, the above change make the two implementations effectively equivalent: only explicitly grouped domains relinquish their VCPUs. Additional configuration options will be desirable and will take code, but in this case domgrps hasn''t created any new problems.> sharing scheduler credits with the HVM guest is a rather special > case, and > one where a master/slave relationship is not unreasonableI agree. It''s not unreasonable, but it''s also not strictly necessary. I would like to avoid any mechanism that induces a domain hierarchy without need.> If you really want resource sharing amongst a peer group of > domains, what does that actually mean? Does each domain have a private > allocation of resource plus membership of one or more pools? If so, > which > resource set satisfies each resource request by a particular > domain? Even if > not, how is resource contention between domains belonging to the > resource > pool (e.g., for the static assignment of memory pages to that > resource pool) > managed and resolved? The obvious place would be the dom0 control > plane -- > in which case these resource pools can probably be implemented as a > high-level tools abstraction with no direct hypervisor involvement > at all.It might be that, in the case of resource management, the VMM doesn''t have to be involved, but the tool chain would benefit from a common configuration system and API to converse about groups of domains.> My fear is that a nice domgrps infrastructure doesn''t itself > resolve the > hard devils in the details, for all that it sounds architecturally > neat.You''re right, it definitely doesn''t make the devils go away. My goal is more modest. I''m advocating for a common framework where tools and the VMM can use group information in a way that''s relationship agnostic. -Chris _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel