Hi, I''ve added a draft of a doc called "the dom0 capacity crunch" to the wiki: http://wiki.xensource.com/xenwiki/XCP_Overview?action=AttachFile&do=get&target=xcp_dom0_capacity_crunch.pdf The doc contains a proposal for dealing with ever-increasing dom0 load, primarily by moving the load out of dom0 (stubdoms, helper domains etc) and, where still necessary, tweaking the number of dom0 vcpus. I think this is becoming pretty important and we''ll need to work on this asap. Comments are welcome. Cheers, Dave _______________________________________________ xen-api mailing list xen-api@lists.xensource.com http://lists.xensource.com/mailman/listinfo/xen-api
On Fri, Oct 08, 2010 at 08:54:25PM +0100, Dave Scott wrote:> Hi, > > I''ve added a draft of a doc called "the dom0 capacity crunch" to the wiki: > > http://wiki.xensource.com/xenwiki/XCP_Overview?action=AttachFile&do=get&target=xcp_dom0_capacity_crunch.pdf > > The doc contains a proposal for dealing with ever-increasing dom0 load, primarily by moving the load out of dom0 (stubdoms, helper domains etc) and, where still necessary, tweaking the number of dom0 vcpus. I think this is becoming pretty important and we''ll need to work on this asap. > > Comments are welcome. >All that makes sense to me. I think oxenstored already has (some) support for running in stubdom, qemu-dm already supports stubdoms, driver-domains should work.. This stuff will require a lof of testing.. -- Pasi _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Thomas.Gazagnaire@inria.fr
2010-Oct-08 22:45 UTC
Re: [Xen-API] Re: [Xen-devel] XCP: dom0 scalability
> I think oxenstored already has (some) support for running in stubdom,the difficult part for oxenstored if I remember correctly is the way xapi communicates with it, ie. using fds. Don''t know if it is easy or not to replace them with ring buffers ... But Vincent should know :-)> qemu-dm already supports stubdoms, driver-domains should work.. > > This stuff will require a lof of testing.. > > -- Pasi > > > _______________________________________________ > xen-api mailing list > xen-api@lists.xensource.com > http://lists.xensource.com/mailman/listinfo/xen-api >_______________________________________________ xen-api mailing list xen-api@lists.xensource.com http://lists.xensource.com/mailman/listinfo/xen-api
This seems rational, but what about overhead? Every domain will require own kernel with memory (main concern about memory). I think, first step shall be splitting xapi to few separate daemons. В Птн, 08/10/2010 в 20:54 +0100, Dave Scott пишет:> Hi, > > I''ve added a draft of a doc called "the dom0 capacity crunch" to the wiki: > > http://wiki.xensource.com/xenwiki/XCP_Overview?action=AttachFile&do=get&target=xcp_dom0_capacity_crunch.pdf > > The doc contains a proposal for dealing with ever-increasing dom0 load, primarily by moving the load out of dom0 (stubdoms, helper domains etc) and, where still necessary, tweaking the number of dom0 vcpus. I think this is becoming pretty important and we''ll need to work on this asap. > > Comments are welcome. > > Cheers, > Dave > > _______________________________________________ > xen-api mailing list > xen-api@lists.xensource.com > http://lists.xensource.com/mailman/listinfo/xen-api_______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
Hi Thomas, Pasi, Thomas wrote:> I think oxenstored already has (some) support for running in stubdom, > the difficult part for oxenstored if I remember correctly is the way > xapi > communicates with it, ie. using fds. Don't know if it is easy or not to > replace them with ring buffers ... But Vincent should know :-)IIRC there's a function 'Xc.daemon_open' which uses a Unix.file_descr and a function 'Xc.domain_open' which uses the ring buffers. I think we'd just need to redefine our 'with_xs' helper function to use 'Xc.domain_open'? Pasi wrote:> > qemu-dm already supports stubdoms, driver-domains should work.. > > > > This stuff will require a lof of testing..It'll be a lot of fun :) Cheers, Dave _______________________________________________ xen-api mailing list xen-api@lists.xensource.com http://lists.xensource.com/mailman/listinfo/xen-api
On 10/08/2010 12:54 PM, Dave Scott wrote:> I''ve added a draft of a doc called "the dom0 capacity crunch" to the wiki: > > http://wiki.xensource.com/xenwiki/XCP_Overview?action=AttachFile&do=get&target=xcp_dom0_capacity_crunch.pdf > > The doc contains a proposal for dealing with ever-increasing dom0 load, primarily by moving the load out of dom0 (stubdoms, helper domains etc) and, where still necessary, tweaking the number of dom0 vcpus. I think this is becoming pretty important and we''ll need to work on this asap.I''m a big fan of disaggregation, but from a performance perspective xenstored seems pretty insignificant. J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
At 00:16 +0100 on 13 Oct (1286928975), Jeremy Fitzhardinge wrote:> I''m a big fan of disaggregation, but from a performance perspective > xenstored seems pretty insignificant.True, xenstored is more a victim of dom0 load than a cause of it. We''ve seen some pretty nasty performance when trying to boot lots of VMs with a loaded dom0; PV drivers in the VMs could block for many seconds waiting for xenstore to get back to them. It still makes sense to get it out of there. Cheers, Tim. -- Tim Deegan <Tim.Deegan@citrix.com> Principal Software Engineer, XenServer Engineering Citrix Systems UK Ltd. (Company #02937203, SL9 0BG) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
> At 00:16 +0100 on 13 Oct (1286928975), Jeremy Fitzhardinge wrote: > > I''m a big fan of disaggregation, but from a performance perspective > > xenstored seems pretty insignificant. > > True, xenstored is more a victim of dom0 load than a cause of it. > We''ve > seen some pretty nasty performance when trying to boot lots of VMs with > a loaded dom0; PV drivers in the VMs could block for many seconds > waiting for xenstore to get back to them. It still makes sense to get > it out of there.Hmmm... if dom0 is getting scheduled enough and the VMs are still blocking on xenstore (i.e. the Linux scheduler in dom0 is the problem), getting xenstore off of dom0 makes sense. But if the reason xenstore is not getting back to them is because dom0 is simply not getting enough cpu time (i.e. the Xen scheduler is the problem), moving xenstore out of there may not only NOT help, but may make things worse. True? _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
At 16:46 +0100 on 13 Oct (1286988381), Dan Magenheimer wrote:> Hmmm... if dom0 is getting scheduled enough and the VMs are still > blocking on xenstore (i.e. the Linux scheduler in dom0 is the problem), > getting xenstore off of dom0 makes sense. But if the reason xenstore > is not getting back to them is because dom0 is simply not getting > enough cpu time (i.e. the Xen scheduler is the problem), moving xenstore > out of there may not only NOT help, but may make things worse. True?The reason that dom0''s not getting enough time in this case is that it''s running with only 1 vcpu and trying to do more than 1 vcpu worth of work. I''d expect things to get very slightly better if some of that work can happen on another vpcu (in another dom). Tim. -- Tim Deegan <Tim.Deegan@citrix.com> Principal Software Engineer, XenServer Engineering Citrix Systems UK Ltd. (Company #02937203, SL9 0BG) _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel
On 10/13/2010 09:07 AM, Tim Deegan wrote:> At 16:46 +0100 on 13 Oct (1286988381), Dan Magenheimer wrote: >> Hmmm... if dom0 is getting scheduled enough and the VMs are still >> blocking on xenstore (i.e. the Linux scheduler in dom0 is the problem), >> getting xenstore off of dom0 makes sense. But if the reason xenstore >> is not getting back to them is because dom0 is simply not getting >> enough cpu time (i.e. the Xen scheduler is the problem), moving xenstore >> out of there may not only NOT help, but may make things worse. True? > The reason that dom0''s not getting enough time in this case is that it''s > running with only 1 vcpu and trying to do more than 1 vcpu worth of > work. I''d expect things to get very slightly better if some of that > work can happen on another vpcu (in another dom).Or just make dom0 multi-vcpu? J _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel