Has anyone done any work in developing some heartbeat scripts (or any other method) for making xen domains highly available across multiple physical machines? By my reckoning the hurdles to overcome are: . make absolutely sure that the same domain doesn''t start twice and hose it''s filesystem (I''ve done this. It isn''t pretty) . migrate a domain in the case of an orderly shutdown . restart a domain on another physical server in the case of a complete xen crash . add some smarts to know how many domains a physical server can support . prioritise critical domains over non-critical domains (eg a primary mail server might take priority over a secondary dns server) Any thoughts? James ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_ide95&alloc_id396&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
> >. migrate a domain in the case of an orderly shutdown > > Not sure I understand this one.Think he meant to auto-migrate domains off a physical server when that server is deliberately taken down for some reason.> >. restart a domain on another physical server in the case of a complete > >xen crash> >. add some smarts to know how many domains a physical server can support> >. prioritise critical domains over non-critical domains (eg a primary > >mail server might take priority over a secondary dns server)These are the sort of things I think could usefully go into a higher-level "cluster controller". We identified a few requirements for a cluster controller some time ago but nobody here has had the spare cycles required to implement it. Cheers, Mark ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
James Harper wrote:>Has anyone done any work in developing some heartbeat scripts (or any >other method) for making xen domains highly available across multiple >physical machines? > >By my reckoning the hurdles to overcome are: >. make absolutely sure that the same domain doesn''t start twice and hose >it''s filesystem (I''ve done this. It isn''t pretty) > >This is a feature I''m going to eventually add to vm-tools. It''s a little non-trivial but I don''t think it will be too difficult once the proper architecture is in place.>. migrate a domain in the case of an orderly shutdown > >Not sure I understand this one.>. restart a domain on another physical server in the case of a complete >xen crash > >You mean have redundant copies of Xen and some sort of monitoring capability? If one phys. server stops responding the other starts it''s domains?>. add some smarts to know how many domains a physical server can support > >This is a very interesting problem. I don''t think there''s a right answer. Any thoughts on appopriate heuristics?>. prioritise critical domains over non-critical domains (eg a primary >mail server might take priority over a secondary dns server) > >You can weight CPUs. That''s essentially what you''re looking for right?>Any thoughts? > >James > > >------------------------------------------------------- >SF email is sponsored by - The IT Product Guide >Read honest & candid reviews on hundreds of IT Products from real users. >Discover which products truly live up to the hype. Start reading now. >http://ads.osdn.com/?ad_ide95&alloc_id396&op=click >_______________________________________________ >Xen-devel mailing list >Xen-devel@lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/xen-devel > > >-- Anthony Liguori anthony@codemonkey.ws ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
On Thu, Feb 17, 2005 at 09:04:25PM +1100, James Harper wrote:> Has anyone done any work in developing some heartbeat scripts (or any > other method) for making xen domains highly available across multiple > physical machines?Yes, but basic work, for a course.> By my reckoning the hurdles to overcome are: > . make absolutely sure that the same domain doesn''t start twice and hose > it''s filesystem (I''ve done this. It isn''t pretty)1. xm list | grep domain 2. filesystem shared by drbd, only one can mount it> . migrate a domain in the case of an orderly shutdown1. xm save 2. xm load> . restart a domain on another physical server in the case of a complete > xen crash1. standard heartbeat script and configuration (xen::intsrv xen::www ...)> . add some smarts to know how many domains a physical server can support > . prioritise critical domains over non-critical domains (eg a primary > mail server might take priority over a secondary dns server)Not at the moment. The heartbeat configuration starts the priority servers first, but a failure starting one causes heartbeat to fail the takeover process.> Any thoughts?On my agenda: 1. export devices (ex, via gnbd) 2. leave heartbeat. create a monitoring daemon on each member with support for dynamic migration and startup due to load on the machines. Regards, Luciano Rocha ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel
>> >. migrate a domain in the case of an orderly shutdown >> >> Not sure I understand this one.>Think he meant to auto-migrate domains off a physical server when thatserver is deliberately taken >down for some reason.> >. restart a domain on another physical server in the case of a > >complete xen crash> >. add some smarts to know how many domains a physical server can > >support> >. prioritise critical domains over non-critical domains (eg a primary> >mail server might take priority over a secondary dns server)>These are the sort of things I think could usefully go into ahigher-level "cluster controller". We >identified a few requirements for a cluster controller some time ago but nobody here has had the>spare cycles required to implement it.I implemented this exact scenario using drbd, heartbeat and tripwire. Drbd handles the replication of domU metadata to the secondary physical server. Heartbeat checks that the primary is ''alive'' and handles resource takeovers in a failover event. Tripwire performs periodic tests on the dom0 and will force migration if a serious error occurs. The point is, this is all doable using already written and more importantly mature tools that have done their time in the trenches. If we are going to bother implementing this sort of thing at the Xen level then we had better make it better than whats already there, or else we are wasting our time. Tom ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_ide95&alloc_id396&op=click _______________________________________________ Xen-devel mailing list Xen-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xen-devel