-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello, weeks ago we moved one x86_32 / breezy badger / xen 3.0.1 machine to our colo for testing purposes. after a remote reboot we lost network connection to the dom0 (interessting, but as read here on the list, other people do have strange networking problems, too). As it was possible to get connection to the domU''s, i scheduled this task because driving to the colo isn''t much fun... Yesterday, I needed to reboot one domU, but this machine didn''t come back. My question, is it possible to investigate this behaviour (and ideally, xm destroy / create) from one of the live domU''s ? I know, this would be a security issue, but is there _any_ access back to the dom0 like the xm console from dom0 to domU''s ? Thanks in advance Stephan Seitz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFEK3ATsU1z66G/Ui4RAtJTAJ9MEHy3xhukRbSDWvtjYM46pctlqwCfV1YZ HS1rXPWKl8GsAqud5wPE92A=pdd5 -----END PGP SIGNATURE----- _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Thu, Mar 30, 2006 at 07:43:47AM +0200, Stephan Seitz wrote:> -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Hello, > > weeks ago we moved one x86_32 / breezy badger / xen 3.0.1 machine to > our colo for testing purposes. after a remote reboot we lost network > connection to the dom0 (interessting, but as read here on the list, > other people do have strange networking problems, too). As it was > possible to get connection to the domU''s, i scheduled this task > because driving to the colo isn''t much fun... > Yesterday, I needed to reboot one domU, but this machine didn''t come back. > > My question, is it possible to investigate this behaviour (and > ideally, xm destroy / create) from one of the live domU''s ? I know, > this would be a security issue, but is there _any_ access back to the > dom0 like the xm console from dom0 to domU''s ?For exactly the reasons you stated (security), the answer is no. -Sean -- Sean Dague IBM Linux Technology Center email: japh@us.ibm.com Open Hypervisor Team alt: sldague@us.ibm.com _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Nils Toedtmann
2006-Mar-30 14:07 UTC
Re: [Xen-users] Re: Access Hypervisor Control from DomU
Am Donnerstag, den 30.03.2006, 08:08 -0500 schrieb Sean Dague:> On Thu, Mar 30, 2006 at 07:43:47AM +0200, Stephan Seitz wrote:[...]> > My question, is it possible to investigate this behaviour (and > > ideally, xm destroy / create) from one of the live domU''s ? I know, > > this would be a security issue, but is there _any_ access back to the > > dom0 like the xm console from dom0 to domU''s ? > > For exactly the reasons you stated (security), the answer is no.I remember reading that the only real difference between a dom0 and a domU kernel is the priviledge to have access to the hypervisor. Why not declaring a special domU to a "fallback" dom0? Not in the sense of having access to hw but control over the hypervisor. That would help if the original dom0 userland dies, but it''s kernel keeps forwarding/bridging packets and blockdevice-I/O, like Stephan''s dom0 did. /nils. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Nils Toedtmann wrote:> Am Donnerstag, den 30.03.2006, 08:08 -0500 schrieb Sean Dague: > >>On Thu, Mar 30, 2006 at 07:43:47AM +0200, Stephan Seitz wrote: > > [...] > >>>My question, is it possible to investigate this behaviour (and >>>ideally, xm destroy / create) from one of the live domU''s ? I know, >>>this would be a security issue, but is there _any_ access back to the >>>dom0 like the xm console from dom0 to domU''s ? >> >>For exactly the reasons you stated (security), the answer is no. > > > I remember reading that the only real difference between a dom0 and a > domU kernel is the priviledge to have access to the hypervisor. Why not > declaring a special domU to a "fallback" dom0? Not in the sense of > having access to hw but control over the hypervisor. > > That would help if the original dom0 userland dies, but it''s kernel > keeps forwarding/bridging packets and blockdevice-I/O, like Stephan''s > dom0 did. > > /nils. >Hi, if this would be a problem you would have to deal with in the real world, you would have a identical box on another location and move the domU''s to it and reboot the problem box. Sincerely, Jan. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Stephan Seitz
2006-Mar-30 19:26 UTC
Re: [Xen-users] Re: Access Hypervisor Control from DomU
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 John Smith schrieb:> Nils Toedtmann wrote: >> >> >> I remember reading that the only real difference between a dom0 >> and a domU kernel is the priviledge to have access to the >> hypervisor. Why not declaring a special domU to a "fallback" >> dom0? Not in the sense of having access to hw but control over >> the hypervisor. >> >> That would help if the original dom0 userland dies, but it''s >> kernel keeps forwarding/bridging packets and blockdevice-I/O, >> like Stephan''s dom0 did. >> >> /nils. >>Thanks for this hint, Nils. I''ll try to setup another test system with two dom0''s, just to see if this is possible. I''ll post results if this leads to a running system.> Hi, > > if this would be a problem you would have to deal with in the real > world, you would have a identical box on another location and move > the domU''s to it and reboot the problem box. > > Sincerely, > > Jan.This was my first thought, but we didn''t setup a fallback system (the one, i talked about, is just for testing purposes installed). Also, this machine has all it''s filesystems on local disk, so a domU migration would last longer than driving to the colo. For live systems, i would also prefer this scenario, but i think this is not managable without the use of a NAS/SAN. Greetings Stephan -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFELDDSsU1z66G/Ui4RAu9EAKCMWclUJReeM5WGRBaS05HrQmK10wCfRtQL RHIjP0k/dy9ZxHaq3LwU4DQ=kTZE -----END PGP SIGNATURE----- _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
John Cianfarani
2006-Mar-31 06:33 UTC
RE: [Xen-users] Re: Access Hypervisor Control from DomU
Look into using heartbeat with DRBD failover would probably be less than a few second hit. Data would also be safe :) This is better than trying to migrate a domU after the box has died... Thanks John -----Original Message----- From: xen-users-bounces@lists.xensource.com [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of Stephan Seitz Sent: Thursday, March 30, 2006 2:26 PM To: XEN User - listmembers Subject: Re: [Xen-users] Re: Access Hypervisor Control from DomU -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 John Smith schrieb:> Nils Toedtmann wrote: >> >> >> I remember reading that the only real difference between a dom0 >> and a domU kernel is the priviledge to have access to the >> hypervisor. Why not declaring a special domU to a "fallback" >> dom0? Not in the sense of having access to hw but control over >> the hypervisor. >> >> That would help if the original dom0 userland dies, but it''s >> kernel keeps forwarding/bridging packets and blockdevice-I/O, >> like Stephan''s dom0 did. >> >> /nils. >>Thanks for this hint, Nils. I''ll try to setup another test system with two dom0''s, just to see if this is possible. I''ll post results if this leads to a running system.> Hi, > > if this would be a problem you would have to deal with in the real > world, you would have a identical box on another location and move > the domU''s to it and reboot the problem box. > > Sincerely, > > Jan.This was my first thought, but we didn''t setup a fallback system (the one, i talked about, is just for testing purposes installed). Also, this machine has all it''s filesystems on local disk, so a domU migration would last longer than driving to the colo. For live systems, i would also prefer this scenario, but i think this is not managable without the use of a NAS/SAN. Greetings Stephan -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFELDDSsU1z66G/Ui4RAu9EAKCMWclUJReeM5WGRBaS05HrQmK10wCfRtQL RHIjP0k/dy9ZxHaq3LwU4DQ=kTZE -----END PGP SIGNATURE----- _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Nils Toedtmann
2006-Mar-31 11:59 UTC
Re: [Xen-users] Re: Access Hypervisor Control from DomU
Am Donnerstag, den 30.03.2006, 18:57 +0200 schrieb John Smith:> Nils Toedtmann wrote: > > Am Donnerstag, den 30.03.2006, 08:08 -0500 schrieb Sean Dague: > > > On Thu, Mar 30, 2006 at 07:43:47AM +0200, Stephan Seitz wrote: > > > > [...] > > > > > > My question, is it possible to investigate this behaviour (and > > > > ideally, xm destroy / create) from one of the live domU''s ? I know, > > > > this would be a security issue, but is there _any_ access back to the > > > > dom0 like the xm console from dom0 to domU''s ? > > > > > > For exactly the reasons you stated (security), the answer is no. > > > > I remember reading that the only real difference between a dom0 and a > > domU kernel is the priviledge to have access to the hypervisor. Why not > > declaring a special domU to a "fallback" dom0? Not in the sense of > > having access to hw but control over the hypervisor. > > > > That would help if the original dom0 userland dies, but it''s kernel > > keeps forwarding/bridging packets and blockdevice-I/O, like Stephan''s > > dom0 did. > > if this would be a problem you would have to deal with in the > real world, you would have a identical box on another location and move > the domU''s to it and reboot the problem box.But that is the exact problem: afaik you cannot move domUs or reboot the "problem box" remotely if dom0 userland is dead! You are locked. Even if you are running all systems redundantly (HA or cluster or whatever) you would like to reboot the stale node remotely. So it would be nice, if you could grant a special domU the right to reboot everything, at least. If you passthrough a second NIC to such a "fallback dom0" and run it entiry from RAM w/o disk you could even recover from a locked dom0 kernel, correct? Or use a watchdog card ... /nils. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Javier Guerra
2006-Mar-31 12:03 UTC
Re: [Xen-users] Re: Access Hypervisor Control from DomU
On Friday 31 March 2006 6:59 am, Nils Toedtmann wrote:> But that is the exact problem: afaik you cannot move domUs or reboot the > "problem box" remotely if dom0 userland is dead! You are locked.i don''t really get how you can get a "dead userland" with a still alive kernel. maybe it''s just the xen daemons that have died? then a DMD (daemon monitoring daemon) would do the trick -- Javier _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Nils Toedtmann
2006-Mar-31 12:57 UTC
Re: [Xen-users] Re: Access Hypervisor Control from DomU
Am Freitag, den 31.03.2006, 07:03 -0500 schrieb Javier Guerra:> On Friday 31 March 2006 6:59 am, Nils Toedtmann wrote: > > But that is the exact problem: afaik you cannot move domUs or reboot the > > "problem box" remotely if dom0 userland is dead! You are locked. > > i don''t really get how you can get a "dead userland" with a still alive > kernel.Several times i experienced a situation where a linux box still forwarded packets and answered echo-requests, but i could not connect to any listening deamon (ssh, http, smtp): The TCP-handshake (SYN - SYN/ACK - ACK) was negotiated, but then the socket did not answer anything, as if all service processes where locked. No idea what really happened, maybe a bug in early 2.6.x. (btw, it was not related to xen)> maybe it''s just the xen daemons that have died? then a DMD (daemon > monitoring daemon) would do the trickThat''s a good idea anyway. /nils. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Stephen C. Tweedie
2006-Apr-04 17:28 UTC
Re: [Xen-users] Re: Access Hypervisor Control from DomU
Hi, On Fri, 2006-03-31 at 13:59 +0200, Nils Toedtmann wrote:> > if this would be a problem you would have to deal with in the > > real world, you would have a identical box on another location and move > > the domU''s to it and reboot the problem box. > > But that is the exact problem: afaik you cannot move domUs or reboot the > "problem box" remotely if dom0 userland is dead! You are locked.Serial console is your friend. :-) Not only does it give you a ttyS0 login to the dom0 even if networking is down; it also gives you a reliable way of capturing log information so you know just how dead the dom0 really is. And it gives you remote reboot too: use ^A^A^A to switch serial console to the hypervisor, and then "R" to reboot the box. Also check out ttywatch, a powerful package for sitting on the other end of a bunch of serial consoles. Of course, with hosted services you might not be able to set this up; but if you can, it''s well worth the effort. --Stephen _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Nils Toedtmann
2006-Apr-04 18:09 UTC
Re: [Xen-users] Re: Access Hypervisor Control from DomU
Moin! Am Dienstag, den 04.04.2006, 13:28 -0400 schrieb Stephen C. Tweedie:> On Fri, 2006-03-31 at 13:59 +0200, Nils Toedtmann wrote: > > > if this would be a problem you would have to deal with in the > > > real world, you would have a identical box on another location and move > > > the domU''s to it and reboot the problem box. > > > > But that is the exact problem: afaik you cannot move domUs or reboot the > > "problem box" remotely if dom0 userland is dead! You are locked. > > Serial console is your friend. :-) > > Not only does it give you a ttyS0 login to the dom0 even if networking > is down; it also gives you a reliable way of capturing log information > so you know just how dead the dom0 really is.[...] True, much better solution! But unfortunately, our situation is like this:> Of course, with hosted services you might not be able to set this upWe have only one colo box. Even if we had two boxes, we would not place them in the same rack (sharing switch, UPS & cooling). Since some ISPs like to screw up their BGB sessions, we maybe would even prefer placing them at different ISPs (using "loose" fallback mechs like rsync) over having Gbit-connection/heartbeat/DRDB. So we need at least four boxes: two HA-pairs, sitting in different BGP-AS'', each HA-pair interconnected via crossover-Gbit (for DRDB) and three serials (on for heartbeat, two for ttyS0 in both directions). Btw, do you have 10k$ you''d like to give away? A "fallback dom0" would be a poor man''s last resort. /nils. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users