Xen users - Nov 2007 - Running OS instances within domU or on physical device

Hi all,

Something interesting happened the other day. Due to a boot device configuration
mix-up, an RHEL 5 instance, which was intended to run as domU, accidentally got
booted up on the physical hardware. What came as quite a surprise is, that it
actually ran pretty good. I haven''t performed extensive tests, but for
all I could see it run without a hitch.
This made me wonder if there are any (conceptual or proven and tested) arguments
against this. What problems can arise from using an OS instance both on physical
hardware, as well as within a domU? Is there any interesting application of
this? (For one, I can imagine a case, when a non 24x7 VM gets started on a
physical machine in order to cope with periodic heavy loads)

For those who wonder how this could happen, a quick overview: I had a couple of
RHEL 5 domUs, each of them installed on dedicated SAN LUNs. I installed RHEL on
the LUNs using Kickstart scripts, which also installed and configured GRUB,
thereby making them bootable. As it turned out, one of the physical
nodes'' BIOS was configured to boot from SAN. So at its first reboot
after the installation of the domUs, the BIOS booted from the SAN LUN, which
contained the RHEL instance, originally intended as a domU. The rest, as they
say, is history ;-)

Regards,
Gábor

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Nyers, Gabor wrote:
> [...]
> Is there any interesting application of this? (For one, I can imagine a
case, when a non
> 24x7 VM gets started on a physical machine in order to cope with
> periodic heavy loads)
> [...]
Hmm, I see not much useful application in this.

For this type of load balancing, you would rather let the domU run alone
on a single host, and migrate other away to less used servers.

Having to reboot every time the load gets higher seems a bit odd for
real production.

So, this experiment might be an interesting result, but I see not
practical use.
On the other hand, it''s not even such a wonder that this works: if you
installed a normal kernel to boot from, there is not much difference
between a "normal" system and a (PV - HVM is even less different, if
at
all)domU - the only thing different is that you have a xen kernel, and
you often disbale all but one serial console - nothing special that
keeps you from using the system directly on the hardware instead with a
hypervisor in between.

Maybe it''s interesting and usable for desktop/laptop users - I
don''t know?

Henning

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

I''m cc-ing Emre because he''s trying to run domUs on baremetal
too and some of
this will be relevant to him.

I think this ought to work, so I''m glad that it worked for you ;-)

The only problem with this sort of thing is where device names etc change 
(e.g. some newer XenLinux kernels use hvc0 for the console, others use xvc0 
for the console, older ones may use ttyS0 - makes it difficult to configure 
for).  However, given modern distros identify filesystems by their label 
rather than their device, the device naming should be less of a problem.  
You''ll probably need to specify different commandline options to the
kernel
depending on whether it''s a native one or not (e.g. different console=
value,
any Xen-specific or native-specific options).

You also need to add the Xen console device to /etc/securetty(s) and the 
native serial console device also, if you use one.

The main gripe with dual booting between native and Xenified is possibly going 
to be that the hardware detection routines of your distros will get confused 
by (virtual or real) hardware coming and going depending on how you booted.  
I''m not quite sure how they handle this, or how much of a problem
it''d be in
practice.  This is also likely to be an issue for HVM/PV dual booting.

Unfortunately, different niggles are also likely to come up depending on which 
XenLinux you use. :-(

Cheers,
Mrak
> > [...]
> > Is there any interesting application of this? (For one, I can imagine
a
> > case, when a non 24x7 VM gets started on a physical machine in order
to
> > cope with periodic heavy loads)
> > [...]
>
> Hmm, I see not much useful application in this.
>
> For this type of load balancing, you would rather let the domU run alone
> on a single host, and migrate other away to less used servers.
>
> Having to reboot every time the load gets higher seems a bit odd for
> real production.
>
> So, this experiment might be an interesting result, but I see not
> practical use.
> On the other hand, it''s not even such a wonder that this works: if
you
> installed a normal kernel to boot from, there is not much difference
> between a "normal" system and a (PV - HVM is even less different,
if at
> all)domU - the only thing different is that you have a xen kernel, and
> you often disbale all but one serial console - nothing special that
> keeps you from using the system directly on the hardware instead with a
> hypervisor in between.
>
> Maybe it''s interesting and usable for desktop/laptop users - I
don''t know?
>
> Henning
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users


-- 
Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
Mark: To answer a question with a question: What use is a skateboard?
Dave: Skateboards have wheels.
Mark: My wheel has a wheel!

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

See replies inline. 

Regards,
Gábor

-----Original Message-----
From: M.A. Williamson [mailto:maw48@hermes.cam.ac.uk] On Behalf Of Mark
Williamson
Sent: dinsdag 27 november 2007 23:42
To: xen-users@lists.xensource.com
Cc: Henning Sprang; Nyers, Gabor; Emre Erenoglu
Subject: Re: [Xen-users] Running OS instances within domU or on physical device

I''m cc-ing Emre because he''s trying to run domUs on baremetal
too and some of this will be relevant to him.

I think this ought to work, so I''m glad that it worked for you ;-)

The only problem with this sort of thing is where device names etc change 
(e.g. some newer XenLinux kernels use hvc0 for the console, others use xvc0 
for the console, older ones may use ttyS0 - makes it difficult to configure 
for).  However, given modern distros identify filesystems by their label 
rather than their device, the devsice naming should be less of a problem.  
You''ll probably need to specify different commandline options to the
kernel
depending on whether it''s a native one or not (e.g. different console=
value, any Xen-specific or native-specific options).

[G.Nyers> ] That''s exactly what happened in my case. Filesystems
were referenced by labels, and xvc0 was already added in /etc/inittab and
/etc/securetty. So after booting on physical hw, I was immediately able to log
in through ssh. (I haven''t tried the console, but /etc/inittab contains
getty entries for tty devices, so that shouldn''t be a problem)


You also need to add the Xen console device to /etc/securetty(s) and the 
native serial console device also, if you use one.

The main gripe with dual booting between native and Xenified is possibly going 
to be that the hardware detection routines of your distros will get confused 
by (virtual or real) hardware coming and going depending on how you booted.

[G.Nyers> ] Well, I''m not sure "confused" is the right
expression, the HW detection went fine, and the OS configured the HW to its best
abilities.
However, because of the OS can''t possible know it is used in this
peculiar way, there will be some unintended side effects, depending on e.g.
distro etc...:
- Network devices: some distros (eg. SLES or OpenSuSE) like to configure network
devices based on MAC address. That configuration will not be transparent when
booting on physical hw or domU. Nothing a little udev tweaking couldn''t
solve though... Btw. in case of RHEL it worked by chance.
- Multipathing: As I mentioned, filesystems have worked, but when the OS was
booted on physical machine, all filesystems on the SAN LUN were accessed through
a single path.


I''m not quite sure how they handle this, or how much of a problem
it''d be in
practice.  This is also likely to be an issue for HVM/PV dual booting.

Unfortunately, different niggles are also likely to come up depending on which
XenLinux you use. :-(
[G.Nyers> ] Can you give some specific examples?


Cheers,
Mrak

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> From: M.A. Williamson [mailto:maw48@hermes.cam.ac.uk] On Behalf Of Mark
> Williamson Sent: dinsdag 27 november 2007 23:42
> To: xen-users@lists.xensource.com
> Cc: Henning Sprang; Nyers, Gabor; Emre Erenoglu
> Subject: Re: [Xen-users] Running OS instances within domU or on physical
> device
>
> I''m cc-ing Emre because he''s trying to run domUs on
baremetal too and some
> of this will be relevant to him.
>
> I think this ought to work, so I''m glad that it worked for you ;-)
>
> The only problem with this sort of thing is where device names etc change
> (e.g. some newer XenLinux kernels use hvc0 for the console, others use xvc0
> for the console, older ones may use ttyS0 - makes it difficult to configure
> for).  However, given modern distros identify filesystems by their label
> rather than their device, the devsice naming should be less of a problem.
> You''ll probably need to specify different commandline options to
the kernel
> depending on whether it''s a native one or not (e.g. different
console> value, any Xen-specific or native-specific options).
>
> [G.Nyers> ] That''s exactly what happened in my case.
Filesystems were
> referenced by labels, and xvc0 was already added in /etc/inittab and
> /etc/securetty. So after booting on physical hw, I was immediately able to
> log in through ssh. (I haven''t tried the console, but /etc/inittab
contains
> getty entries for tty devices, so that shouldn''t be a problem)
Cool.

Maybe we should make a howto on the wiki with details of what needs to be 
setup to make this work.
> You also need to add the Xen console device to /etc/securetty(s) and the
> native serial console device also, if you use one.
>
> The main gripe with dual booting between native and Xenified is possibly
> going to be that the hardware detection routines of your distros will get
> confused by (virtual or real) hardware coming and going depending on how
> you booted.
>
> [G.Nyers> ] Well, I''m not sure "confused" is the
right expression, the HW
> detection went fine, and the OS configured the HW to its best abilities.
> However, because of the OS can''t possible know it is used in this
peculiar
> way, there will be some unintended side effects, depending on e.g. distro
> etc...: - Network devices: some distros (eg. SLES or OpenSuSE) like to
> configure network devices based on MAC address. That configuration will not
> be transparent when booting on physical hw or domU. Nothing a little udev
> tweaking couldn''t solve though... Btw. in case of RHEL it worked
by chance.
> - Multipathing: As I mentioned, filesystems have worked, but when the OS
> was booted on physical machine, all filesystems on the SAN LUN were
> accessed through a single path.
OK.  I was concerned what the OS would do when seeing devices appear and 
disappear regularly due to virtual / physical switching: I''d expect
this
would be a problem primarily due to annoying boot screens saying your 
hardware is different and due to the risk of losing configuration details.
> I''m not quite sure how they handle this, or how much of a problem
it''d be
> in practice.  This is also likely to be an issue for HVM/PV dual booting.
>
> Unfortunately, different niggles are also likely to come up depending on
> which XenLinux you use. :-( [G.Nyers> ] Can you give some specific
> examples?
First thing that occurs to me: depending on which XenLinux you use the virtual 
serial console can have different names.  On old XenLinux it may default to 
tty1 (or ttyS0 if you set a commandline switch).  On newer XenLinux I think 
it''s xvc0.  On mainline kernel.org Linux with Xen support compiled in
it is
instead hvc0.

Which XenLinux kernel you use could obviously also effect what virtual devices 
are available and get Kudzu (or your favourite hardware detector) all 
agitated.

Adding HVM into the mix, there''s a whole other load of different
devices that
could appear or disappear depending on if you boot native or under HVM.  
Again I''m not sure how Linux distros will react to this.  I seem to
recall
once booting a native Windows install under VMware and it spent about half an 
hour redetecting all the changed hardware ;-)  Modern Linux distros - and 
modern Windows releases, come to that - may cope with this better for all I 
know.  I''d suspect it''d still upset windows though - just a
hunch ;-)

Another thing I''d be quite interested to get working is dual booting
the same
guest under PV and HVM.  I myself would use this functionality for testing 
stuff out.  I expect it''d work already, but it''d be nice to
make it user
friendly as possible.  I''ve been considering what pygrub changes could
make
this easier.

Cheers,
Mark


-- 
Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
Mark: To answer a question with a question: What use is a skateboard?
Dave: Skateboards have wheels.
Mark: My wheel has a wheel!

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users