Xen devel - Apr 2005 - [PATCH] Guest boot loader support [1/2]

This first patch adds support to xend and xm to run an executable to use
as a bootloader for passing back the kernel, initrd and kernel arguments
to use for the domain.

Signed-off-by: Jeremy Katz <katzj@redhat.com>

Jeremy


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Jeremy Katz wrote:
> This first patch adds support to xend and xm to run an executable to use
> as a bootloader for passing back the kernel, initrd and kernel arguments
> to use for the domain.
> 
> Signed-off-by: Jeremy Katz <katzj@redhat.com>
> 
> Jeremy
Thanks for the patch. There are some issues that need sorting out before
we can apply it. Headlines here, some more detail in-line below.

- the code only supports booting with the bootloader when called from xm,
   since xm is running the loader. For general use it needs to support
configuring
   the bootloader in the domain config and have xend run it.

- need to be able to configure the kernel to boot in the domain config
   when not using a console.

- need to remove calls to exit from xend-called code
- replace prints to stderr with logging

- handle errors when calling external scripts

- it would be better to be able to remove the assumption that the first
   device is the disk to boot from

Feel free to ask if you need any more details or more info.

Mike
> 
> 
> ------------------------------------------------------------------------
> 
> # This is a BitKeeper generated diff -Nru style patch.
> #
> # ChangeSet
> #   2005/04/13 23:01:42-04:00 katzj@bree.local.net 
> #   Add framework for running a bootloader from within xend and xm. To
> #   specify the executable to run, set in your domain config like
> #     bootloader="/usr/bin/pygrub"
> #   
> #   The bootloader will get executed on both domain creation and domain
> #   reboot to get the default kernel/initrd specified in the bootloader
> #   config for the domain.  If you auto-connect the console on domain
> #   creation, then your bootloader will be run in an interactive mode.
> # 
> # BitKeeper/etc/logging_ok
> #   2005/04/13 23:01:42-04:00 katzj@bree.local.net +1 -0
> #   Logging to logging@openlogging.org accepted
> # 
> # tools/python/xen/xend/XendBootloader.py
> #   2005/04/13 23:01:37-04:00 katzj@bree.local.net +79 -0
> # 
> # tools/python/xen/xm/create.py
> #   2005/04/13 23:01:37-04:00 katzj@bree.local.net +29 -2
> #   Run bootloader on domain creation.  If console is being connected,
> #   then run in an interactive mode, else just go with the default.
> # 
> # tools/python/xen/xend/XendDomainInfo.py
> #   2005/04/13 23:01:37-04:00 katzj@bree.local.net +41 -1
> #   Add running of a boot loader from xend on domain reboot.  Runs in
> #   an uninteractive mode to get the default kernel/initrd for the
> #   domain.  Also removes any temporary kernels created by the 
> #   bootloader.
> # 
> # tools/python/xen/xend/XendBootloader.py
> #   2005/04/13 23:01:37-04:00 katzj@bree.local.net +0 -0
> #   BitKeeper file
/home/katzj/cvs/xen/xen-pygrub/tools/python/xen/xend/XendBootloader.py
> # 
> diff -Nru a/tools/python/xen/xend/XendBootloader.py
b/tools/python/xen/xend/XendBootloader.py
> --- /dev/null	Wed Dec 31 16:00:00 196900
> +++ b/tools/python/xen/xend/XendBootloader.py	2005-04-13 23:04:06 -04:00
> @@ -0,0 +1,79 @@
> +#
> +# XendBootloader.py - Framework to run a boot loader for picking the
kernel
> +#
> +# Copyright 2005 Red Hat, Inc.
> +# Jeremy Katz <katzj@redhat.com>
> +#
> +# This software may be freely redistributed under the terms of the GNU
> +# general public license.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write to the Free Software
> +# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
> +#
> +
> +import os, sys, select
> +import sxp
> +
> +BL_FIFO = "/var/lib/xen/xenbl"
> +
> +def bootloader(blexec, disk, quiet = 0, vcpus = None):
> +    """Run the boot loader executable on the given disk and
return a
> +    config image.
> +    @param blexec  Binary to use as the boot loader
> +    @param disk Disk to run the boot loader on."""
> +    
> +    if not os.access(blexec, os.X_OK):
> +        print >> sys.stderr, "Bootloader isn''t
executable"
> +        sys.exit(1)
Use logging instead of stderr. Don''t call exit, raise a XendError.
> +    if not os.access(disk, os.R_OK):
> +        print >> sys.stderr, "Disk isn''t
accessible"
> +        sys.exit(1)
> +
The code below looks like it''s reading data back from the child process
-
Python has the popen2 module to do this for you. Maybe use that instead?
Using a constant name for the fifo means only one instance of the code
can run at once - which could be a problem.

The sxpr parser can be called incrementally as you read instead
of building up the whole string.
> +    os.mkfifo(BL_FIFO, 0600)
> +
> +    child = os.fork()
> +    if (not child):
> +        args = [ blexec, "-o", BL_FIFO ]
> +        if quiet:
> +            args.append("-q")
> +        args.append(disk)
> +
> +        try:
> +            os.execvp(args[0], args)
> +        except OSError, e:
> +            print e
> +            pass
> +        os._exit(1)
> +
> +    while 1:
> +        try:
> +            r = os.open(BL_FIFO, os.O_RDONLY)
> +        except OSError, e:
> +            if e.errno == 4:
> +                continue
> +        break
> +    ret = ""
> +    while 1:
> +        select.select([r], [], [])
> +        s = os.read(r, 1024)
> +        ret = ret + s
> +        if len(s) == 0:
> +            break
> +        
> +    (pid, status) = os.waitpid(child, 0)
> +    os.close(r)
> +    os.unlink(BL_FIFO)
> +
> +    pin = sxp.Parser()
> +    pin.input(ret)
> +    pin.input_eof()
> +
> +    config_image = pin.val
> +    if vcpus and sxp.child_value(config_image, "vcpus") is None:
> +        config_image.append([''vcpus'', vcpus])
> +
> +    config = [[''image'', pin.val]]
> +    config.append([''bootloader'', blexec])
> +    return config
> +
> diff -Nru a/tools/python/xen/xend/XendDomainInfo.py
b/tools/python/xen/xend/XendDomainInfo.py
> --- a/tools/python/xen/xend/XendDomainInfo.py	2005-04-13 23:04:06 -04:00
> +++ b/tools/python/xen/xend/XendDomainInfo.py	2005-04-13 23:04:06 -04:00
> @@ -21,6 +21,7 @@
>  import xen.util.ip
>  from xen.util.ip import _readline, _readlines
>  from xen.xend.server import channel
> +from xen.xend.XendBootloader import bootloader
>  
>  import sxp
>  
> @@ -323,6 +324,7 @@
>          self.image_handler = None
>          self.is_vmx = 0
>          self.vcpus = 1
> +        self.bootloader = None
>  
>      def setdom(self, dom):
>          """Set the domain id.
> @@ -458,6 +460,7 @@
>  
>              self.find_image_handler()
>              self.init_domain()
> +            self.configure_bootloader()
>              self.configure_console()
>              self.configure_backends()
>              self.construct_image()
> @@ -795,7 +798,7 @@
>                        	ramdisk        = ramdisk,
>                        	flags          = flags)
>  	else:
> -        	log.warning(''building dom with %d vcpus'',
self.vcpus)
> +        	log.warning(''building dom with %d vcpus''
%(self.vcpus,))
There''s no need to use % here - log.warning does the formatting for
you.

>          	err = buildfn(dom            = dom,
>                 	      	image          = kernel,
>                        	control_evtchn = self.console.getRemotePort(),
> @@ -803,6 +806,14 @@
>                        	ramdisk        = ramdisk,
>                        	flags          = flags,
>                        	vcpus          = self.vcpus)
> +
> +        if self.bootloader:
> +            try:
> +                if kernel: os.unlink(kernel)
> +                if ramdisk: os.unlink(ramdisk)
> +            except OSError, e:
> +                log.warning(''unable to unlink kernel/ramdisk:
%s'' %(e,))
> +       
>          if err != 0:
>              raise VmError(''Building domain failed: type=%s dom=%d
err=%d''
>                            % (ostype, dom, err))
> @@ -961,6 +972,13 @@
>              maxmem = self.memory
>          xc.domain_setmaxmem(self.dom, maxmem_kb = maxmem * 1024)
>  
> +    def configure_bootloader(self):
> +        """Configure boot loader.
> +        """
> +        bl = sxp.child_value(self.config, "bootloader")
> +        if bl is not None:
> +            self.bootloader = bl
> +
>      def configure_console(self):
>          """Configure the vm console port.
>          """
> @@ -1034,6 +1052,26 @@
>          try:
>              self.restart_check()
>              self.restart_state = STATE_RESTART_BOOTING
> +            # if we''re restarting with a bootloader, we need to
run it
> +            if self.bootloader:
> +                # FIXME: this assumes the disk is the first device and
> +                # that we''re booting from the first disk
> +                blcfg = None
> +                
> +                dev = sxp.child_value(self.config, "device")
> +                if dev:
> +                    log.info("dev is %s" %(dev,))
> +                    disk = sxp.child_value(dev, "uname")
> +                    (type, fn) = disk.split(":")
> +                    if type == "phy" and not
fn.startswith("/dev/"):
> +                        fn = "/dev/%s" %(fn,)

There are functions to manipulate the block device name in xend/server/blkif.py.
Probably better to use one of those.
Not keen on ''%(fn,)'' - why not use ''% fn''?
> +                    blcfg = bootloader(self.bootloader, fn, 1, self.vcpus)
> +
> +                if blcfg is None:
> +                    log.warning("Had a boot loader, but
can''t find the disk")
Shouldn''t this be a fatal error?
> +                else:
> +                    s = "(vm %s)" %(sxp.to_string(blcfg[0]),)
> +                    self.config = sxp.merge(sxp.from_string(s),
self.config)
If blcfg[0] is already a list, no need to convert to string and back just to
append.
Better to do

s = [ sxp.name(self.config) ] + blcfg[0]
sxp.merge(s, self.config)
>              d = self.construct(self.config)
>          finally:
>              self.restart_state = None
> @@ -1163,6 +1201,7 @@
>      if args:
>          cmdline += " " + args
>      ramdisk = sxp.child_value(image, "ramdisk",
'''')
> +    log.debug("creating linux domain with cmdline: %s"
%(cmdline,))
>      vm.create_domain("linux", kernel, ramdisk, cmdline)
>      return vm
>  
> @@ -1377,6 +1416,7 @@
>  add_config_handler(''device'',     vm_field_ignore)
>  add_config_handler(''backend'',    vm_field_ignore)
>  add_config_handler(''vcpus'',      vm_field_ignore)
> +add_config_handler(''bootloader'', vm_field_ignore)
>  
>  # Register other config handlers.
>  add_config_handler(''maxmem'',     vm_field_maxmem)
> diff -Nru a/tools/python/xen/xm/create.py b/tools/python/xen/xm/create.py
> --- a/tools/python/xen/xm/create.py	2005-04-13 23:04:06 -04:00
> +++ b/tools/python/xen/xm/create.py	2005-04-13 23:04:06 -04:00
> @@ -10,6 +10,7 @@
>  from xen.xend import sxp
>  from xen.xend import PrettyPrint
>  from xen.xend.XendClient import server, XendError
> +from xen.xend.XendBootloader import bootloader
>  
>  from xen.util import console_client
>  
> @@ -94,6 +95,10 @@
>            fn=set_value, default=None,
>            use="Domain name. Must be unique.")
>  
> +gopts.var(''bootloader'', val=''FILE'',
> +          fn=set_value, default=None,
> +          use="Path to bootloader.")
> +
>  gopts.var(''kernel'', val=''FILE'',
>            fn=set_value, default=None,
>            use="Path to kernel image.")
> @@ -375,6 +380,23 @@
>      config_devs.append([''device_model'', device_model])
>      config_devs.append([''device_config'', device_config])
>  
> +def run_bootloader(config, vals):
> +    if not os.access(vals.bootloader, os.X_OK):
> +        print >> sys.stderr, "Bootloader isn''t
executable"
> +        sys.exit(1)
> +    if len(vals.disk) < 1:
> +        print >> sys.stderr, "No disks configured and boot
loader requested"
> +        sys.exit(1)
> +    (uname, dev, mode, backend) = vals.disk[0]
> +    (typ, file) = uname.split(":")
> +    if typ == "phy" and not file.startswith("/dev/"):
> +        file = "/dev/%s" %(file,)
> +
> +    blcfg = bootloader(vals.bootloader, file,
> +                       not vals.console_autoconnect, vals.vcpus)
> +
> +    config.extend(blcfg)
This is a repeat of the reboot code in xend - and initial boot needs
to be in xend too.
> +
>  def make_config(vals):
>      """Create the domain configuration.
>      """
> @@ -396,8 +418,11 @@
>          config.append([''restart'', vals.restart])
>      if vals.console:
>          config.append([''console'', vals.console])
> -    
> -    configure_image(config, vals)
> +
> +    if vals.bootloader:
> +        run_bootloader(config, vals)
> +    else:
> +        configure_image(config, vals)
>      config_devs = []
>      configure_disks(config_devs, vals)
>      configure_pci(config_devs, vals)
> @@ -405,6 +430,7 @@
>      configure_usb(config_devs, vals)
>      configure_vmx(config_devs, vals)
>      config += config_devs
> +
>      return config
>  
>  def preprocess_disk(opts, vals):
> @@ -588,6 +614,7 @@
>          if not opts.getopt(''name'') and
opts.getopt(''defconfig''):
>              opts.setopt(''name'',
os.path.basename(opts.getopt(''defconfig'')))
>          config = make_config(opts.vals)
> +
>      if opts.vals.dryrun:
>          PrettyPrint.prettyprint(config)
>      else:
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Thu, 2005-04-14 at 13:20 +0100, Mike Wray wrote:
> Jeremy Katz wrote:
> > This first patch adds support to xend and xm to run an executable to
use
> > as a bootloader for passing back the kernel, initrd and kernel
arguments
> > to use for the domain.
> > 
> > Signed-off-by: Jeremy Katz <katzj@redhat.com>
> Thanks for the patch. There are some issues that need sorting out before
> we can apply it. Headlines here, some more detail in-line below.
Thanks for looking over it.
> - the code only supports booting with the bootloader when called from xm,
>    since xm is running the loader. For general use it needs to support
configuring
>    the bootloader in the domain config and have xend run it.
It also runs from xend on the domain reboot case.  Right now only in
non-interactive/quiet mode since finding where the console is at that
point looked treacherous.  See the changes to XendDomainInfo.restart.
> - need to be able to configure the kernel to boot in the domain config
>    when not using a console.
Also handled, if the console isn''t connected it runs with quiet mode
which should just pick the default.
> - need to remove calls to exit from xend-called code
> - replace prints to stderr with logging
I thought I had got all of these (from the xend path), but I''ll look
again once I get in to the office.  There is some printing/exiting from
the xm side since if you can''t find a kernel there, having xm exit with
an error message gives the user some idea of what''s going on.
> - handle errors when calling external scripts
Should be easy enough to add a little bit more here.
> - it would be better to be able to remove the assumption that the first
>    device is the disk to boot from
The problem is that if you don''t assume the first device is the boot
disk, you essentially have to add a new configuration directive for
specifying what it is.  I''m fine with adding one, but I was just trying
to minimize the number of config changes needed.  I guess it''s
reasonable to add a configuration option and default to the first disk
if it''s not there.  I''ll also do this once I get into the
office.
> > +        print >> sys.stderr, "Bootloader isn''t
executable"
> > +        sys.exit(1)
> 
> Use logging instead of stderr. Don''t call exit, raise a XendError.
Yeah, artifact of code moving around as I wrote it (and not having
logging where it originally was).  Will switch.
> > +    if not os.access(disk, os.R_OK):
> > +        print >> sys.stderr, "Disk isn''t
accessible"
> > +        sys.exit(1)
> > +
> 
> The code below looks like it''s reading data back from the child
process -
> Python has the popen2 module to do this for you. Maybe use that instead?
> Using a constant name for the fifo means only one instance of the code
> can run at once - which could be a problem.
The problem with using popen is that unless I''ve missed a way of using
it, you replace stdout/stderr for the process you''re exec''ing.
Which
isn''t what you want if you''re going to run an interactive
curses
program.
> The sxpr parser can be called incrementally as you read instead
> of building up the whole string.
Sure, but the returned sxp will be small so it shouldn''t make a big
difference one way or another.  
> > +                    disk = sxp.child_value(dev, "uname")
> > +                    (type, fn) = disk.split(":")
> > +                    if type == "phy" and not
fn.startswith("/dev/"):
> > +                        fn = "/dev/%s" %(fn,)
> 
> There are functions to manipulate the block device name in
xend/server/blkif.py.
> Probably better to use one of those.
Yes, yes it is :)
> Not keen on ''%(fn,)'' - why not use ''%
fn''?
%(fn,) is far more explicit and ensures you don''t get bit if something
becomes a tuple when you least expect it.  After hitting problems once
or twice, I''ve gotten to where I''m very pedantic about always
ensuring I
specify format strings in this fashion.
> > +                    blcfg = bootloader(self.bootloader, fn, 1,
self.vcpus)
> > +
> > +                if blcfg is None:
> > +                    log.warning("Had a boot loader, but
can''t find the disk")
> 
> Shouldn''t this be a fatal error?
Yep.
> > +                else:
> > +                    s = "(vm %s)"
%(sxp.to_string(blcfg[0]),)
> > +                    self.config = sxp.merge(sxp.from_string(s),
self.config)
> 
> If blcfg[0] is already a list, no need to convert to string and back just
to append.
> Better to do
> 
> s = [ sxp.name(self.config) ] + blcfg[0]
> sxp.merge(s, self.config)
I vaguely remember hitting problems trying to do this, but it could well
have been PEBKAC.  Will try.
> This is a repeat of the reboot code in xend - and initial boot needs
> to be in xend too.
Is there a way to kick off a full initial boot within xend without using
xm?  I saw only reboots.

Thanks for the review, will make the cleanups suggested this morning and
get out a second pass.

Jeremy


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Attached patch replaces the previous one fixing things as described
below.

Signed-off-by: Jeremy Katz <katzj@redhat.com

On Thu, 2005-04-14 at 13:20 +0100, Mike Wray wrote:
> - need to remove calls to exit from xend-called code
> - replace prints to stderr with logging
> - handle errors when calling external scripts
Error handling cleaned up and better use of logging instead of just
printing things.
> - it would be better to be able to remove the assumption that the first
>    device is the disk to boot from
Left this for the first pass.  If there''s some agreement on my previous
suggestion of how to handle this, then that can easily be implemented on
top without really needing to be done to get the first pass in.  I think
a lot of uses right now are really only one vbd anyway.
> There are functions to manipulate the block device name in
xend/server/blkif.py.
> Probably better to use one of those.
I didn''t actually find one that did what was needed for take the uname
to a filename, so I added one and use it both places I had the code.

Jeremy


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> Thanks for the patch. There are some issues that need sorting 
> out before we can apply it. Headlines here, some more detail 
> in-line below.
> 
> - the code only supports booting with the bootloader when 
> called from xm,
>    since xm is running the loader. For general use it needs 
> to support configuring
>    the bootloader in the domain config and have xend run it.
Yep, it would be much better if we could hide this all in xend. Ideally,
it should be possible to start a domain, then connect to its console
within the bootloader''s timeout and be able to interact with it.
 
> - need to be able to configure the kernel to boot in the domain config
>    when not using a console.
This would certainly be nice.
> - it would be better to be able to remove the assumption that 
> the first
>    device is the disk to boot from
I could live with this. Crap PC BIOSes have conditioned everyone to
expect oddities like this :-)


Cheers,
Ian

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Rumor has it that on Thu, Apr 14, 2005 at 06:54:50PM +0100 Ian Pratt
said:
> > - it would be better to be able to remove the assumption that 
> > the first
> >    device is the disk to boot from
> 
> I could live with this. Crap PC BIOSes have conditioned everyone to
> expect oddities like this :-)
> 
Besides, in a virtualized environment this assumption is
not hard to live up to. 

As a slight but related digression, has any thought been given to
using something other than /dev/[hs]d* names for specifying block
devices? The name /dev/sdb only has meaning for the current
state of the running dom0 OS. It may not mean the same device
on a different dom0. This can lead to problems in say migration.
In fact, it may not be the same when the domain is restored locally.

Having to configure all the possible migration targets to see all 
the same devices in the same order at all times is not realistic.
In a dynamic environment it would be ideal to make a device visible
to the target dom0 only as it was needed to migrate a given VM to 
that dom0. Naming it /dev/sdb may not be a possiblity in this case.


Cheers,

Phil

> 
> Cheers,
> Ian
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
-- 
Philip R. Auld, Ph.D.  	        	       Egenera, Inc.    
Software Architect                            165 Forest St.
(508) 858-2628                            Marlboro, MA 01752

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Thu, 2005-04-14 at 18:54 +0100, Ian Pratt wrote:
>  > Thanks for the patch. There are some issues that need sorting 
> > out before we can apply it. Headlines here, some more detail 
> > in-line below.
> > 
> > - the code only supports booting with the bootloader when 
> > called from xm,
> >    since xm is running the loader. For general use it needs 
> > to support configuring
> >    the bootloader in the domain config and have xend run it.
> 
> Yep, it would be much better if we could hide this all in xend. Ideally,
> it should be possible to start a domain, then connect to its console
> within the bootloader''s timeout and be able to interact with it.
If the console code is changing around, though, hooking up to the
current stuff might be more trouble than its worth.  Would be better in
the long run, though.
 
> > - need to be able to configure the kernel to boot in the domain config
> >    when not using a console.
> 
> This would certainly be nice.
Already there, just change your default in the config.
> > - it would be better to be able to remove the assumption that 
> > the first
> >    device is the disk to boot from
> 
> I could live with this. Crap PC BIOSes have conditioned everyone to
> expect oddities like this :-)
Since we''re not on real hardware, we should be able to do better ;) 
But
I''m of the opinion that incremental steps are the way to go.

Jeremy


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> As a slight but related digression, has any thought been given to
> using something other than /dev/[hs]d* names for specifying block
> devices? The name /dev/sdb only has meaning for the current
> state of the running dom0 OS. It may not mean the same device
> on a different dom0. This can lead to problems in say migration.
> In fact, it may not be the same when the domain is restored locally.
You could work round this currently using the block config scripts.  I.e. 
similar to the file: syntax now, you could easily hack up nbd:, iscsi:, etc.  
Then network block devices can be connected up at create / migrate / restore 
time, without relying on a particular device node.

This can be done by writing a shell script to bind the device to a node and 
adding a config parameter to Xend.

I''d imagine we can rely on actual physical device numbers generally
staying
the same but this could be fixed by a similar mechanism.

Cheers,
Mark
> Having to configure all the possible migration targets to see all
> the same devices in the same order at all times is not realistic.
> In a dynamic environment it would be ideal to make a device visible
> to the target dom0 only as it was needed to migrate a given VM to
> that dom0. Naming it /dev/sdb may not be a possiblity in this case.
>
>
> Cheers,
>
> Phil
>
> > Cheers,
> > Ian
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> > Yep, it would be much better if we could hide this all in xend. 
> > Ideally, it should be possible to start a domain, then 
> connect to its 
> > console within the bootloader''s timeout and be able to 
> interact with it.
> 
> If the console code is changing around, though, hooking up to 
> the current stuff might be more trouble than its worth.  
> Would be better in the long run, though.
I think Rik may have volunteered to take a look at the console stuff :-)

I actually think this is worth implementing right now, as it sets a key
requirement for the new console stuff -- it must be possible to steal a
guest console and inject a bootloader session. We might want something
similar in future for a gdb session too.
  
> > > - need to be able to configure the kernel to boot in the 
> domain config
> > >    when not using a console.
> > 
> > This would certainly be nice.
> 
> Already there, just change your default in the config.
Just to make sure, you mean the VM config, not the grub config, right?
Do you specify the label name, or the crappy grub menu number?

Ian

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Thu, 14 Apr 2005, Ian Pratt wrote:
> I think Rik may have volunteered to take a look at the console stuff :-)
NOOOOOOOOOOOOOOOOOOOOOOO !!!

*runs away screaming, leaving a Rik-shaped hole in the wall*

(I volunteered to look at some of the memory management and
 upstream merging bits - which I will look at as soon as the
 xenU kernel works again in rawhide)

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi Mark,

Rumor has it that on Thu, Apr 14, 2005 at 08:11:18PM +0100 Mark Williamson
said:
> > As a slight but related digression, has any thought been given to
> > using something other than /dev/[hs]d* names for specifying block
> > devices? The name /dev/sdb only has meaning for the current
> > state of the running dom0 OS. It may not mean the same device
> > on a different dom0. This can lead to problems in say migration.
> > In fact, it may not be the same when the domain is restored locally.
> 
> You could work round this currently using the block config scripts.  I.e. 
> similar to the file: syntax now, you could easily hack up nbd:, iscsi:,
etc.
> Then network block devices can be connected up at create / migrate /
restore
> time, without relying on a particular device node.
Thanks. I''m not really looking for a work around though :) I think 
this is an issue that we should look at in more depth. 

My concern is for SAN attached devices primarily.  These will
show up as scsi devices with /dev/sdXX names (barring custom 
udev configs). 

BTW, how would you do this with iSCSI? iSCSI target LUNs show 
up as scsi devices and get block device names. It''ll have the same
issues as FC SAN. 

I suppose though that if xen doesn''t care about the actual names. Then
making a link in /dev/ could work. Maybe this is not really an issue after 
all.  e.g. /dev/my_domU_1_disk -> /dev/sdb. Then as long as something
on the other end can tell where the disk showed up it can make the
appropriate link -- /dev/my_domU_1_disk -> /dev/sdq.
 
> 
> This can be done by writing a shell script to bind the device to a node and
> adding a config parameter to Xend.
> 
> I''d imagine we can rely on actual physical device numbers
generally staying
> the same but this could be fixed by a similar mechanism.
> 
What is a physical device number in this context? The name it 
ends up with is often detection order dependent. /dev/sdb has little
relation to physical device numbers. Also, with the better support
for hotplug in 2.6 devices can come and go. 

Cheers,
Phil
> Cheers,
> Mark
> 
> > Having to configure all the possible migration targets to see all
> > the same devices in the same order at all times is not realistic.
> > In a dynamic environment it would be ideal to make a device visible
> > to the target dom0 only as it was needed to migrate a given VM to
> > that dom0. Naming it /dev/sdb may not be a possiblity in this case.
> >
> >
> > Cheers,
> >
> > Phil
> >
> > > Cheers,
> > > Ian
> > >
> > > _______________________________________________
> > > Xen-devel mailing list
> > > Xen-devel@lists.xensource.com
> > > http://lists.xensource.com/xen-devel
-- 
Philip R. Auld, Ph.D.  	        	       Egenera, Inc.    
Software Architect                            165 Forest St.
(508) 858-2628                            Marlboro, MA 01752

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> > I think Rik may have volunteered to take a look at the 
> console stuff 
> > :-)
> 
> NOOOOOOOOOOOOOOOOOOOOOOO !!!
> 
> *runs away screaming, leaving a Rik-shaped hole in the wall*
> 
> (I volunteered to look at some of the memory management and  
> upstream merging bits - which I will look at as soon as the  
> xenU kernel works again in rawhide)
Ah, OK, sorry.

I''m sure someone came up to me last week and volunteered to look at
''screen''. I remember thinking "ah, that''s in
safe hands". Will the real
new console maintainer please stand up? :-)

Ian

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> > You could work round this currently using the block config scripts. 
I.e.
> > similar to the file: syntax now, you could easily hack up nbd:,
iscsi:,
> > etc. Then network block devices can be connected up at create /
migrate /
> > restore time, without relying on a particular device node.
>
> Thanks. I''m not really looking for a work around though :) I think
> this is an issue that we should look at in more depth.
Well for network storage that need configuration I''d recommend this as
the way
to go, anyway.  For local block devices, I guess things are a bit less 
clearcut but I think this should be workable too.
> My concern is for SAN attached devices primarily.  These will
> show up as scsi devices with /dev/sdXX names (barring custom
> udev configs).
OK. (as a side note, you could presumably set up udev to name the nodes 
consistently?)
> BTW, how would you do this with iSCSI? iSCSI target LUNs show
> up as scsi devices and get block device names. It''ll have the same
> issues as FC SAN.
Is there no way of configuring which node an iSCSI lun gets bound to?  How 
would you (manually) figure out which it is bound to in order to configure 
the domain?  Could you not script that procedure?

If you can script the procedure for (maybe connecting to a server) and then 
figuring out what node the device is connected to, Xend will call your script 
and sort things out automatically.  It''s the most straightforward way
to do
this in the 2.x releases, although things might be different in 3.0 depending 
on how much of the toolchain is rewritten.
> > This can be done by writing a shell script to bind the device to a
node
> > and adding a config parameter to Xend.
> >
> > I''d imagine we can rely on actual physical device numbers
generally
> > staying the same but this could be fixed by a similar mechanism.
>
> What is a physical device number in this context? The name it
> ends up with is often detection order dependent. /dev/sdb has little
> relation to physical device numbers. Also, with the better support
> for hotplug in 2.6 devices can come and go.
Oops, I really meant node rather than number.  

Cheers,
Mark

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> > You could work round this currently using the block config scripts. 
I.e.
> > similar to the file: syntax now, you could easily hack up nbd:,
iscsi:, etc.
> > Then network block devices can be connected up at create / migrate /
restore
> > time, without relying on a particular device node.
> 
> Thanks. I''m not really looking for a work around though :) I think
> this is an issue that we should look at in more depth.
> 
I don''t think this is necessarily a "workaround". On filers
(and
presumably everyone else''s boxes) iSCSI and FCP LUNs have full path
names. The device number is purely a function of the local machine.
Adding some logic to xend so that it thinks in terms of LUNs instead
of device nodes would be highly useful to some of us.
> My concern is for SAN attached devices primarily.  These will
> show up as scsi devices with /dev/sdXX names (barring custom
> udev configs).
> 
> I suppose though that if xen doesn''t care about the actual names.
Then
> making a link in /dev/ could work. Maybe this is not really an issue after
> all.  e.g. /dev/my_domU_1_disk -> /dev/sdb. Then as long as something
> on the other end can tell where the disk showed up it can make the
> appropriate link -- /dev/my_domU_1_disk -> /dev/sdq.
I have a script that takes as an argument the number of LUNs, the size
of the LUNs, and the name of the VM. These LUNs are all mapped in a
probe and creation dependent order. I have a script that iterates
through all the scsi generic devices (passthrough), sends them an
inquiry - if they''re LUNs it sends the device a vendor specific mode
page to get the full pathname. The pathname contains the VM the LUN
belongs to allowing the script to create a device node corresponding
to the sd<n> device in /dev/vm/<vmname>. I''ve added the
following
snippet for mapping the LUNs in at VM creation time:

globstr = "/dev/vm/%s/*"%name

disknamelist = glob.glob(globstr)
count = 1
for diskname in disknamelist:
    disk.append(''phy:%s,loop%s,w''%(diskname,count))
    count += 1


I''m not entirely thrilled with SAN management in this model. If I
delete LUNs on the server all the mappings get shifted down due to the
1-dimensional nature of the device node namespace.
> 
> What is a physical device number in this context? The name it
> ends up with is often detection order dependent. /dev/sdb has little
> relation to physical device numbers. Also, with the better support
> for hotplug in 2.6 devices can come and go.
Yep. There are some open issues here. major / minor numbers that are a
function of probe order are clumsy at best.


              -Kip

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Thu, 2005-04-14 at 14:22 -0400, Philip R Auld wrote:
> As a slight but related digression, has any thought been given to
> using something other than /dev/[hs]d* names for specifying block
> devices? The name /dev/sdb only has meaning for the current
> state of the running dom0 OS. It may not mean the same device
> on a different dom0. This can lead to problems in say migration.
> In fact, it may not be the same when the domain is restored locally.
To be even slightly more radical, I don''t think that for the guests,
the
vbds should be being exposed as SCSI or IDE since they''re not.  By
being
presented as such, there''s a set of assumptions about ioctls and
behaviors which aren''t the case for disks on blkfront.  Even with all
of
the pain using your own major/minor for a disk device is, I think it
probably is the right thing to do.  And I''ve added the code to
parted/lvm2/.../etc enough to be able to do it in my sleep this point if
this route is taken :)

Jeremy


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Thu, 2005-04-14 at 20:21 +0100, Ian Pratt wrote:
> > > Yep, it would be much better if we could hide this all in xend. 
> > > Ideally, it should be possible to start a domain, then 
> > connect to its 
> > > console within the bootloader''s timeout and be able to 
> > interact with it.
> > 
> > If the console code is changing around, though, hooking up to 
> > the current stuff might be more trouble than its worth.  
> > Would be better in the long run, though.
> 
> I think Rik may have volunteered to take a look at the console stuff :-)
> 
> I actually think this is worth implementing right now, as it sets a key
> requirement for the new console stuff -- it must be possible to steal a
> guest console and inject a bootloader session. We might want something
> similar in future for a gdb session too.
The problem is that implementing it on top of the current console stuff
is going to be a little bit grotesque.  As it stands right now, I don''t
see any obvious way of doing it other than some sort of hijacking of the
TCP socket.
  
> > > > - need to be able to configure the kernel to boot in the 
> > domain config
> > > >    when not using a console.
> > > 
> > > This would certainly be nice.
> > 
> > Already there, just change your default in the config.
> 
> Just to make sure, you mean the VM config, not the grub config, right?
> Do you specify the label name, or the crappy grub menu number?
Ooooohhhh (light bulb goes off here).  Right now, the default comes from
the grub.conf in the domU.  If you want to change the default from the
vm config, it''s easy enough to add.  It could even be flexible enough
to
let you specify the label name or the menu number.  

But before doing it, what''s the use case in mind?  In general, I
don''t
think that''s something you want to hardcode in the VM config -- I guess
I could see specifying it on the xm create command line, but just am not
100% sure what that buys you over having the interface pop up.

Jeremy


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

"Ian Pratt" <m+Ian.Pratt@cl.cam.ac.uk> writes:
> I''m sure someone came up to me last week and volunteered to look
at
> ''screen''. I remember thinking "ah, that''s
in safe hands". Will the real
> new console maintainer please stand up? :-)
#include <disclaimer/im_not_the_new_console_maintainer>

Well, I don''t think screen needs changes.  screen can be used to
manage any terminal device (including multiple of these at the same
time in different screen windows).

I''m using "screen -c /dev/null /dev/ttyS0 115200" to manage
the serial
console of the crashing devel machine ;)

"screen /dev/pts/<nr>" works fine as well, I use that sometimes
with
user mode linux which can redirect the consoles devices of the uml
virtual machine to pseudo terminals on the host machine.  Something
simliar would probably work fine for xen as well, although it probably
isn''t as easy as with uml when considering migration ...

  Gerd

-- 
#define printk(args...) fprintf(stderr, ## args)

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel