CentOS virt - Jun 2014 - Preferred method of provisioning VM images

Hi all,

following the discussion on about documentation, I was wondering whether 
we need to look at a standard way in which we recommend how to provision 
images for VMs. Am starting this with a Xen hat, but the discussion 
should not be specific to this. There are a number of options, but all 
have some trade-offs

== #1 virt-install =
Advantages: similar to KVM

Disadvantages: may cause weird issues / confusion with people switching 
back to xl. The core issue is that with the current version of xen and 
libvirt, this only works with xm (when xl is used, this can create some 
undefined behavior). However as we have seen in some recent threads on 
this list, people tend to mix which can cause problems.

== #2 xen-tools =
Advantages: Very flexible. Many other distros use xen-tools, so we have 
lots of beginners docs that just need to be tweaked

Disadvantages: needs porting/packaging for CentOS. Does not work for 
kvm. Says "xen". (Maybe that's an advantage.)
We know that xen-tools works with Fedora (see 
http://blog.xen.org/index.php/2013/01/24/using-xen-tools-on-fedora/), so 
the porting effort may be small

Unknowns: What would be needed to make it work for CentOS

== #3 virt-builder (http://libguestfs.org/virt-builder.1.html) =
Advantages: supports KVM, Xen and other VM inages. Seems easy to use.
- if so, it would avoid xm / xl confusion.

Unknowns: Not sure at which level virt-builder integrates with Xen and 
other hypervisors. It seems to operate at disk image format (similar to 
xen-tools) . I don't know whether virt-builder is restricted to some 
hypervisors in RHEL7.

Disadvantages: may need porting/packaging for CentOS. It appears as if 
it will be in RHEL7, so it may just appear with CentOS 7. If not, some 
porting work may need to be done.

== #4 Cloud Image from Cloud Image SIG =We could rely on pre-built cloud images
from the Cloud Images SIG.
People could just download the cloud image once it's done and customize 
it, rather than installing / building their own.

Advantages: seems easy

Disadvantages: coordination with Cloud Images SIG. May not be flexible 
enough

I just wanted to start a discussion about this and ask for input. This 
topic which has come up a number of times in SIG meetings as a facgtor 
influencing libvirt and other package versions.

Regards
Lars

On 10/06/14 12:21 PM, Lars Kurth wrote:
> Hi all,
>
> following the discussion on about documentation, I was wondering whether
> we need to look at a standard way in which we recommend how to provision
> images for VMs. Am starting this with a Xen hat, but the discussion
> should not be specific to this. There are a number of options, but all
> have some trade-offs
>
> == #1 virt-install =>
> Advantages: similar to KVM
>
> Disadvantages: may cause weird issues / confusion with people switching
> back to xl. The core issue is that with the current version of xen and
> libvirt, this only works with xm (when xl is used, this can create some
> undefined behavior). However as we have seen in some recent threads on
> this list, people tend to mix which can cause problems.
>
> == #2 xen-tools =>
> Advantages: Very flexible. Many other distros use xen-tools, so we have
> lots of beginners docs that just need to be tweaked
>
> Disadvantages: needs porting/packaging for CentOS. Does not work for
> kvm. Says "xen". (Maybe that's an advantage.)
> We know that xen-tools works with Fedora (see
> http://blog.xen.org/index.php/2013/01/24/using-xen-tools-on-fedora/), so
> the porting effort may be small
>
> Unknowns: What would be needed to make it work for CentOS
>
> == #3 virt-builder (http://libguestfs.org/virt-builder.1.html) =>
> Advantages: supports KVM, Xen and other VM inages. Seems easy to use.
> - if so, it would avoid xm / xl confusion.
>
> Unknowns: Not sure at which level virt-builder integrates with Xen and
> other hypervisors. It seems to operate at disk image format (similar to
> xen-tools) . I don't know whether virt-builder is restricted to some
> hypervisors in RHEL7.
>
> Disadvantages: may need porting/packaging for CentOS. It appears as if
> it will be in RHEL7, so it may just appear with CentOS 7. If not, some
> porting work may need to be done.
>
> == #4 Cloud Image from Cloud Image SIG => We could rely on pre-built
cloud images from the Cloud Images SIG.
> People could just download the cloud image once it's done and customize
> it, rather than installing / building their own.
>
> Advantages: seems easy
>
> Disadvantages: coordination with Cloud Images SIG. May not be flexible
> enough
>
> I just wanted to start a discussion about this and ask for input. This
> topic which has come up a number of times in SIG meetings as a facgtor
> influencing libvirt and other package versions.
>
> Regards
> Lars
I would recommend the default be the libvirtd tools, as it tries to be 
hypervisor-agnostic, so it's most portable. Also, it provides a good 
amount of flexibility for various install type, where images/docker is 
more restricted. Again, speaking about defaults/generalities.

I would, of course, point to the other tools so that users know what 
exists and what might work best for their use case.

-- 
Digimer
Papers and Projects: https://alteeve.ca/w/
What if the cure for cancer is trapped in the mind of a person without 
access to education?

Lars Kurth <lars.kurth at xen.org> writes:
> Hi all,
>
> following the discussion on about documentation, I was wondering whether 
> we need to look at a standard way in which we recommend how to provision 
> images for VMs. Am starting this with a Xen hat, but the discussion 
> should not be specific to this. There are a number of options, but all 
> have some trade-offs
Let me wear the hat of the user.  The major hurdles were network setup,
installing something in a vm, and the chaotic state the documentation is
in.

I knew before I started that network setup would be a PITA because years
ago, I set up a VM for someone who didn't have a 64bit system to compile
a 64bit version of some software.  The network setup being so
ridiculously difficult has kept me from touching VMs ever again for
years.  It's just too difficult and not worth the effort unless you're
really forced to do it.

As a user, I'm used to get an ISO of an installer or of a life system,
put that into a DVD drive or write it to an USB stick and to boot from
that to do the installation.  Why can't I do that with xen?

> == #1 virt-install => == #2 xen-tools => == #3 virt-builder
(http://libguestfs.org/virt-builder.1.html) => == #4 Cloud Image from Cloud
Image SIG =
I wouldn't want #4.  I want to be able to just use the installer of
whatever distribution I'm installing in a VM and a simple way of saying
"run this in a VM and let me install".  Or perhaps I merely want to
run
a life system in a VM to try it out, with network access.  What easy way
is there to do that?

I haven't heared of #3.

#2 and #1 lead to total confusion.  Xen doesn't have any way to use an
installer ISO to install a VM.  The documentation is confusing because
it lacks clarity and refers to many versions of xen at the same time
which suggest to do things differently so that you don't know how you
are supposed to do something.  There is no documentation for centos
about how to set up a VM.

#2 and #1 use different ways and different formats of configuration
files, and apparently #1 doesn't support everything #2 does and brings
about an undesirable overhead while apparently being a good choice for
users who also use other things than xen and don't want to learn
different ways of managing VMs.

When I use xen, I do not need #1, and it seems to make more sense to me
to stick with #2.

#2 needs clarity of documentation.  I've got an installer ISO.  The
documentation needs to tell me straightforwardly what I need to do to
get a VM running with this installer ISO.  I don't care about #1--4 at
that point.

It's that simple.  More details can be learned over time.


Insofar the documentation needs to deal with different versions of xen,
just split the documentation entirely.  When I use version X, I do not
want to read the documentation for versions A, B, C or Z.  That's
confusing and I have to ask myself all the time which part of what I'm
reading might apply to the software I'm using, and that makes reading
the documentation very painful.

It's like you buy a new TV and it comes with 50000 pages of
documentation because the manufacturer of the TV mentions everything for
every model of TV they ever manufactured, rather than documenting no
more than the TV you actually bought.  Trying to figure out how to
switch to a different channel or how to increase the volume takes at
least two days each.


For example: http://wiki.xen.org/wiki/Credit_Scheduler:

"It is now the default scheduler in the xen-unstable trunk."

So is it the default in the version I'm using or what?

"The SEDF and BVT schedulers are still optionally available but the plan
of record is for them to be phased out and eventually removed."

What are those and have they been removed in the version I'm using?  But
then, it probably doesn't matter what they are because they might have
been removed anyway.  So what is this information supposed to tell me??

"A domain with a weight of 512 will get twice as much CPU as a domain
with a weight of 256 on a contended host. Legal weights range from 1 to
65535 and the default is 256."

Where would I change this default?  What happens when I set a weight of
100 or of 33?  I can only guess that it might be rounded up or down to
something like 128 or 64 or 32 or 64.  Or perhaps actually floating
point math is employed to compute the relations of the different
weights?  Probably not ...

"Schedule Rate Limiting (added in Xen 4.2)"

That doesn't belong there.  It needs to go into the documentation for
that particular version of xen.

"Timeslice (added in Xen 4.2)
[...]
# xl sched-credit -t [n]"

dito

"Usage

The xm sched-credit command may be used to tune the per VM guest scheduler
parameters."

So what am I supposed to use? xm or xl?  Or something else, maybe virsh?

I always admire the documentation exim has ...  As to your original
question:
> I was wondering whether we need to look at a standard way in which we
> recommend how to provision images for VMs.
I'd suggest to start with splitting the documentation into different
versions, one version of documentation for each version of xen.  Some
way to get a VM running may then be discovered and improved upon.

What actually is the way to do it?  I found something that works, and it
allows me to install Debian squeeze, from which then I upgrade to
wheezy.  For anything else, I don't know.


-- 
Knowledge is volatile and fluid.  Software is power.

On 06/11/2014 04:21 AM, Lars Kurth wrote:
> Hi all,
> 
> following the discussion on about documentation, I was wondering whether 
> we need to look at a standard way in which we recommend how to provision 
> images for VMs. Am starting this with a Xen hat, but the discussion 
> should not be specific to this. There are a number of options, but all 
> have some trade-offs
> 
> == #1 virt-install => == #2 xen-tools => == #3 virt-builder
(http://libguestfs.org/virt-builder.1.html) => == #4 Cloud Image from Cloud
Image SIG =
This is not a complete list of the ways you can install a VM either.  My
personal preference is to manually create the filesystem for the VM and
then install the OS core with yum.  Then after tweaking some config
files you can start up the VM and finish installing whatever else you
want with yum as well.

While I don't think that this should be the recommended install method
it might be worth mentioning and even giving a wiki page with some
instructions on how to do it this way.  Doing an install like this is
actually very good for a newbie because you "get your hands dirty" and
get a really good understanding of how yum works and the internals of
the distro.


Peter

On mar, 2014-06-10 at 17:21 +0100, Lars Kurth wrote:
> Hi all,
> 
Hi!
> == #1 virt-install => 
> Advantages: similar to KVM
> 
> Disadvantages: may cause weird issues / confusion with people switching 
> back to xl. The core issue is that with the current version of xen and 
> libvirt, this only works with xm (when xl is used, this can create some 
> undefined behavior). However as we have seen in some recent threads on 
> this list, people tend to mix which can cause problems.
> 
When you're saying "this only works with xm (when xl is used, this can
create some undefined behavior", that is because of the particular
version of libvirt (and Xen) is available in CentOS?

I'm asking because, with current enough versions of both, situation
should not be that bad, at least not as far as virt-install and VM
provisioning are concerned... :-O

Personally, I'm not sure whether this is the best solution, but I see it
like something that should not be left in a 'non-working' state.

Parhaps, a disadvantage I see in going for it as the default and
recommended mechanism is that you really need:
 - a recent version of libvit to start with
 - a way to update libvirt quite often, to take advantage of new 
   features being added
> == #2 xen-tools => 
> Advantages: Very flexible. Many other distros use xen-tools, so we have 
> lots of beginners docs that just need to be tweaked
> 
> Disadvantages: needs porting/packaging for CentOS. Does not work for 
> kvm. Says "xen". (Maybe that's an advantage.)
> We know that xen-tools works with Fedora (see 
> http://blog.xen.org/index.php/2013/01/24/using-xen-tools-on-fedora/), so 
> the porting effort may be small
> 
Yep, and I went beyond that, and produced some spec files. This should
be the latest one, although it's for version 4.3.1 of xen-tools, so no,
not the latest possible or the tip of the repo:
http://dariof.fedorapeople.org/SPECS/xen-tools.spec

My main problem with xen-tools is that it is kind of debian (and
derivatives) biased, when it comes to building the guest file system.
I.e., last time I tried, provisioning a Fedora or CentOS guest was
possible, but not as straightforward as creating a debian one. That is
not xen-tool's fault, is the fact there were no good alternatives to
debootstrap.

However, it looks like rinse (one of said alternatives to debootstrap)
is getting some steam again:
https://gitorious.org/rinse

so that may actually be worth a try.

Also, I recently saw this in xen-tools-discuss mailing list:

"anouncement : Xen-tools update for centos 6.5"
 http://xen-tools.org/pipermail/xen-tools-discuss/2014-May/001053.html

Even if only for the title, it looks like something to investigate! :-P
> Unknowns: What would be needed to make it work for CentOS
> 
Talking a bit with Axel (xen-tools' maintainer) at the time, the main
issue in packaging subsequent versions of xen-tools for Fedora was going
to be some dependencies from a bunch of perl module he wrote, and
packaged himself for Debian, as separate packages. Nothing terrible, but
that would mean making the RPMs for those modules as well.

TBH, it is a while back, and I'm not sure what happened, whether went
that route. This link, from the xen-tools-discuss email mentioned above:
https://github.com/remsnet/xen-tools-rpmbuild

seems to have (more) updated spec files. I haven't tried them, though.
> == #3 virt-builder (http://libguestfs.org/virt-builder.1.html) => 
> Advantages: supports KVM, Xen and other VM inages. Seems easy to use.
> - if so, it would avoid xm / xl confusion.
> 
> Unknowns: Not sure at which level virt-builder integrates with Xen and 
> other hypervisors. It seems to operate at disk image format (similar to 
> xen-tools) . I don't know whether virt-builder is restricted to some 
> hypervisors in RHEL7.
> 
> Disadvantages: may need porting/packaging for CentOS. It appears as if 
> it will be in RHEL7, so it may just appear with CentOS 7. If not, some 
> porting work may need to be done.
> 
Not commenting on this as I don't know it.

All that being said, and FWIW, I think I'd prefer #1, provided a recent
enough version of Xen and Libvirt could be used.

Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part
URL:
<http://lists.centos.org/pipermail/centos-virt/attachments/20140612/a2fa077f/attachment-0001.sig>

On Tue, 2014-06-10 at 17:21 +0100, Lars Kurth wrote:
> Hi all,
> 
> following the discussion on about documentation, I was wondering whether 
> we need to look at a standard way in which we recommend how to provision 
> images for VMs. Am starting this with a Xen hat, but the discussion 
> should not be specific to this. There are a number of options, but all 
> have some trade-offs
> 
> == #1 virt-install => 
> Advantages: similar to KVM
> 
> Disadvantages: may cause weird issues / confusion with people switching 
> back to xl. The core issue is that with the current version of xen and 
> libvirt, this only works with xm (when xl is used, this can create some 
> undefined behavior). However as we have seen in some recent threads on 
> this list, people tend to mix which can cause problems.
> ...
  I've chosen the virt-install method on CentOS 5 precisely because it
is like KVM.  I was hoping it would fulfill the promise of being
hypervisor agnostic.  I'm hoping it continues to be available on future
versions of CentOS with Xen.

  Though it is a waste of resources, I make all my virtual machines,
Linux and MS Windows alike, fully virtualized.  I can then move any of
the VM's with the same virt-install --import or virsh dumpxml/edit/virsh
define process.

  When moving a VM, usually the only thing I have to do outside of
virt-install/virt-manager is add <acpi/>, <apic/> or <pae/>,
which can
be done with virsh edit.  I don't know why some of my virtual servers
need them and other don't but I have higher priority things to think
about.

  I'm the only technical support person and I don't work 24/7.  The
graphical interface of virt-manager makes it possible for non-tech
people to see what is running and see consoles to restart any
misbehaving VM's (usually MS Windows VM's).

  I have completely eliminated my use of xm.

On 06/10/2014 05:21 PM, Lars Kurth wrote:
> == #4 Cloud Image from Cloud Image SIG => We could rely on pre-built
cloud images from the Cloud Images SIG.
> People could just download the cloud image once it's done and customize
> it, rather than installing / building their own.
> 
> Advantages: seems easy
> 
> Disadvantages: coordination with Cloud Images SIG. May not be flexible 
> enough
We ship a test/devel grade CentOS-6-x86_64-pv image ( well, its a qcow2
image, should work for pvhvm as well, the fstab is label driven ).[1]

The biggest problem in doing pre-baked images is the instance metadata.
We need to find an easy way to get network settings into the instance
and the root password ( or key ), and finally - in some cases, console
redirection/setup, but i dont think the console is a deal breaker or a
big deal. The network and access credentials however are.

In a typical cloud environ this info would come from the cloud
controller's metadata service; on a typical virtualised setup though
this becomes an issue ( and isnt really Xen specific ).

We could work around this by making some assumptions, we could 'own'
dnsmasq and ensure that either libvirt is running and doing dhcp,
otherwise we do the dhcp with some sane defaults, or we setup a script
to 'instantiate image', which asks how the user wants to setup the
instance ( pvhvm, hvm, pv ), the root password or key to use, and the
network settings ( and if this is run on the dom0, we could even ask
what bridge or device to connect with as well as the settings ).[2]

Ofcourse, having these images pushed from here mean that clouds or
virtualised environs that have metadata services are able to just-use
the image as is, not needing any more tooling etc. And we can easily
push monthly image updates and when things like heartbleed come around,
there is a single place we need to update.

- KB
[1]: http://cloud.centos.org/centos/6/devel/

[2] might need to pull in all of libguestfs to make the changes, which
in turn has its own challenges if run inside a virtualised environ.

-- 
Karanbir Singh
+44-207-0999389 | http://www.karan.org/ | twitter.com/kbsingh
GnuPG Key : http://www.karan.org/publickey.asc