Ovirt devel - Mar 2018 - [virt-tools-list] Project for profiles and defaults for libvirt domains

On Tue, Mar 20, 2018 at 03:20:31PM +0100, Martin Kletzander
wrote:
> 1) Default devices/values
> 
> Libvirt itself must default to whatever values there were before any
> particular element was introduced due to the fact that it strives to
> keep the guest ABI stable.  That means, for example, that it can't just
> add -vmcoreinfo option (for KASLR support) or magically add the pvpanic
> device to all QEMU machines, even though it would be useful, as that
> would change the guest ABI.
> 
> For default values this is even more obvious.  Let's say someone
figures
> out some "pretty good" default values for various HyperV
enlightenment
> feature tunables.  Libvirt can't magically change them, but each one of
> the projects building on top of it doesn't want to keep that list
> updated and take care of setting them in every new XML.  Some projects
> don't even expose those to the end user as a knob, while others might.
This gets very tricky, very fast.

Lets say that you have an initial good set of hyperv config
tunables. Now sometime passes and it is decided that there is a
different, better set of config tunables. If the module that is
providing this policy to apps like OpenStack just updates itself
to provide this new policy, this can cause problems with the
existing deployed applications in a number of ways.

First the new config probably depends on specific versions of
libvirt and QEMU,  and you can't mandate to consuming apps which
versions they must be using. So you need a matrix of libvirt +
QEMU + config option settings.

Even if you have the matching libvirt & QEMU versions, it is not
safe to assume the application will want to use the new policy.
An application may need live migration compatibility with older
versions. Or it may need to retain guaranteed ABI compatibility
with the way the VM was previously launched and be using transient
guests, generating the XML fresh each time.

The application will have knowledge about when it wants to use new
vs old hyperv tunable policy, but exposing that to your policy module
is very tricky because it is inherantly application specific logic
largely determined by the way the application code is written.

> One more thing could be automatically figuring out best values based on
> libosinfo-provided data.
> 
> 2) Policies
> 
> Lot of the time there are parts of the domain definition that need to be
> added, but nobody really cares about them.  Sometimes it's enough to
> have few templates, another time you might want to have a policy
> per-scenario and want to combine them in various ways.  For example with
> the data provided by point 1).
> 
> For example if you want PCI-Express, you need the q35 machine type, but
> you don't really want to care about the machine type.  Or you want to
> use SPICE, but you don't want to care about adding QXL.
> 
> What if some of these policies could be specified once (using some DSL
> for example), and used by virtuned to merge them in a unified and
> predictable way?
> 
> 3) Abstracting the XML
> 
> This is probably just usable for stateless apps, but it might happen
> that some apps don't really want to care about the XML at all.  They
> just want an abstract view of the domain, possibly add/remove a device
> and that's it.  We could do that as well.  I can't really tell how
much
> of a demand there is for it, though.
It is safe to say that applications do not want to touch XML at all.
Any non-trivial application has created an abstraction around XML,
so that they have an API to express what they want, rather than
manipulating of strings to format/parse XML.

The libvirt-gconfig project aims to provide a C API for manipulating
XML documents, with language bindings available via GObject
introspection so you can use it from Vala, Perl, Python, JavaScript,
etc. Go is notable missing, but for that we have libvirt-go-xml
which provides a set of native Go structs to represent the XML.

The problem we've faced with libvirt-gconfig is that it is a really
hard sell to get applications to convert existing code to use it.
We've only had success where an applicaiton has been written to use
libvirt-gconfig from day one - eg GNOME Boxes and libvirt-sandbox.

Virt-manager is the poster-child for using libvirt-gconfig, but I
don't see it adopting it any time soon, as it is a massive effort
to change all existing code - even if libvirt-gconfig had full
XML schema coverge, which it doesn't :-(

I also wanted to use libvirt-gconfig in OpenStack, but there was
resistance for adding a dependancy on another native library, so
there we've basically copied what virt-manager does and defined a
set of pure python objects to represent config which is serialized
to / from XML :-(  If libvirt-gconfig were to be used by OpenStack
it would need to deal with fact that not every distro has that
available, so the existing pure python config objects would need
to be maintained in parallel for an indefinite amount of time.
So in fact using libvirt-gconfig would increase maint burden for
OpenStack, rather than reduce it.

libvirt-gconfig would be a hard sell for Go apps when you compare
it to libvirt-go-xml, because the latter is following the common
Go paradigm for XML manipulation.

If there was something higher level that gets more interesting,
but the hard bit is that you still need a way to get at all the
low level bits becuase a higher level abstracted API will never
cover every niche use case. 
> 4) Identifying devices properly
> 
> In contrast to the previous point, stateful apps might have a problem
> identifying devices after hotplug.  For example, let's say you
don't
> care about the addresses and leave that up to libvirt.  You hotplug a
> device into the domain and dump the new XML of it.  Depending on what
> type of device it was, you might need to identify it based on different
> values.  It could be <target dev=''/> for disks, <mac
address=''/> for
> interfaces etc.  For some devices it might not even be possible and you
> need to remember the addresses of all the previous devices and then
> parse them just to identify that one device and then throw them away.
> 
> With new enough libvirt you could use the user aliases for that, but
> turns out it's not that easy to use them properly anyway.  Also the
> aliases won't help users identify that device inside the guest.
NB, relating between host device config and guest visible device
config is a massive problem space in its own right, and not very
easy to address. In OpenStack we ended up defining a concept of
"device tagging" via cloud-init metadata, where openstack allows
users to set opaque string tags against devices their VM has.
OpenStack that generates a metadata file that records various
pieces of identifying hardware attributes (PCI address, MAC
addr, disk serial, etc) alongside the user tag. This metadata
file is exposed to the guest with the hope that there's enough
info to allow the user to decide which device is to be used for
which purpose

https://specs.openstack.org/openstack/nova-specs/specs/mitaka/approved/virt-device-role-tagging.html
https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/10/html/networking_guide/use-tagging
> <rant>
> We really should've gone with new attribute for the user alias instead
> of using an existing one, given how many problems that is causing.
> </rant>
> 
> 5) Generating the right XML snippet for device hot-(un)plug
> 
> This is kind of related to some previous points.
> 
> When hot-plugging a device and creating an XML snippet for it, you want
> to keep the defaults from point 1) and policies from 2) in mind.  Or
> something related to the already existing domain which you can describe
> systematically.  And adding something for identification (see previous
> point).
> 
> Doing the hot-unplug is easy depending on how much information about
> that device is saved by your application.  The less you save about the
> device (or show to the user in a GUI, if applicable) the harder it might
> be to generate an XML that libvirt will accept.  Again, some problems
> with this should be fixed in libvirt, some of them are easy to
> workaround.  But having a common ground that takes care of this should
> help some projects.
> 
> Hot-unplug could be implemented just based on the alias.  This is
> something that would fit into libvirt as well.
> 
> =======================================================================>
> To mention some pre-existing solutions:
> 
> - I understand OpenStack has some really sensible and wisely chosen
>  and/or tested default values.
In terms of default devices and OS specific choices, OpenStack's
decisions have been largely inspired by previous work in oVirt
and / or virt-manager. So there's obviously overlap in the
conceptual area, but there's also plenty that is very specific
to OpenStack - untangling the two extract the common bits from
the app specific bits is hard. 
> - I know KubeVirt has VirtualMachinePresets.  That is something closely
>  related to points 1) and 2).  Also their abstraction of the XML might
>  be usable for point 3).
> 
> - There was an effort on creating policy based configuration of libvirt
>  objects called libvirt-designer.  This is closely related to points 2)
>  and 3).  Unfortunately there was no much going on lately and part of
>  virt-manager repository has currently more features implemented with
>  the same ideas in mind, just not exported for public use.
This is the same kind of problem we faced wrt libvirt-gconfig and
libvirt-gobject usage from virt-manager - it has an extensive code
base that already works, and rewriting it to use something new
is alot of work for no short-term benefit. libvirt-gconfig/gobject
were supposed to be the "easy" bits for virt-manager to adopt, as
they don't really include much logic that would step on virt-manager's
toes. libvirt-designer was going to be a very opinionated library
and in retrospective that makes it even harder to consider adopting
it for usage in virt-manager, as it'll have signficant liklihood
of making functionally significant changes in behaviour.

There's also the problem with use of native libraries that would
impact many apps. We only got OpenStack to grudgingly allow the
use of libosinfo native library via GObject Introspection, by
promising to do work to turn the osinfo database into an approved
stable format which OpenStack could then consume directly, dropping
the native API usage :-(  Incidentally, the former was done (formal
spec for the DB format), but the latter was not yet (direct DB usage
by OpenStack)


BTW, I don't like that I'm being so negative to your proposal :-(
I used to hope that we would be able to build higher level APIs on
top of libvirt to reduce the overlap between different applications
reinventing the wheel. Even the simplest bits we tried like the
gconfig/gobject API are barely used. libvirt-designer is basically
a failure. Though admittedly it didn't have enough development resource
applied to make it compelling, in retrospect adoption was always going
to be a hard sell except in greenfield developments.

Libosinfo is probably the bit we've had most success with, and has
most promise for the future, particularly now that we formally allow
apps to read the osinfo database directly and bypass the API. It is
quite easy to fit into existing application codebases which helps alot.
Even there I'm still disappointed that we only have GNOME Boxes using
the kickstart generator part of osinfo - oVirt and Oz both still have
their own kickstart generator code for automating OS installs.

In general though, I fear anything API based is going to be a really
hard sell to get wide adoption for based on what we've seen before.

I think the biggest bang-for-buck is identifying more areas where we
can turn code into data. There's definitely scope for recording more
types of information in the osinfo database. There might also be
scope for defining entirely new databases to complement the osinfo
data, if something looks out of scope for libosinfo.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

On Tue, Mar 20, 2018 at 03:10:12PM +0000, Daniel P. Berrang?
wrote:
> On Tue, Mar 20, 2018 at 03:20:31PM +0100, Martin Kletzander wrote:
> > 1) Default devices/values
> > 
> > Libvirt itself must default to whatever values there were before any
> > particular element was introduced due to the fact that it strives to
> > keep the guest ABI stable.  That means, for example, that it can't
just
> > add -vmcoreinfo option (for KASLR support) or magically add the
pvpanic
> > device to all QEMU machines, even though it would be useful, as that
> > would change the guest ABI.
> > 
> > For default values this is even more obvious.  Let's say someone
figures
> > out some "pretty good" default values for various HyperV
enlightenment
> > feature tunables.  Libvirt can't magically change them, but each
one of
> > the projects building on top of it doesn't want to keep that list
> > updated and take care of setting them in every new XML.  Some projects
> > don't even expose those to the end user as a knob, while others
might.
> 
> This gets very tricky, very fast.
> 
> Lets say that you have an initial good set of hyperv config
> tunables. Now sometime passes and it is decided that there is a
> different, better set of config tunables. If the module that is
> providing this policy to apps like OpenStack just updates itself
> to provide this new policy, this can cause problems with the
> existing deployed applications in a number of ways.
> 
> First the new config probably depends on specific versions of
> libvirt and QEMU,  and you can't mandate to consuming apps which
> versions they must be using.  [...]
This is true.
>                       [...]  So you need a matrix of libvirt +
> QEMU + config option settings.
But this is not.  If config options need support on the lower
levels of the stack (libvirt and/or QEMU and/or KVM and/or host
hardware), it already has to be represented by libvirt host
capabilities somehow, so management layers know it's available.

This means any new config generation system can (and must) use
host(s) capabilities as input before generating the
configuration.

> 
> Even if you have the matching libvirt & QEMU versions, it is not
> safe to assume the application will want to use the new policy.
> An application may need live migration compatibility with older
> versions. Or it may need to retain guaranteed ABI compatibility
> with the way the VM was previously launched and be using transient
> guests, generating the XML fresh each time.
Why is that a problem?  If you want live migration or ABI
guarantees, you simply don't use this system to generate a new
configuration.  The same way you don't use the "pc" machine-type
if you want to ensure compatibility with existing VMs.
> 
> The application will have knowledge about when it wants to use new
> vs old hyperv tunable policy, but exposing that to your policy module
> is very tricky because it is inherantly application specific logic
> largely determined by the way the application code is written.
We have a huge set of features where this is simply not a
problem.  For most virtual hardware features, enabling them is
not even a policy decision: it's just about telling the guest
that the feature is now available.  QEMU have been enabling new
features in the "pc" machine-type for years.

Now, why can't higher layers in the stack do something similar?

The proposal is equivalent to what already happens when people
use the "pc" machine-type in their configurations, but:
1) the new defaults/features wouldn't be hidden behind a opaque
   machine-type name, and would appear in the domain XML
   explicitly;
2) the higher layers won't depend on QEMU introducing a new
   machine-type just to have new features enabled by default;
3) features that depend on host capabilities but are available on
   all hosts in a cluster can now be enabled automatically if
   desired (which is something QEMU can't do because it doesn't
   have enough information about the other hosts).

Choosing reasonable defaults might not be a trivial problem, but
the current approach of pushing the responsibility to management
layers doesn't improve the situation.


[...]
> > 2) Policies
[...]
> > 3) Abstracting the XML
[...]
> > 4) Identifying devices properly
[...]
> > 5) Generating the right XML snippet for device hot-(un)plug
[...]

These parts are trickier and I need to read the discussion more
carefully before replying.

-- 
Eduardo

On Wed, Mar 21, 2018 at 03:00:41PM -0300, Eduardo Habkost
wrote:
> On Tue, Mar 20, 2018 at 03:10:12PM +0000, Daniel P. Berrangé wrote:
> > On Tue, Mar 20, 2018 at 03:20:31PM +0100, Martin Kletzander wrote:
> > > 1) Default devices/values
> > > 
> > > Libvirt itself must default to whatever values there were before
any
> > > particular element was introduced due to the fact that it strives
to
> > > keep the guest ABI stable.  That means, for example, that it
can't just
> > > add -vmcoreinfo option (for KASLR support) or magically add the
pvpanic
> > > device to all QEMU machines, even though it would be useful, as
that
> > > would change the guest ABI.
> > > 
> > > For default values this is even more obvious.  Let's say
someone figures
> > > out some "pretty good" default values for various
HyperV enlightenment
> > > feature tunables.  Libvirt can't magically change them, but
each one of
> > > the projects building on top of it doesn't want to keep that
list
> > > updated and take care of setting them in every new XML.  Some
projects
> > > don't even expose those to the end user as a knob, while
others might.
> > 
> > This gets very tricky, very fast.
> > 
> > Lets say that you have an initial good set of hyperv config
> > tunables. Now sometime passes and it is decided that there is a
> > different, better set of config tunables. If the module that is
> > providing this policy to apps like OpenStack just updates itself
> > to provide this new policy, this can cause problems with the
> > existing deployed applications in a number of ways.
> > 
> > First the new config probably depends on specific versions of
> > libvirt and QEMU,  and you can't mandate to consuming apps which
> > versions they must be using.  [...]
> 
> This is true.
> 
> >                       [...]  So you need a matrix of libvirt +
> > QEMU + config option settings.
> 
> But this is not.  If config options need support on the lower
> levels of the stack (libvirt and/or QEMU and/or KVM and/or host
> hardware), it already has to be represented by libvirt host
> capabilities somehow, so management layers know it's available.
> 
> This means any new config generation system can (and must) use
> host(s) capabilities as input before generating the
> configuration.
I don't think it is that simple. The capabilities reflect what the
current host is capable of only, not whether it is desirable to
actually use them. Just because a host reports that it has q35-2.11.0
machine type doesn't mean that it should be used. The mgmt app may
only wish to use that if it is available on all hosts in a particular
grouping. The config generation library can't query every host directly
to determine this. The mgmt app may have a way to collate capabilities
info from hosts, but it is probably then stored in a app specific
format and data source, or it may just ends up being a global config
parameter to the mgmt app per host.

There have been a number of times where a feature is available in
libvirt and/or QEMU, and the mgmt app still doesn't yet may still
not wish to use it because it is known broken / incompatible with
certain usage patterns. So the mgmt app would require an arbitrarily
newer libvirt/qemu before considering using it, regardless of
whether host capabilities report it is available.
> > Even if you have the matching libvirt & QEMU versions, it is not
> > safe to assume the application will want to use the new policy.
> > An application may need live migration compatibility with older
> > versions. Or it may need to retain guaranteed ABI compatibility
> > with the way the VM was previously launched and be using transient
> > guests, generating the XML fresh each time.
> 
> Why is that a problem?  If you want live migration or ABI
> guarantees, you simply don't use this system to generate a new
> configuration.  The same way you don't use the "pc"
machine-type
> if you want to ensure compatibility with existing VMs.
In many mgmt apps, every VM potentially needs live migration, so
unless I'm misunderstanding, you're effectively saying don't ever
use this config generator in these apps.
> > The application will have knowledge about when it wants to use new
> > vs old hyperv tunable policy, but exposing that to your policy module
> > is very tricky because it is inherantly application specific logic
> > largely determined by the way the application code is written.
> 
> We have a huge set of features where this is simply not a
> problem.  For most virtual hardware features, enabling them is
> not even a policy decision: it's just about telling the guest
> that the feature is now available.  QEMU have been enabling new
> features in the "pc" machine-type for years.
> 
> Now, why can't higher layers in the stack do something similar?
> 
> The proposal is equivalent to what already happens when people
> use the "pc" machine-type in their configurations, but:
> 1) the new defaults/features wouldn't be hidden behind a opaque
>    machine-type name, and would appear in the domain XML
>    explicitly;
> 2) the higher layers won't depend on QEMU introducing a new
>    machine-type just to have new features enabled by default;
> 3) features that depend on host capabilities but are available on
>    all hosts in a cluster can now be enabled automatically if
>    desired (which is something QEMU can't do because it doesn't
>    have enough information about the other hosts).
> 
> Choosing reasonable defaults might not be a trivial problem, but
> the current approach of pushing the responsibility to management
> layers doesn't improve the situation.
The simple cases have been added to the "pc" machine type, but
more complex cases have not been dealt with as they often require
contextual knowledge of either the host setup or the guest OS
choice.

We had a long debate over the best aio=threads,native setting for
OpenStack. Understanding the right defaults required knowledge about
the various different ways that Nova would setup its storage stack.
We certainly know enough now to be able to provide good recommendations
for the choice, with perf data to back it up, but interpreting those
recommendations still requires the app specific knowledge about its
storage mgmt approach, so ends up being code dev work.

Another case is the pvpanic device - while in theory that could
have been enabled by default for all guests, by QEMU or a config
generator library, doing so is not useful on its own. The hard
bit of the work is adding code to the mgmt app to choose the
action for when pvpanic triggers, and code to handle the results
of that action.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|