Ovirt devel - Apr 2009 - Unification of cobbler/cloud/deployment APIs / other deployment topics

Hi folks!

I talked a little bit about this already with Hugh and Iain, but figured 
it would be nice to bring up on the list.  

One thing a lot of us are interested in is a common "deployment" 
building block between various OSS management applications.

Ideally I'd like to be able to go to the same place and say

"add a distro"
"now add a profile"
"install this on physical X"
"on pool of machines Y, add a guest Z".

Basically this means the entire model of the deployment is stored in the 
deployment server, all in one place, one system of objects, one API, etc.

Right now it's stored in a mixture of OVirt API and Cobbler API, so an 
application needs to install both to do basic deployment tasks in 
"cloud" context, and we can't use
the full deployment server API to query things about the deployment 
infrastrucutre... nor the ovirt API ... because things are in a mixture 
of each.  Kind of messy.

I'm starting to work some of this into cobbler's development branch, but
it currently doesn't know about storage pools.  We need to fix this, for 
starters.

Long term, my strategy is:
  * Teach cobbler about libvirt-qmf (and also provide an objecty QMF 
interface to cobblerd, including events for things changing in cobbler)
  * have cobbler "deploy" API calls send out commands via libvirt-qmf,
so koan is required only for legacy installations
  * teach cobler to, at minimum, remember the name of each storage pool 
a machine is attached to.
  * have cobbler "deploy" support primative concepts of cloud
deployment
(deploy into this pool and keep track of which machine is deployed 
where).  Right now it thinks
     in terms of "virt groups" not storage pools.

This is done by creating a Cobbler system record for each host, and a 
system record for each guest, and then using the deploy API call to 
instantiate a guest on each system.

So, what I'm basicaly proposing is we move the "cloud deployment
API"
backend at the lowest levels into cobblerd, so that the workflow can 
encourage consistent
usage of cobbler objects.    Obviously I don't want to own all of this 
or do it myself and would like to see lots of contributions and upgrades 
to what we have now to get us there --
but it would be hugely beneficial to have services we could talk to for 
deployment, versus having to talk to full-stack applications -- at least 
for installation purposes.

I don't want to approach anything resembling monitoring, configuration 
management, definining the nodes, or so forth -- only to be able to say 
"install this system foo into this group
of systems bar", and get Ovirt to use that API to do it so we can have 
an API in common.

The idea is that for a local cloud, the API to deploy something point to 
point, or into the cloud, should be the same API, and that API callers 
shouldn't have to use two APIs (Cobbler+OVirt)
to get what they want arranged/done.

For instance, right now, spacewalk does a pretty good job of creating 
Distros, Profiles, and Systems for each things it wants to manage .... 
OVirt primarily
seems to create image records, but the cloud API doesn't know about 
systems.   Beaker does it a little different, so does Genome.  The goal 
is to make sure the deployment
server is fully featured enough so that everything can go through that 
central pipe, and where it's not, we can add to the deployment server to 
make it doable.

What I'm looking for is standardization around the various ways the apps 
use the deployment API, and making "cloud" part of the deployment API.

Anyway, please take a look at 
https://fedorahosted.org/cobbler/wiki/DeployFeature

Right now this is /rather/ primative, so I'm hoping some folks like Ian 
can offer some suggestions.   Making a storage pool object in cobbler 
seems to be a requirement for sure, so we
can point systems at the storage pools.

Heuristics for where to deploy would probably be pluggable.  Right now 
the easiest thing to do is "system with least amount of deployed VMs",
somewhat like the current round robin impl, but
we should also make it pluggable to say "ask ovirt where this thinks it 
should go".  

The other thing I'd like to see is to get OVirt to create cobbler system 
records for each VM.   Once we start creating system records for each VM 
(as well as each host) we can start enabling
things like using the Provisioning API to manage DHCP, DNS, and physical 
host power (via fence), and virt power -- you can, for instance, use the 
API to power cycle a given system by
looking it up via hostname -- without knowing whether it's physical or 
virtual, etc.

Another topic we should probably talk about is what kind of events we 
might want the deployment server to transmit, as well as what types of 
commands we want it to send.

Right now, I'm planning on doing a QMF API that is mostly like the 
Python API (so there should be few suprises), with basic events that say 
when a cobbler object is edited or when a command completes (like a sync 
or import), much like the triggers system today.

Comments welcome!

--Michael