Puppet users - Feb 2007 - What I''m doing with Puppet

Hi all,

Just reading through Luke''s interview on computerworld.com.au and I
came across this statement:

"This topic comes up on the list periodically, however, and most
people are very tight-lipped about what they''re doing with
Puppet."

Being the chattermouth that I am, I''d like to take this opportunity to
tell you all what we''re (that is, my company, Cidev, based in The
Netherlands) doing with Puppet and why we like it. Even though I''ve
only started using it about a week ago...

Our new customer (which I''m not going to name at this time, but you
probably haven''t heard of him, because it''s a pure b2b
company) needed
a high-available setup, consisting of 14 servers and providing several
services like web, servlets, database, you name it. Think about your
ideal web platform, add some proprietary stuff and that''s basically
it. Required uptime: 99.9999999% (we only offered 99.99% though).

In the years doing service maintenance we''ve learned the power of
virtualisation. Even when a server only provides 1 service, we put it
in a virtualised environment. It helps a lot in maintenance and
general "stuff you can do at a distance". I.e. updating kernels, if
you want to be really sure it doesn''t give you downtime, requires a
test setup and usually your direct presence or remote console stuff.
With Xen (the virtualisation we use) we can do it simply from the
dom0. There''s more, but, since Puppet is a sysadmin tool and I expect
most of you to be sysadmins, you can probably think of some stuff
yourself.

But, that also means you at least duplicate the number of servers you
maintain. So 28 servers needed to be maintained. We splitted some
services onto their own virtual domain, for safety, so that can amount
quickly to 35~40 servers in the end. Ow and they have to be ready
yesterday. Yeah, we know you had to start two weeks late because the
machines were delivered late and Internet wasn''t available for two
week, but we still got our deadlines. Well... needless to say, not
only did we need a convenient way to adminster the servers, we also
needed fast deployment.

Now, xen-create-image (on Debian at least) supports roles, which are
basically scripts that can do some stuff after installation to fix up
the installation the way you want it. Great. It took me almost two
days to create a role that did exactly what I wanted, create a basic
installation the way we do it. Then I switched to puppet. I learned
puppet and recreated the basic installation script in one day. And now
we got advanced config replication foo. That''s important since
high-availability means you have several servers configged exactly the
same.

A deadline passed today and I''m quite sure I only made it because of
the use of Puppet. It still took me to 3.30am last night, but once I
had one server running, I only needed to add the script to the other
servers which were supposed to be about the same. Done. Loadbalancing,
webserver, HA-NFS working.

Thank you for Puppet!

Now, for my wishlist, Santa! What I would like to see support for in Puppet:
- LVM (I need to create large amounts of partitions, would be great if
I could do that centrally and no, sorry, the RH conga framework
doesn''t work for me)
- DRBD (if we can make the partitions, we can network-raid-1 them...)
- Xen domU (just a convenient way to create domU *configs*, not create
the entire image, since I guess that would be a bit difficult)
- Xen domU (the entire image... yeah, it''s a wishlist so why not)
- apt-get --purge option
- apt-get dist-upgrade (preferably on demand only)
- report-back framework (oh noes! client 375 had an error while trying
to run manifest Foo, let''s have the puppetmaster mail the Sysadmin
Donkey so he can fix it)
- nagios integration (I can dream, can''t I?)
- support for something like:

exec { "mycommand with some arguments":
  unless => file { "/tmp/idonotexists": test => exists }
}

-- 
Kind regards,
Tim

2007/2/6, Tim Stoop <tim.stoop@gmail.com>:
>
> Hi all,
>
> [snip]
> Now, for my wishlist, Santa! What I would like to see support for in
> Puppet:
> [snip]
> - support for something like:
>
> exec { "mycommand with some arguments":
>   unless => file { "/tmp/idonotexists": test => exists }
> }
>

Sorry, but maybe my english is confusing me... isn''t this the same as
the
creates parameter in the exec type? [1]

[1] http://reductivelabs.com/projects/puppet/reference/typedocs.html#exec

Best regards
Jose


_______________________________________________
Puppet-users mailing list
Puppet-users@madstop.com
https://mail.madstop.com/mailman/listinfo/puppet-users

On Tue, Feb 06, 2007 at 11:22:45AM +0100, Tim Stoop
wrote:
> - DRBD (if we can make the partitions, we can network-raid-1 them...)
I think that DRBD would require some inter-server orchestration support,
if it''s to be done fully. For example, you wouldn''t want to
re-newfs a
filesystem whilst it''s still mounted by another client.

Admittedly, i think that''d be more likely to appear as a layer on top
of
puppet, rather than in puppet itself  in the near future.
> - report-back framework (oh noes! client 375 had an error while trying
> to run manifest Foo, let''s have the puppetmaster mail the Sysadmin
> Donkey so he can fix it)
That''s already there--I''m reasonably sure that the tagmail
report does
that. (See http://reductivelabs.com/projects/puppet/reference/reports.html )
> - nagios integration (I can dream, can''t I?)
You could do something similar using virtual resources, collections and
templating, I think. It''s not quite as good as native support,
I''ll
admit :)
> - support for something like:
> exec { "mycommand with some arguments":
>   unless => file { "/tmp/idonotexists": test => exists }
> }
I think I''d second that. Especially if they could be anonymous.

-- 
Ceri Storey <cez@necrofish.org.uk>
''What I really want is "apt-get smite"'' 
    --Rob Partington
http://unix.culti.st/

Hi José,

On 2/6/07, José González Gómez <jgonzalez.openinput@gmail.com>
wrote:
> Sorry, but maybe my english is confusing me... isn''t this the same
as the
> creates parameter in the exec type? [1]
Only if the script actually creates the file. But I admit, my example
wasn''t fully clear. I meant more the complete test-structure. To be
able to check if a file is a link to another file, to be able to check
if a file contains a certain string, etc. etc. At least a bit more
tests then there are now.

What I''ve been doing recently is checking if a certain file is a link
and if it points to the right file. For an example, look at my blog:
<http://cvd.cidev.nl/wordpress/2007/02/01/puppet-debian-and-update-alternatives/>

Would be cool if I could forgo the shell tests for something more substantial.

-- 
Gegroet,
Tim

Hi Ceri,

On 2/6/07, Ceri Storey <cez@necrofish.org.uk>
wrote:
> I think that DRBD would require some inter-server orchestration support,
> if it''s to be done fully. For example, you wouldn''t want
to re-newfs a
> filesystem whilst it''s still mounted by another client.
True, but I don''t think there are a lot of those use cases. I want to
be able to say something like "These two servers are going to be a
DRBD pair, one being primary, one being secondary. I want server A to
use /dev/sda6 and server B to use /dev/hda3, which should both be the
same size. I want to use /dev/sda8 on server A for the meta-data and
on server B I want to use /dev/hdb3. Have the rate between them be
80Mbyte/sec max. On split-brain situations with 2 secondaries do X, on
split-brain situations with 1 primary, do Y.... etc. etc. etc.".

Puppet should check if the filesystems are available for DRBD and if
so, create the config. After the config is created, it''s simply a
"drbdadm create-md configN; /etc/init.d/drbd restart" on both nodes.
Don''t mount or do anything else. A separate fstab handler should
create the fs or something? Don''t know about that part.

Honestly, it''s not like I really thought this through, I''m
only giving
a summary of the stuff for which I still needed to log in on my other
machines :)
> That''s already there--I''m reasonably sure that the
tagmail report does
> that. (See http://reductivelabs.com/projects/puppet/reference/reports.html
)
Yeah, they told me that on IRC tonight, but HOW? I''m a stupid donkey,
I need examples to steal... How do I make sure a tag only shows up
when that condition applies? For example, to hook into the previous
subject, I want to check if DRBD is installed. Don''t install it
automagically, just check for it. If so, great. If not, email me to
tell me my node isn''t ready for production yet.
> You could do something similar using virtual resources, collections and
> templating, I think. It''s not quite as good as native support,
I''ll
> admit :)
Luke pointed me to Naginator recently. I looked at it but it still
needs a lot of work. It''s still cfengine only :( That''s a
great
project. Tweaking Nagios is one of the things that really takes up a
lot of my time and I dislike it very much. I know, I should put some
effort into Naginator then, but I''m afraid I''m a bad
programmer :|

-- 
Gegroet,
Tim

On Tue, Feb 06, 2007 at 02:02:27PM +0100, Tim Stoop
wrote:
> Hi Ceri,
> 
> On 2/6/07, Ceri Storey <cez@necrofish.org.uk> wrote:
> > I think that DRBD would require some inter-server orchestration
support,
> > if it''s to be done fully. For example, you wouldn''t
want to re-newfs a
> > filesystem whilst it''s still mounted by another client.
> 
> True, but I don''t think there are a lot of those use cases. 
I''m imagining that it''ll have a wide number of use cases,
myself. For
example, when one wants to upgrade an application in production, you''d
want to ensure N out of M servers are taken out of a load balancer, say
(something not really supported by puppet :), and only then would you
apply someething like "package { bigapp: ensure => latest }". Then,
once
they''re all done, test, then bring them back into service.


I''d imagine that this would be a similar pattern to the current
directed
acyclic graph that puppet uses, where we specify what we want at the
verteces, but you could annote the edges with preconditions (ie: take
this path if <foo> is the case). 

IIRC something like this was discussed on lssconf-discuss a while back.
I''d really need to go-back and re-read it, although I might need to
brush up on my formal logic first. (I think the example given was for N
out of M tuners must always be in service, but you need to upgrade
them). 



-- 
Ceri Storey <cez@necrofish.org.uk>
''What I really want is "apt-get smite"'' 
    --Rob Partington
http://unix.culti.st/

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Tuesday 06 February 2007 14:02, Tim Stoop wrote:
> > You could do something similar using virtual resources, collections
and
> > templating, I think. It''s not quite as good as native
support, I''ll
> > admit :)
>
> Luke pointed me to Naginator recently. I looked at it but it still
> needs a lot of work. It''s still cfengine only :( That''s a
great
> project. Tweaking Nagios is one of the things that really takes up a
> lot of my time and I dislike it very much. I know, I should put some
> effort into Naginator then, but I''m afraid I''m a bad
programmer :|
I''m also currently hacking together config file snippets by
postprocessing
virtual Files. 

Basically on each "source" system you do 

@@File { "/var/local/collection_directory/$fqdn": content =>
template(foo) }

and on the "destination":

File <<||>>
file{ "/var/local/collection_directory": checksum => mtime, notify
=>
Exec[create_config] }

where the Exec[create_config] uses the files within the collection_directory 
to build something. Depending on the config file capabilities you can use 
directory includes (think puppets "import
/var/local/collection_directory/*")
or direct access into the directory (think exim''s dsearch) instead.



Regards, David
- -- 
- - hallo... wie gehts heute?
- - *hust* gut *rotz* *keuch*
- - gott sei dank kommunizieren wir über ein septisches medium ;)
 -- Matthias Leeb, Uni f. angewandte Kunst, 2005-02-15
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFFyJX5/Pp1N6Uzh0URAoaOAJ0QxqQYItY7uLvPKrVM4/kUwrPYmwCfcRFo
hVszkOkOULnZs4BiM7BNs5I=f5oG
-----END PGP SIGNATURE-----

On Feb 6, 2007, at 4:22 AM, Tim Stoop wrote:
> Just reading through Luke''s interview on computerworld.com.au and
I
> came across this statement:
>
> "This topic comes up on the list periodically, however, and most
> people are very tight-lipped about what they''re doing with
Puppet."
>
> Being the chattermouth that I am, I''d like to take this
opportunity to
> tell you all what we''re (that is, my company, Cidev, based in The
> Netherlands) doing with Puppet and why we like it. Even though
I''ve
> only started using it about a week ago...
[snip]

Hi Tim,

Thanks for sending this out; it''s very helpful knowing what
you''re
doing.  Your post inspired me to create a "Testimonials" page:

https://reductivelabs.com/trac/puppet/wiki/Testimonials

A similar page has worked out well for Nagios, and I hope we can  
gather enough testimonials to make the nature of Puppet''s community  
clear.

  --
  The Roman Rule
          The one who says it cannot be done should never interrupt the
          one who is doing it.
  ---------------------------------------------------------------------
  Luke Kanies | http://reductivelabs.com | http://madstop.com

On Feb 6, 2007, at 7:02 AM, Tim Stoop wrote:
> Hi Ceri,
>
> On 2/6/07, Ceri Storey <cez@necrofish.org.uk> wrote:
>> I think that DRBD would require some inter-server orchestration  
>> support,
>> if it''s to be done fully. For example, you wouldn''t
want to re-
>> newfs a
>> filesystem whilst it''s still mounted by another client.
>
> True, but I don''t think there are a lot of those use cases. I want
to
> be able to say something like "These two servers are going to be a
> DRBD pair, one being primary, one being secondary. I want server A to
> use /dev/sda6 and server B to use /dev/hda3, which should both be the
> same size. I want to use /dev/sda8 on server A for the meta-data and
> on server B I want to use /dev/hdb3. Have the rate between them be
> 80Mbyte/sec max. On split-brain situations with 2 secondaries do X, on
> split-brain situations with 1 primary, do Y.... etc. etc. etc.".
The basic setup should probably be possible with Puppet, but any kind  
of inter-node checkpointing will require some kind of layer above the  
current system.

I expect that layer to exist at some point, but I''m not exactly in a  
hurry to write it.
> Puppet should check if the filesystems are available for DRBD and if
> so, create the config. After the config is created, it''s simply a
> "drbdadm create-md configN; /etc/init.d/drbd restart" on both
nodes.
> Don''t mount or do anything else. A separate fstab handler should
> create the fs or something? Don''t know about that part.
>
> Honestly, it''s not like I really thought this through,
I''m only giving
> a summary of the stuff for which I still needed to log in on my other
> machines :)
This should probably be a wiki page, too:  Stuff I can''t do with
Puppet.
>> That''s already there--I''m reasonably sure that the
tagmail report
>> does
>> that. (See http://reductivelabs.com/projects/puppet/reference/ 
>> reports.html )
>
> Yeah, they told me that on IRC tonight, but HOW? I''m a stupid
donkey,
> I need examples to steal... How do I make sure a tag only shows up
> when that condition applies? For example, to hook into the previous
> subject, I want to check if DRBD is installed. Don''t install it
> automagically, just check for it. If so, great. If not, email me to
> tell me my node isn''t ready for production yet.
That''s more a question of generating the alerts than of creating a  
report from them.

If DRBD is installed via a packaging system, then use noop:

package { drbd: ensure => installed, noop => true, tag => drbd }

Then just set up a report for that tag:

drbd: me@mystuff.com

That''s a pretty trivial example, and you probably aren''t going
to
want many per-application reports like this, but at least it gives  
you the basic idea.

If you''re not using a packaging system...  Right now, you''ll
probably
have to hack it, unfortunately.  I need to copy the checks from exec  
to notify, so that you could do something like this:

   notify { "your stuff is broken": unless =>
"/bin/stuff_is_working" }

But I seem to not quite have enough hours in the day.
>> You could do something similar using virtual resources,  
>> collections and
>> templating, I think. It''s not quite as good as native support,
I''ll
>> admit :)
>
> Luke pointed me to Naginator recently. I looked at it but it still
> needs a lot of work. It''s still cfengine only :( That''s a
great
> project. Tweaking Nagios is one of the things that really takes up a
> lot of my time and I dislike it very much. I know, I should put some
> effort into Naginator then, but I''m afraid I''m a bad
programmer :|
The most useful part of naginator is the configuration parser/ 
generator.  It shouldn''t take too much work to use that to create  
Puppet types.

But, as David Schmitt points out, you can always fake it with file  
snippets for now.

  --
  It''s very hard to predict things . . . Especially the future.
                  -- Prof. Charles Kelemen, Swarthmore CS Dept.
  ---------------------------------------------------------------------
  Luke Kanies | http://reductivelabs.com | http://madstop.com

On Tue, 2007-02-06 at 11:22 +0100, Tim Stoop wrote:
> Now, xen-create-image (on Debian at least) supports roles, which are
> basically scripts that can do some stuff after installation to fix up
> the installation the way you want it. Great. It took me almost two
> days to create a role that did exactly what I wanted, create a basic
> installation the way we do it. Then I switched to puppet. I learned
> puppet and recreated the basic installation script in one day. And now
> we got advanced config replication foo. That''s important since
> high-availability means you have several servers configged exactly the
> same.
This is really interesting stuff; I am _very_ interested in using puppet
to manage virtual machines and ultimately distribute appliances based on
puppet.
> - LVM (I need to create large amounts of partitions, would be great if
> I could do that centrally and no, sorry, the RH conga framework
> doesn''t work for me)
That seems to be a fairly common request lately; can you elaborate a
little more on your requirements for this ?
> - Xen domU (just a convenient way to create domU *configs*, not create
> the entire image, since I guess that would be a bit difficult)
That''s pretty reasonable (though I would base that off libvirt, since
it
[will] give you an immediate solution for other virt backends)
> - Xen domU (the entire image... yeah, it''s a wishlist so why not)
I am not sure that puppet is the right tool for this - this seems
inherently like an action you want to perform, not so much something
that you want to keep in sync with a definition.

David

On 2/6/07, Luke Kanies <luke@madstop.com> wrote:
> > Honestly, it''s not like I really thought this through,
I''m only giving
> > a summary of the stuff for which I still needed to log in on my other
> > machines :)
>
> This should probably be a wiki page, too:  Stuff I can''t do with
Puppet.
Well, this raises another question: How do you actually want to deploy
Puppet? Should it be possible to install a node, in the clean install
add a puppet client, do some cert-signing-foo and you have a new
machine? Or should the installation be part of the stuff you do
manually and only use puppet for maintaining it after the initial
installation is done?

I''d prefer to do the first, but hey, I have no say it in because I
don''t program it :) What is your take on this?
> The most useful part of naginator is the configuration parser/
> generator.  It shouldn''t take too much work to use that to create
> Puppet types.
Ah well, maybe a good point in time to learn ruby. After I''m done with
the installation of the current assignment.

-- 
Gegroet,
Tim

On 2/6/07, David Lutterkort <dlutter@redhat.com>
wrote:
> This is really interesting stuff; I am _very_ interested in using puppet
> to manage virtual machines and ultimately distribute appliances based on
> puppet.
Yeah, that was my take on it too. Debian can install based on
preseeding (I think this is called kickstart in RH and Suse?), so when
you need a certain type of server, you make a seed and it''s a
fire-and-forget thing. Make a netboot image that boots an install CD
and points to the preseeding and you''re done. Making a preseed thingy
is a lot of work, though.

Besides, the problem with this is that you''re always tweaking the
installs. At least, that''s what we do. If you want to keep the
convenience of the preseeding, you''ll need to incorporate all your
tweaks in the preseed-config. Which amounts to doing the work twice
(first on the live servers, then in the preseed config). Waste of
time. I was hoping for puppet to fill in the gap and it does for a
large part. At least with puppet I can create a preseed config and
have puppet update it to the latest production environment used,
including the latest tweaks.

I''d prefer a solution that did it all, though, since in my mind, it
isn''t all that different. But hey, maybe I have a strange mind.
> > - LVM (I need to create large amounts of partitions, would be great if
> > I could do that centrally and no, sorry, the RH conga framework
> > doesn''t work for me)
>
> That seems to be a fairly common request lately; can you elaborate a
> little more on your requirements for this ?
Sure. As I said before, I''m abusing puppet mostly to create the
appliance I need from a default working installation. We usually
create two different VG during the installation, dom0 and domU. The
dom0 VG contains the LVs used for dom0 and the domU contains all the
LVs used by the domUs, prefixed with the hostname of the domU. So if I
create a tmp partition for the domU "donkey", I''d issue a:

lvcreate -L 500M -n donkey-tmp domU

Creating, formatting and mounting those LVs takes a lot of time, while
I can easily describe up front what I want. (We have 4 partitions by
default, root, swap, tmp and var, sometimes more depending on the
service delivered by the domU, we only use ext3 and we want each LV
repartitioned at creation, after that formatting is out of the
question.) I expect machines running 4 or more domUs with a total of
1T diskspace or more. Formatting, even without a badblocks check,
takes a lot of time then. Doing something else while it''s formatting
is an option, but you''ll always do less of a job because you keep one
eye on your installation or you forget about the installation, which
is then idle for longer than you could handle. I want to speed things
up.

Another thing is resizing. I think it should be fairly easy for a LVM
to be resized. At least made bigger. Both LVM and ext3 can handle that
without a problem. Would be nice if a provider handles that on the
fly.

"What? Partition X on those three redundant, fail-over, DRBD-connected
machines is almost full? No problem, mate, let me just change this
100G in this single file to 500G and we''re ready to go."  --
That''s
almost a sysadmin''s wet dream. (Almost, mind you.)
> > - Xen domU (just a convenient way to create domU *configs*, not create
> > the entire image, since I guess that would be a bit difficult)
>
> That''s pretty reasonable (though I would base that off libvirt,
since it
> [will] give you an immediate solution for other virt backends)
Don''t know libvirt, but if it''s portable, that''s a
very good solution.
> > - Xen domU (the entire image... yeah, it''s a wishlist so why
not)
>
> I am not sure that puppet is the right tool for this - this seems
> inherently like an action you want to perform, not so much something
> that you want to keep in sync with a definition.
I totally agree with you that it''s an action and not something to
sync. Still, pointing at the way that we''re (ab)using puppet, I
don''t
think it''s unreasonable.

Also, the way we''re using puppet, like a deployment tool, seems an
obvious spin-off to me. The point of having all your configuration in
one place for me is have a clear overview of your complete setup. If
you''re using a lot of virtual servers, this information is important
for that overview...

I was wondering out loud a few days ago if it would be wise if puppet
new of the concept and relation between machine <-> dom0 <-> domU.
Couldn''t say why or how or something, though. Just a gut feeling.

-- 
Gegroet,
Tim

On Tuesday 06 February 2007 23:29, David Lutterkort
wrote:
> On Tue, 2007-02-06 at 11:22 +0100, Tim Stoop wrote:
> > Now, xen-create-image (on Debian at least) supports roles, which are
> > basically scripts that can do some stuff after installation to fix up
> > the installation the way you want it. Great. It took me almost two
> > days to create a role that did exactly what I wanted, create a basic
> > installation the way we do it. Then I switched to puppet. I learned
> > puppet and recreated the basic installation script in one day. And now
> > we got advanced config replication foo. That''s important
since
> > high-availability means you have several servers configged exactly the
> > same.
>
> This is really interesting stuff; I am _very_ interested in using puppet
> to manage virtual machines and ultimately distribute appliances based on
> puppet.
I have a simple xen-create-image role I cloned from one of the default roles 
to install puppet after creating the image. Since I didn''t want puppetd
to
start in the chroot, there is also some debian magic to suppress that. See 
attachment.


Regards, David
-- 
- hallo... wie gehts heute?
- *hust* gut *rotz* *keuch*
- gott sei dank kommunizieren wir über ein septisches medium ;)
 -- Matthias Leeb, Uni f. angewandte Kunst, 2005-02-15



_______________________________________________
Puppet-users mailing list
Puppet-users@madstop.com
https://mail.madstop.com/mailman/listinfo/puppet-users

On Thu, Feb 08, 2007 at 01:23:58AM +0100, Tim Stoop
wrote:
> On 2/6/07, David Lutterkort <dlutter@redhat.com> wrote:
> > This is really interesting stuff; I am _very_ interested in using
puppet
> > to manage virtual machines and ultimately distribute appliances based
on
> > puppet.
> 
> Yeah, that was my take on it too. Debian can install based on
> preseeding (I think this is called kickstart in RH and Suse?), so when
> you need a certain type of server, you make a seed and it''s a
> fire-and-forget thing. Make a netboot image that boots an install CD
> and points to the preseeding and you''re done. Making a preseed
thingy
> is a lot of work, though.
That''s why you use a LinuxCOE image instead.  Does most of the hard
work for
you.  If you ask nicely I might share one of the tweaked ones I''ve got,
too.
<grin>
> Besides, the problem with this is that you''re always tweaking the
> installs. At least, that''s what we do. If you want to keep the
> convenience of the preseeding, you''ll need to incorporate all your
> tweaks in the preseed-config. Which amounts to doing the work twice
> (first on the live servers, then in the preseed config). Waste of
> time. I was hoping for puppet to fill in the gap and it does for a
> large part. At least with puppet I can create a preseed config and
> have puppet update it to the latest production environment used,
> including the latest tweaks.
My process, which I''ve never regretted instituting, is that the
automated
install or xen-create-image run does *nothing* that isn''t needed to get
Puppet running, basically.  If I can run Puppet on the machine, then the
install is done.  From there, Puppet takes over.  I''ve got two
installation
CD images now (each 22MB), and the only difference is the partitioning
scheme I''ve got on them.  There''s only one "flavour"
of Xen installation
process, too.  So I''ve got three different ways to install machines,
and I
haven''t had to do any maintenance on them in months.  Everything else
is
Puppet''s problem.

- Matt

On Thu, 2007-02-08 at 01:23 +0100, Tim Stoop wrote:
> On 2/6/07, David Lutterkort <dlutter@redhat.com> wrote:
> > This is really interesting stuff; I am _very_ interested in using
puppet
> > to manage virtual machines and ultimately distribute appliances based
on
> > puppet.
> 
> Yeah, that was my take on it too. Debian can install based on
> preseeding (I think this is called kickstart in RH and Suse?), so when
> you need a certain type of server, you make a seed and it''s a
> fire-and-forget thing. Make a netboot image that boots an install CD
> and points to the preseeding and you''re done. Making a preseed
thingy
> is a lot of work, though.
I don''t know much about Debian, but my understanding is that one
important difference between provisioning Debian and RedHat/Suse guests
is that Debian requires running Debian-specific code (debootstrap) in
the dom0, which of course works better if dom0 is running Debian. It''s
making running a paravirt Debian guest on a non-Debian host somewhat
interesting.

As for kickstart: it''s entirely driven through a text file; writing
them
isn''t too much work. The hardest part is getting a good handle on
selecting packages and making sure you don''t pull in stuff you
don''t
want, and that''s more a distro/dependency problem. You can use
something
like cobbler (http://cobbler.et.redhat.com/) to make the whole
provisioning process very painless.
> Besides, the problem with this is that you''re always tweaking the
> installs.
Yeah, people commit similar crimes with kickstart; it''s quite common to
have a kickstart file with lots and lots of post-install scripts
embedded in them. I would much prefer if we could get people to use
puppet to do that; it''s a little awkward that you need to deal with two
tools to fully provision a machine though (kickstart and puppet) I''d be
very interested in ideas how to remedy that.
> I''d prefer a solution that did it all, though, since in my mind,
it
> isn''t all that different.
Exactly. One tool to rule them all ;)
> Sure. As I said before, I''m abusing puppet mostly to create the
> appliance I need from a default working installation. We usually
> create two different VG during the installation, dom0 and domU. The
> dom0 VG contains the LVs used for dom0 and the domU contains all the
> LVs used by the domUs, prefixed with the hostname of the domU.
Is that domU vg initialized from dom0 or from domU ?
> Creating, formatting and mounting those LVs takes a lot of time, while
> I can easily describe up front what I want. (We have 4 partitions by
> default, root, swap, tmp and var, sometimes more depending on the
> service delivered by the domU, we only use ext3 and we want each LV
> repartitioned at creation, after that formatting is out of the
> question.) 
That''s one of the difficult issues with managing storage with puppet:
initial provisioning of the storage has to happen before puppet has a
chance to run, so you''d always have to resort to another tool (aka the
installer) 
> Formatting, even without a badblocks check,
> takes a lot of time then. Doing something else while it''s
formatting
> is an option, but you''ll always do less of a job because you keep
one
> eye on your installation or you forget about the installation, which
> is then idle for longer than you could handle. I want to speed things
> up.
Are you more concerned about the time it takes you to babysit the
process or to generally speed up how long the whole install process
takes ? The latter would require some parallelization on puppet''s part.
> Another thing is resizing. I think it should be fairly easy for a LVM
> to be resized. At least made bigger. Both LVM and ext3 can handle that
> without a problem. Would be nice if a provider handles that on the
> fly.
Agreed, and it would probably handle a large percentage of the cases
where you want to modify storage on a running machine.
> > That''s pretty reasonable (though I would base that off
libvirt, since it
> > [will] give you an immediate solution for other virt backends)
> 
> Don''t know libvirt, but if it''s portable, that''s
a very good solution.
It''s at http://www.libvirt.org/ and is part of RH-based distros; AFAIK,
it''s not in Debian, but all that is missing is somebody who understands
Debian packaging to package it - I''ve mentioned that to the libvirt
maintainers, and they are very interested in helping whoever is picking
this up to get there.
> Also, the way we''re using puppet, like a deployment tool, seems an
> obvious spin-off to me. The point of having all your configuration in
> one place for me is have a clear overview of your complete setup. If
> you''re using a lot of virtual servers, this information is
important
> for that overview...
Couldn''t agree more.
> I was wondering out loud a few days ago if it would be wise if puppet
> new of the concept and relation between machine <-> dom0 <->
domU.
> Couldn''t say why or how or something, though. Just a gut feeling.
The dom0/domU relation is a lot more dynamic than what puppet is
supposed to do; I doubt that you would want to contol VM migration
through puppet. One of the advantages of puppet _not_ knowing about that
is that puppet focuses entirely on the ''insides'' of a system
(dom0 or
domU) whereas some other tool is responsible of pushing the domU''s
around as black boxes.

David

On Thu, 2007-02-08 at 01:02 +0100, Tim Stoop wrote:
> Well, this raises another question: How do you actually want to deploy
> Puppet? Should it be possible to install a node, in the clean install
> add a puppet client, do some cert-signing-foo and you have a new
> machine?
That''s what I do with kickstarts [1], and it works well; it has the
advantage that you know that puppet is installed properly on your
machines when the reboot after provisioning.

[1]
http://watzmann.net/blog/index.php/2006/12/05/kickstarting_into_puppet

On 2/9/07, David Lutterkort <dlutter@redhat.com>
wrote:
> > Besides, the problem with this is that you''re always tweaking
the
> > installs.
>
> Yeah, people commit similar crimes with kickstart; it''s quite
common to
> have a kickstart file with lots and lots of post-install scripts
> embedded in them. I would much prefer if we could get people to use
> puppet to do that; it''s a little awkward that you need to deal
with two
> tools to fully provision a machine though (kickstart and puppet)
I''d be
> very interested in ideas how to remedy that.
Well, in my mind, we do local changes because we''re already at that
machine for the install. If we could somehow do the install from the
puppetmaster (either through puppet or some other tool) I think people
are much more inclined to actually stay working in the puppetmaster
files instead of the node.

One way of doing that is netboot installation with some sort of
automagic installation all the way through to puppet (Debian''s
preseeding or RH/Suse kickstart). The netboot image can be on the same
machine as the puppetmaster. That would actually solve a lot of the
initial problems, now I think about it. Maybe we should create a
preseeding script that sets things up until puppet. Shouldn''t be that
much work.
> > I''d prefer a solution that did it all, though, since in my
mind, it
> > isn''t all that different.
>
> Exactly. One tool to rule them all ;)
And only one tree to add to your local puppetmaster SCM repository :)
At least, I hope you people are all putting your /etc and related
configs in some version control thingy... ;-)
> > Sure. As I said before, I''m abusing puppet mostly to create
the
> > appliance I need from a default working installation. We usually
> > create two different VG during the installation, dom0 and domU. The
> > dom0 VG contains the LVs used for dom0 and the domU contains all the
> > LVs used by the domUs, prefixed with the hostname of the domU.
>
> Is that domU vg initialized from dom0 or from domU ?
From dom0. Our domUs don''t want to see any LVM :) We do all the LVM
stuff in the dom0s, so the domUs only need to see actual block
devices.
> That''s one of the difficult issues with managing storage with
puppet:
> initial provisioning of the storage has to happen before puppet has a
> chance to run, so you''d always have to resort to another tool (aka
the
> installer)
Yeah, but only for the initial installation. I don''t mind having to
manually install a dom0, but it would be soooo nice to have a tool
that can do the rest of the LVM partitioning from one centrally
maintained recipe.
> Are you more concerned about the time it takes you to babysit the
> process or to generally speed up how long the whole install process
> takes ? The latter would require some parallelization on puppet''s
part.
Both, actually. The latter, mostly. If I can setup stuff quickly, my
customers are happy puppies. If I don''t have to babysit, I can spend
time on other stuff and that makes my customers even happier...
("Finally you found the time to fix our corporate firewall! Thank you!
Now excuse me, I''ve got three months of email backlog to filter
through...")
> Agreed, and it would probably handle a large percentage of the cases
> where you want to modify storage on a running machine.
I''ve never had to resize a partition to something smaller. We leave
room to spare in each of our installation. If I ever need to resize
smaller, I''m probably screwing so much with the installation, I
don''t
*want* to do it automagically.
> It''s at http://www.libvirt.org/ and is part of RH-based distros;
AFAIK,
> it''s not in Debian, but all that is missing is somebody who
understands
> Debian packaging to package it - I''ve mentioned that to the
libvirt
> maintainers, and they are very interested in helping whoever is picking
> this up to get there.
I can look into that this weekend, I hope. Debian packages aren''t that
difficult once you know how to make them :)
> The dom0/domU relation is a lot more dynamic than what puppet is
> supposed to do; I doubt that you would want to contol VM migration
> through puppet. One of the advantages of puppet _not_ knowing about that
> is that puppet focuses entirely on the ''insides'' of a
system (dom0 or
> domU) whereas some other tool is responsible of pushing the domU''s
> around as black boxes.
Well, we don''t really use migration of domUs yet. Migration gets kinda
useless if your machine breaks down. We deploy fail-over with
heartbeat and DRBD or something where we need high-availability, don''t
really see a use for migration in that. I might be wrong, though, I''m
the suit of our little company, my coworker is the one with the
technical knowledge :) Migration is nice for scheduled maintenance,
but we use fail-over for that too.

But, as I said, it''s only a gut feeling. I''m probably wrong.
The only
thing I can think of right now in which case it would be convenient is
when you''re using the puppet manifests as a central documentation for
your network (see other mails about documentation). Something that''s
high on my priority list too...

-- 
Gegroet,
Tim

> Yeah, people commit similar crimes with kickstart; it''s quite
common to
> have a kickstart file with lots and lots of post-install scripts
> embedded in them. I would much prefer if we could get people to use
> puppet to do that; it''s a little awkward that you need to deal
with two
> tools to fully provision a machine though (kickstart and puppet)
I''d be
> very interested in ideas how to remedy that.
I''ve been thinking about this one, and here''s the direction
I''ve been
leaning towards.  Haven''t done it yet so I could be wide off - someone,
please let me know if I''m being crazy...

There''s two directions that most people have for kickstarting - either
kickstart a system in place(in which case, it hopefully has a name,
final ip, and all of the distinguishing characteristics defined for it),
or kickstart on a private network and move the host to the final
location.

In the first case, puppet should be able to gather everything it needs
from the sysimage environment of the %post except for the cert and
anything possibly some facter pieces that don''t work well in the temp
environment(<scratching target=head>can''t think of anything right
now
but I could see it happening</scratching>).  Even if it doesn''t
catch
it during kickstart, after firstboot it should take(maybe even add
something to firstboot to make it easier - a remote cert sign call
like when people add Windows boxes to an AD domain).  Either way, it
wouldn''t be much different then any other node you have configured
in puppet - so same puppetmaster setup, etcetc.  Should work, right?

In the second case, puppet needs to stay agnostic until it is put in
a final location.  That''s true whether you use puppet or not - 
without more info, you can''t possibly configure it fully.  But there
are probably some base things that you''d do - bootstrapping, base
universal configurations, etcetc.  If you''re cranking out generic
systems, you''ll need a generic config, which probably means you want
the puppetmaster to run --nonodes.  So you''ll need a separate
instance of puppetmaster then you''re normal setup - which
shouldn''t
be too much of a challenge since you can specify server on the
command line and the install server seems like a good candidate
for that.

Another direction for the --nonodes setup would be to run without
a puppetmaster, and just have something pull down generic manifests
and run puppet directly.  Though, it might just be easier to do
something along the lines of:

%post --interpreter /usr/bin/puppet
file { ....

Not sure if that qualifies as a crime against kickstartmanity, but
it seems like a cool hack...

One of the kickers for both cases - I could see instances where
hardware issues crop up with a circular dependency(need to configure
additional network pieces to make it work or something).  Not sure
how to approach those...

Looking back over that - maybe location is the wrong word.  Location
is just a piece of it.  I think the right word would be role, but
that may be semantics...

I think the crux really comes down to how you can structure your
manifests so that you can easily have the generic manifest and node
specific manifests which have the same coverage but without having
to write separate manifests for both cases.

--mac

On Sat, 2007-02-10 at 00:56 +0100, Tim Stoop wrote:
> One way of doing that is netboot installation with some sort of
> automagic installation all the way through to puppet (Debian''s
> preseeding or RH/Suse kickstart). The netboot image can be on the same
> machine as the puppetmaster. That would actually solve a lot of the
> initial problems, now I think about it. Maybe we should create a
> preseeding script that sets things up until puppet. Shouldn''t be
that
> much work.
Two snippets: (1) attached is a kickstart file that kickstarts a machine
into a minimal installation, just enough to run puppet and yum (2)
cobbler [1] makes some of the netbooting easier, at least for RH-based
distros ;)
> And only one tree to add to your local puppetmaster SCM repository :)
> At least, I hope you people are all putting your /etc and related
> configs in some version control thingy... ;-)
Of course, scout''s honor.

David

[1] http://cobbler.et.redhat.com/


_______________________________________________
Puppet-users mailing list
Puppet-users@madstop.com
https://mail.madstop.com/mailman/listinfo/puppet-users

On Feb 13, 2007, at 2:34 PM, David Lutterkort wrote:
> On Sat, 2007-02-10 at 00:56 +0100, Tim Stoop wrote:
>> One way of doing that is netboot installation with some sort of
>> automagic installation all the way through to puppet (Debian''s
>> preseeding or RH/Suse kickstart). The netboot image can be on the  
>> same
>> machine as the puppetmaster. That would actually solve a lot of the
>> initial problems, now I think about it. Maybe we should create a
>> preseeding script that sets things up until puppet. Shouldn''t
be that
>> much work.
Here''s a crazy idea that I would love to see happen:

1.  Create an initial, distro agnostic image that''s used for PXE  
booting to run the puppet client.
2.  Configure puppet to be able to morph this initial image into ANY  
distro.
3.  Drink margaritas.

-Blake

On Wed, 2007-02-14 at 12:24 -0800, Blake Barnett wrote:
> On Feb 13, 2007, at 2:34 PM, David Lutterkort wrote:
> 
> > On Sat, 2007-02-10 at 00:56 +0100, Tim Stoop wrote:
> >> One way of doing that is netboot installation with some sort of
> >> automagic installation all the way through to puppet
(Debian''s
> >> preseeding or RH/Suse kickstart). The netboot image can be on the
> >> same
> >> machine as the puppetmaster. That would actually solve a lot of
the
> >> initial problems, now I think about it. Maybe we should create a
> >> preseeding script that sets things up until puppet.
Shouldn''t be that
> >> much work.
> 
> Here''s a crazy idea that I would love to see happen:
> 
> 1.  Create an initial, distro agnostic image that''s used for PXE  
> booting to run the puppet client.
> 2.  Configure puppet to be able to morph this initial image into ANY  
> distro.
> 3.  Drink margaritas.
Trouble is that distros don''t necessarily agree on how to automate
installs ;) The kickstart I sent yesterday should get you to the point
of having puppet up and running and talking to your puppetmaster.

I am sure something similar can be done for non-RH distros (and I would
love to see those files ;)

David

On Feb 14, 2007, at 2:24 PM, Blake Barnett wrote:
>
> Here''s a crazy idea that I would love to see happen:
>
> 1.  Create an initial, distro agnostic image that''s used for PXE
> booting to run the puppet client.
> 2.  Configure puppet to be able to morph this initial image into ANY
> distro.
> 3.  Drink margaritas.
I''m not sure how much value there is to this right now, considering  
that the system install tools are pretty decent and there are  
companies like openQRM creating cross-platform system installers.

That being said, it''s still something that should be a long-term  
goal, although I''m not sure it''s any nearer-term than creating
a
thinner package type that uses Puppet manifests instead of their own  
custom version.  I''ve got some hopes and dreams for integrating  
Puppet into packaging so that packages don''t need finish scripts and  
don''t have opaque ways of modifying the system, but it''d be
more like
a master''s thesis than a useful production tool, I expect.

  --
  Think twice before you speak, and then you may be able to say  
something
  more insulting than if you spoke right out at once.     - Evan Esar
  ---------------------------------------------------------------------
  Luke Kanies | http://reductivelabs.com | http://madstop.com

On Feb 14, 2007, at 12:59 PM, David Lutterkort wrote:
> On Wed, 2007-02-14 at 12:24 -0800, Blake Barnett wrote:
>> On Feb 13, 2007, at 2:34 PM, David Lutterkort wrote:
>>
>>> On Sat, 2007-02-10 at 00:56 +0100, Tim Stoop wrote:
>>>> One way of doing that is netboot installation with some sort of
>>>> automagic installation all the way through to puppet
(Debian''s
>>>> preseeding or RH/Suse kickstart). The netboot image can be on
the
>>>> same
>>>> machine as the puppetmaster. That would actually solve a lot of
the
>>>> initial problems, now I think about it. Maybe we should create
a
>>>> preseeding script that sets things up until puppet.
Shouldn''t be
>>>> that
>>>> much work.
>>
>> Here''s a crazy idea that I would love to see happen:
>>
>> 1.  Create an initial, distro agnostic image that''s used for
PXE
>> booting to run the puppet client.
>> 2.  Configure puppet to be able to morph this initial image into ANY
>> distro.
>> 3.  Drink margaritas.
>
> Trouble is that distros don''t necessarily agree on how to automate
> installs ;) The kickstart I sent yesterday should get you to the point
> of having puppet up and running and talking to your puppetmaster.
You could go as far as having the puppet image be an nfsroot.  And  
then it actually runs anaconda, or debian-installer, etc.  I''m sure  
there are gotchas, but it might be fun to try.

-Blake

On 2/14/07, David Lutterkort <dlutter@redhat.com>
wrote:
> I am sure something similar can be done for non-RH distros (and I would
> love to see those files ;)
If you want, I could send you my puppet role, which I use for Debian
domUs? I''ll attach it.

It''s unedited, so don''t use without making sure this is
exactly what
you want! (It probably isn''t, but should be easy to adept.)

-- 
Gegroet,
Tim


_______________________________________________
Puppet-users mailing list
Puppet-users@madstop.com
https://mail.madstop.com/mailman/listinfo/puppet-users