Puppet users - Nov 2012 - Seeking some Puppet advice for a newbie (specifically Virtualmin/CSF related)

HI

Forgive me if this is such a newbie issue. I am learning Puppet to manage 
several servers we have. The tutorials on manifests etc. seem a bit 
simplistic for I want, if what I want can be achieved with Puppet. 

Our servers are web hosting servers. We use Virtualmin (and Webmin) to 
manage the server and virtual servers. installing Virtualmin takes care of 
installing all the software required for virtual servers and websites, like 
Apache MySQL, PHP, BIND, Postfix etc. So we dont install those separately. 
Virtualmin does it. 

To setup a server we create the VPS, install Virtualmin, configure it, 
install CSF (Firewall), configure it and then our monitoring system agent. 
This is the minimum. We may want to then install something like Varnish 
perhaps, or anything else. At the moment I copy another server and delete 
its virtual servers as I have everything configured. But in time the 
configs just move away from each other. So I also want a way to manage all 
the configs on all servers, so they are the same. CSF in particular. 

I wondered if all this can be done in Puppet. I have no idea if it can 
because I cannot find any examples of this. 

I wondered if this is very possible with Puppet so I can continue to learn, 
OR if not then I will give up now as I already spent a while on this. 

Thanks

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/VH01FFHKx5MJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On 11/19/2012 08:57 PM, Laurence Cope wrote:
> I wondered if this is very possible with Puppet so I can continue to
> learn, OR if not then I will give up now as I already spent a while on
> this. 
Yes, you can manage firewall and everything else with puppet. Just write
your own modules or use existing ones and you''ll be fine. Prepare to
learn and to be slow in the beginning with deploying servers - like 2-3
times slower than installing all manually. But it will pay of in the
end, you''ll see.


-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Thanks. But my main stumbling block is installing Virtualmin. I am not sure
it''s a "package". How do I know? What defines a package?

I could use exec to install it I think maybe. But what about the questions it
asks during the manual install? Can Puppet answer manual questions asked during
install?

Thanks

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/6sxlVGcMd2QJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Whe I ask what is a package, I mean is a package installed using yum in a repo.
And if so, is it any repo installed on the server, or the default ones or what?
For packages not in the default repo but somewhere else I guess I add that repo
using puppet first then install the package?

I don''t know if Virtualmin is available as a package hence asking about
exec.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/hy1-H9hYU7wJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Hi,

Yes, you probably want to do something like this:

class virtualmin::install {

  $version = ''1.2.3-foo''

  exec { ''get installer script'':
   command => ''wget
http://software.virtualmin.com/gpl/scripts/install.sh'',
   cwd => ''/tmp'',
   creates => ''/tmp/install.sh'',
  }
  file { ''/tmp/install.sh'':
    mode => 0755,
    # checksum =>
''the_checksum_of_that_script_if_you_feed_paranoid''
  }
  exec { ''install it now'':
    command => ''/tmp/install.sh'',
    unless => ''test -x /usr/local/virtualmin/bin/foo &&
/usr/local/virtualmin/bin/foo --version | grep $version'',
    require => File[''/tmp/install.sh''],
  }
}

hope this kind of makes sense and please excuse any errors this pseudocode
might give you :)

Cheers,
-- 
Ilya

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Thanks Ilya, I will give this a try. 

BUT if I remember correctly, the install process will ask input during the 
install, to set some options... can Puppet handle entering options during 
installations? 

Thanks

On Monday, November 19, 2012 10:15:57 PM UTC, Ilya Dmitrichenko
wrote:
>
> Hi,
>
> Yes, you probably want to do something like this:
>
> class virtualmin::install {
>
>   $version = ''1.2.3-foo''
>
>   exec { ''get installer script'':
>    command => ''wget
http://software.virtualmin.com/gpl/scripts/install.sh
> '',
>    cwd => ''/tmp'',
>    creates => ''/tmp/install.sh'',
>   }
>   file { ''/tmp/install.sh'':
>     mode => 0755,
>     # checksum =>
''the_checksum_of_that_script_if_you_feed_paranoid''
>   }
>   exec { ''install it now'':
>     command => ''/tmp/install.sh'',
>     unless => ''test -x /usr/local/virtualmin/bin/foo &&
> /usr/local/virtualmin/bin/foo --version | grep $version'',
>     require => File[''/tmp/install.sh''],
>   }
> }
>
> hope this kind of makes sense and please excuse any errors this pseudocode 
> might give you :)
>
> Cheers,
> -- 
> Ilya 
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/AxVrJPPnfDgJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On 11/20/2012 10:44 AM, Laurence Cope wrote:
> Thanks Ilya, I will give this a try. 
> 
> BUT if I remember correctly, the install process will ask input during
> the install, to set some options... can Puppet handle entering options
> during installations? 
Puppet is not a shell script, it''s a state-ensuring tool. So with
puppet
you are forcing your system to be in the state you want it to be. So,
you should avoid running ''installation scripts''.

Best way to solve your problem would be to distribute virtualmin through
distro native packages, and later just modify configuration files
according to your needs.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

So does that mean I still have to manually setup server and install what I 
need, then just use Puppet to keep them all configured the way I want them? 

So I would still copy my virtual servers to create a new one (because it 
takes too long to setup a virtual server from scratch and install 
everything I need). 

I was hoping to find some sort of "server setup" automation tool so I
can
create a new CentOS VM for example, then just have something install all 
the software I need automatically. 

Thanks


On Monday, November 19, 2012 7:57:33 PM UTC, Laurence Cope
wrote:
>
> HI
>
> Forgive me if this is such a newbie issue. I am learning Puppet to manage 
> several servers we have. The tutorials on manifests etc. seem a bit 
> simplistic for I want, if what I want can be achieved with Puppet. 
>
> Our servers are web hosting servers. We use Virtualmin (and Webmin) to 
> manage the server and virtual servers. installing Virtualmin takes care of 
> installing all the software required for virtual servers and websites, like
> Apache MySQL, PHP, BIND, Postfix etc. So we dont install those separately. 
> Virtualmin does it. 
>
> To setup a server we create the VPS, install Virtualmin, configure it, 
> install CSF (Firewall), configure it and then our monitoring system agent. 
> This is the minimum. We may want to then install something like Varnish 
> perhaps, or anything else. At the moment I copy another server and delete 
> its virtual servers as I have everything configured. But in time the 
> configs just move away from each other. So I also want a way to manage all 
> the configs on all servers, so they are the same. CSF in particular. 
>
> I wondered if all this can be done in Puppet. I have no idea if it can 
> because I cannot find any examples of this. 
>
> I wondered if this is very possible with Puppet so I can continue to 
> learn, OR if not then I will give up now as I already spent a while on 
> this. 
>
> Thanks
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/fsdRJsMHrT8J.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Tue, Nov 20, 2012 at 10:40 AM, Laurence Cope
<amitywebsolutions@gmail.com> wrote:
> So does that mean I still have to manually setup server and install what I
> need, then just use Puppet to keep them all configured the way I want them?
>
> So I would still copy my virtual servers to create a new one (because it
> takes too long to setup a virtual server from scratch and install
everything
> I need).
>
> I was hoping to find some sort of "server setup" automation tool
so I can
> create a new CentOS VM for example, then just have something install all
the
> software I need automatically.
Puppet can certainly do the ''install all the software I need
automatically'' bit for you, but its and your job is made much easier
by asking/telling it to install software in your OSes ''native''
packaging format.  As you mention CentOS, that would be RPM.  RPMs for
Virtualmin can be found at http://www.webmin.com/vdownload.html.  In
order to have Puppet install that for you, you will need to create
your own Yum repository, have Puppet configure yum to make use of that
repo, then create a manifest that installs the package.

Hope this helps,

Matt.
>
> Thanks
>
>
> On Monday, November 19, 2012 7:57:33 PM UTC, Laurence Cope wrote:
>>
>> HI
>>
>> Forgive me if this is such a newbie issue. I am learning Puppet to
manage
>> several servers we have. The tutorials on manifests etc. seem a bit
>> simplistic for I want, if what I want can be achieved with Puppet.
>>
>> Our servers are web hosting servers. We use Virtualmin (and Webmin) to
>> manage the server and virtual servers. installing Virtualmin takes care
of
>> installing all the software required for virtual servers and websites,
like
>> Apache MySQL, PHP, BIND, Postfix etc. So we dont install those
separately.
>> Virtualmin does it.
>>
>> To setup a server we create the VPS, install Virtualmin, configure it,
>> install CSF (Firewall), configure it and then our monitoring system
agent.
>> This is the minimum. We may want to then install something like Varnish
>> perhaps, or anything else. At the moment I copy another server and
delete
>> its virtual servers as I have everything configured. But in time the
configs
>> just move away from each other. So I also want a way to manage all the
>> configs on all servers, so they are the same. CSF in particular.
>>
>> I wondered if all this can be done in Puppet. I have no idea if it can
>> because I cannot find any examples of this.
>>
>> I wondered if this is very possible with Puppet so I can continue to
>> learn, OR if not then I will give up now as I already spent a while on
this.
>>
>> Thanks
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/puppet-users/-/fsdRJsMHrT8J.
>
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

>
> you will need to create your own Yum repository, have Puppet configure yum 
> to make use of that repo, then create a manifest that installs the package.

Ah right... this bit helps a lot. never thought of creating an own repo, 
that makes sense now. so if its in a repo puppet can do it. I will look 
into that, and also request Virtualmin do it because I asked them about 
this on their forum, but they had no experience with Puppet. Makes sense 
for it to come from a repo they manage. 

thanks


On Monday, 19 November 2012 19:57:33 UTC, Laurence Cope
wrote:
>
> HI
>
> Forgive me if this is such a newbie issue. I am learning Puppet to manage 
> several servers we have. The tutorials on manifests etc. seem a bit 
> simplistic for I want, if what I want can be achieved with Puppet. 
>
> Our servers are web hosting servers. We use Virtualmin (and Webmin) to 
> manage the server and virtual servers. installing Virtualmin takes care of 
> installing all the software required for virtual servers and websites, like
> Apache MySQL, PHP, BIND, Postfix etc. So we dont install those separately. 
> Virtualmin does it. 
>
> To setup a server we create the VPS, install Virtualmin, configure it, 
> install CSF (Firewall), configure it and then our monitoring system agent. 
> This is the minimum. We may want to then install something like Varnish 
> perhaps, or anything else. At the moment I copy another server and delete 
> its virtual servers as I have everything configured. But in time the 
> configs just move away from each other. So I also want a way to manage all 
> the configs on all servers, so they are the same. CSF in particular. 
>
> I wondered if all this can be done in Puppet. I have no idea if it can 
> because I cannot find any examples of this. 
>
> I wondered if this is very possible with Puppet so I can continue to 
> learn, OR if not then I will give up now as I already spent a while on 
> this. 
>
> Thanks
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/YSLhcnZ-Vx0J.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Tue, Nov 20, 2012 at 1:54 PM, Laurence Cope
<amitywebsolutions@gmail.com> wrote:
>> you will need to create your own Yum repository, have Puppet configure
yum
>> to make use of that repo, then create a manifest that installs the
package.
>
> Ah right... this bit helps a lot. never thought of creating an own repo,
> that makes sense now. so if its in a repo puppet can do it. I will look
into
> that, and also request Virtualmin do it because I asked them about this on
> their forum, but they had no experience with Puppet. Makes sense for it to
> come from a repo they manage.
I generally favour my own private yum repositories rather than
upstream repositories for the following reasons:

1) Most client environments I''ve worked in have no Internet access; or
if they do it''ll just be 1 or 2 servers, of which neither will be my
Puppet/Yum server.
2) I can control what versions of which packages are installed when
the Puppet manifest states ''ensure=>latest''.  With a public
repository, I''d be at the mercy of the upstream vendor; as soon as
they release a new package all of my systems would be upgraded with no
testing/staging possible (there was a fairly recent thread in this
group when PuppetLabs released puppet-3.0 into the same repository as
puppet-2.x.  It caught a lot of folks out).  This can be mitigated, of
course, by using ''ensure=>installed'', but then that means
upgrades are
painful.
3) It''s quicker; your internal network should be much faster and more
reliable than going out to the wider Internet.

Thanks,

Matt.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

In regard to: Re: [Puppet Users] Re: Seeking some Puppet advice for a...:
> On Tue, Nov 20, 2012 at 1:54 PM, Laurence Cope
> <amitywebsolutions@gmail.com> wrote:
>>> you will need to create your own Yum repository, have Puppet
configure yum
>>> to make use of that repo, then create a manifest that installs the
package.
>>
>> Ah right... this bit helps a lot. never thought of creating an own
repo,
>> that makes sense now. so if its in a repo puppet can do it. I will look
into
>> that, and also request Virtualmin do it because I asked them about this
on
>> their forum, but they had no experience with Puppet. Makes sense for it
to
>> come from a repo they manage.
>
> I generally favour my own private yum repositories rather than
> upstream repositories for the following reasons:
+1, for all the reasons Matt mentioned.

Making your own repo may seem daunting, but it''s not bad at all.  If
you
have an internal server that''s running http and has enough disk space
to
store your RPMs, you''re already most of the way there.

Tim
-- 
Tim Mooney                                             Tim.Mooney@ndsu.edu
Enterprise Computing & Infrastructure                  701-231-1076 (Voice)
Room 242-J6, IACC Building                             701-231-8541 (Fax)
North Dakota State University, Fargo, ND 58105-5164

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Puppet can''t provide input when a script prompts for it, but usually
there
are alternative ways to install the software. Most install scripts only 
untar the software to a directory, ask some questions and write some 
configuration files based on your answers. Most popular software has some 
way to automate installation.

Install the software on a test-VM, and see what the end-result is. Which 
files are created, which daemons are started, which config files are 
modified. That''s what you need to recreate with Puppet. You
don''t need to
recreate all the steps the installer does. You simply need to achieve the 
same end-result.

Either: 

   1. The install script may allow command-line parameters to specify the 
   various settings. Simply provide those parameters when executing the script 
   via Puppet.
   2. If the installation is not too complicated you can skip the 
   installation script and perform all the steps via Puppet, i.e. untar to a 
   directory, drop a config file somewhere in /etc, configure the service to 
   start.
   3. You can create your own package that performs the steps that the 
   install script would and simply install the package via Puppet.

For simple installations I would choose option 2, especially since you 
don''t have your own package repo yet. For more complex installations,
I''d
go for 3.

Might be worth making your own repo anyway, since it''s quite a good way
to
install custom software in a structured way. All those tools you drop in 
/usr/local could easily be packaged, providing you with all the advantages 
a packaging system has. Also, Puppet knows how to deal with packages, so 
your Puppet manifests will be simpler as well.

Regards, Martijn Heemels

Op dinsdag 20 november 2012 10:44:01 UTC+1 schreef Laurence Cope het 
volgende:
>
> Thanks Ilya, I will give this a try. 
>
> BUT if I remember correctly, the install process will ask input during the 
> install, to set some options... can Puppet handle entering options during 
> installations? 
>
> Thanks
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/ru9YtcrT9_IJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On 11/20/2012 03:12 PM, Matthew Burgess wrote:
> (there was a fairly recent thread in this
> group when PuppetLabs released puppet-3.0 into the same repository as
> puppet-2.x.  It caught a lot of folks out).
I didn''t use ensure=>latest and didn''t plan to upgrade so
soo, but was
also in trouble because of Cobbler using external puppet repo for 
deploying puppet onto freshly installed machines... offcourse every one 
of them was puppet agent 3.0 :D So I decided it was much easier for me 
to upgrade master then to implement other solutions.


 > This can be mitigated, of
> course, by using ''ensure=>installed'', but then that
means upgrades are
> painful.
Why? You can always use cexec or mcollective with some kind of custom 
plugin for doing those sort of things.


-- 
Jakov Sosic
www.srce.unizg.hr

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Tue, Nov 20, 2012 at 9:37 PM, Jakov Sosic <jsosic@srce.hr>
wrote:
> On 11/20/2012 03:12 PM, Matthew Burgess wrote:
>> This can be mitigated, of
>>
>> course, by using ''ensure=>installed'', but then
that means upgrades are
>> painful.
>
> Why? You can always use cexec or mcollective with some kind of custom
plugin
> for doing those sort of things.
I knew I should have qualified that statement: It''s painful because
I''ve not had to do it
yet because I use my own repos and couldn''t immediately think of a
non-painful way of upgrading packages :-)

Thanks for the MCollective pointer; it''s been on my list of things to
investigate for a while now!

Regards,

Matt.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

You can actually install in a custom directory and build a package from
that pretty easily. Look at directory provider from
https://github.com/jordansissel/fpm on how to do that. Of course you
will then want to tweak specific configuration options like IP address
through puppet. 

Hope that helps :)

Best, Nikola


On Tue, Nov 20, 2012 at 01:44:01AM -0800, Laurence Cope
wrote:
> Thanks Ilya, I will give this a try. 
> 
> BUT if I remember correctly, the install process will ask input during the 
> install, to set some options... can Puppet handle entering options during 
> installations? 
> 
> Thanks
> 
> On Monday, November 19, 2012 10:15:57 PM UTC, Ilya Dmitrichenko wrote:
> >
> > Hi,
> >
> > Yes, you probably want to do something like this:
> >
> > class virtualmin::install {
> >
> >   $version = ''1.2.3-foo''
> >
> >   exec { ''get installer script'':
> >    command => ''wget
http://software.virtualmin.com/gpl/scripts/install.sh
> > '',
> >    cwd => ''/tmp'',
> >    creates => ''/tmp/install.sh'',
> >   }
> >   file { ''/tmp/install.sh'':
> >     mode => 0755,
> >     # checksum =>
''the_checksum_of_that_script_if_you_feed_paranoid''
> >   }
> >   exec { ''install it now'':
> >     command => ''/tmp/install.sh'',
> >     unless => ''test -x /usr/local/virtualmin/bin/foo
&&
> > /usr/local/virtualmin/bin/foo --version | grep $version'',
> >     require => File[''/tmp/install.sh''],
> >   }
> > }
> >
> > hope this kind of makes sense and please excuse any errors this
pseudocode
> > might give you :)
> >
> > Cheers,
> > -- 
> > Ilya 
> >
> 
> -- 
> You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
> To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/AxVrJPPnfDgJ.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
> 
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.